JP7296340B2 - Excavator - Google Patents

Description

The present invention relates to excavators.

2. Description of the Related Art Conventionally, there is known a surroundings monitoring device that uses a laser radar attached to an upper revolving body of an excavator to monitor the surroundings while obtaining the distance and direction of an object existing around the excavator with respect to the excavator (for example, Patent Document 1).

The surroundings monitoring device determines that an object is a stationary obstacle when the distance and direction of the object with respect to the excavator do not change for a predetermined period of time, and determines the object as a moving obstacle when the distance and direction change.

JP 2008-163719 A

However, the laser radar described in Patent Document 1 irradiates an object with one laser beam and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between itself and a very limited portion of the surface of the object.

SUMMARY OF THE INVENTION In view of the above, it is an object of the present invention to provide an excavator capable of obtaining the distance and direction to the excavator of objects present in a wider range around the excavator.

In order to achieve the above object, an excavator according to an embodiment of the present invention includes a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, and a plurality of rotating bodies arranged on the upper revolving body. and a stereo camera, each of the plurality of stereo cameras includes a plurality of imaging mechanisms in one housing, and further includes a display device for displaying an image acquired by the imaging mechanism, wherein the display device Frames are displayed for cliffs, depressions, slopes, or holes as the ground and people or obstacles to be displayed .

By the means described above, the present invention can provide an excavator capable of obtaining the distance and direction to the excavator of objects present in a wider range around the excavator.

1 is a block diagram schematically showing a configuration example of an image generation device according to an embodiment of the present invention; FIG. It is a figure which shows the structural example of the excavator in which an image generation apparatus is mounted. FIG. 4 is a diagram showing an example of a space model onto which an input image is projected; FIG. 4 is a diagram showing an example of the relationship between a spatial model and an image plane to be processed; FIG. 4 is a diagram for explaining correspondence between coordinates on an input image plane and coordinates on a space model; It is a figure for demonstrating matching between coordinates by a coordinate matching means. It is a figure for demonstrating the effect|action of a parallel line group. It is a figure for demonstrating the effect|action of an auxiliary|assistant line group. 5 is a flow chart showing the flow of processing target image generation processing and output image generation processing. It is a display example (part 1) of an output image. 1 is a top view (part 1) of a shovel on which an image generation device is mounted; FIG. FIG. 2 is a diagram showing input images of three cameras mounted on a shovel and output images generated using the input images; FIG. 10 is a diagram for explaining image disappearance prevention processing for preventing disappearance of an object in an overlapping area of imaging ranges of two cameras; 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and an output image obtained by applying image loss prevention processing to the output image of FIG. 12; FIG. FIG. 3 is a diagram showing input range images of three stereo cameras mounted on a shovel and output range images generated using the input range images; FIG. 16 is a comparison diagram showing the difference between the output distance image shown in FIG. 15 and an output distance image obtained by applying distance image loss prevention processing to the output distance image of FIG. 15; FIG. 10 is a diagram showing an example of a synthesized output image obtained by synthesizing an output distance image with an output image; FIG. 10 is a diagram showing another example of a synthesized output image obtained by synthesizing an output distance image with an output image; FIG. 4 is a partial right side view of a shovel on which the image generation device is mounted; FIG. 20 is an enlarged view of the ridge in FIG. 19; It is a figure which shows the example of an output distance image. FIG. 10 is a diagram showing an example of an output image; FIG. 4 is a flowchart showing the flow of first pixel extraction processing; FIG. 4 is a partial right side view of a shovel on which the image generation device is mounted; 25 is an enlarged view of the shallow depression in FIG. 24; FIG. It is a figure which shows the example of an output distance image. FIG. 10 is a diagram showing an example of an output image; FIG. 9 is a flowchart showing the flow of second pixel extraction processing; It is a flow chart which shows a flow of driving support processing. It is a figure which shows the excavator in which an image generation apparatus is mounted. FIG. 10 is a comparison diagram showing the difference between the pre-correction output range image and the corrected output range image; FIG. 11 is a comparison diagram (part 1) showing the difference between the pre-correction post-combination output image and the post-correction post-combination output image; FIG. 11 is a comparison diagram (part 2) showing the difference between the post-correction post-combination output image and the post-correction post-combination output image;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generation device 100 according to an embodiment of the invention.

The image generation device 100 is an example of a work machine perimeter monitoring device for monitoring the perimeter of the work machine, and is composed of a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and a stereo camera 6. be done. Specifically, the image generation device 100 generates an output image based on an input image captured by the camera 2 mounted on the work machine and an input distance image output by the stereo camera 6, and provides the output image to the operator. Present.

FIG. 2 is a diagram showing a configuration example of an excavator 60 as a work machine on which the image generation device 100 is mounted. A revolving body 63 is mounted so as to be rotatable around a turning axis PV. The excavator 60 generates an output image using the two-dimensional array of distance information.

The upper swing body 63 has a cab (driver's cab) 64 on its front left side, an excavation attachment E on its front central part, and cameras 2 (right side camera 2R, rear camera 2B) on its right side and rear side. ) and a stereo camera 6 (a right side stereo camera 6R and a rear stereo camera 6B). A display unit 5 is installed at a position within the cab 64 that is easily visible to the operator.

Next, each component of the image generation device 100 will be described.

The control unit 1 is a computer including a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), NVRAM (Non-Volatile Random Access Memory), etc. Programs corresponding to each of the adding means 10, the image generating means 11, the pixel extracting means 12, the distance correcting means 13, the distance image synthesizing means 14, and the driving support means 15 are stored in ROM or NVRAM, and RAM is used as a temporary storage area. The CPU is made to execute the processing corresponding to each means while using .

The camera 2 is a device for acquiring an input image showing the surroundings of the excavator 60, and is attached to the right side surface and the rear surface of the upper revolving body 63 so as to capture an image of the blind spot of the operator in the cab 64, for example. (See FIG. 2.), a right side camera 2R and a rear camera 2B, each of which has an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side and the rear side of the upper rotating body 63 (for example, the front side and the left side), and is equipped with a wide-angle lens or a fisheye lens so as to capture a wide range. may be

Also, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1 . Note that when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting the apparent distortion and tilt caused by the use of the lens is sent to the control unit 1. However, the input image may be output to the control unit 1 as it is without correcting its apparent distortion or tilt. In that case, the control unit 1 corrects the apparent distortion and tilt.

The input unit 3 is a device that enables the operator to input various information to the image generation device 100, and includes, for example, a touch panel, button switches, pointing device, keyboard, and the like.

The storage unit 4 is a device for storing various information, such as a hard disk, an optical disk, or a semiconductor memory.

The display unit 5 is a device for displaying image information. Display an image.

The stereo camera 6 is a device for acquiring a two-dimensional array of distance information of objects existing around the excavator 60 . (See FIG. 2). Note that the stereo camera 6 may be attached to any one of the front surface, left side surface, right side surface, and rear surface of the upper rotating body 63, or may be attached to all surfaces.

The stereo camera 6 is a device having imaging functions of a plurality of cameras. In this embodiment, the stereo camera 6 is a device that integrates the imaging functions of two cameras, and includes two sets of imaging mechanisms (a combination of an imaging element and a lens mechanism) and one shared shutter mechanism. It is provided in one housing. However, the stereo camera 6 may be composed of a plurality of independent cameras.

Moreover, the plural sets of imaging mechanisms in the stereo camera 6 are arranged at predetermined intervals from each other. In this embodiment, the two sets of imaging mechanisms are arranged side by side with a predetermined interval in the longitudinal direction (vertical direction). Note that the two sets of imaging mechanisms may be arranged side by side with a predetermined interval in the lateral direction (horizontal direction), or may be arranged with a predetermined interval in each of the vertical direction (vertical direction) and the lateral direction (horizontal direction). may be juxtaposed.

Further, while the camera 2 acquires luminance, hue value, saturation value, etc. for each pixel, the stereo camera 6 acquires distance information for each pixel. In this embodiment, the stereo camera 6 detects each pixel of an image output by one imaging mechanism (hereinafter referred to as a “basic image”) based on the parallax between the images output by each of the two sets of imaging mechanisms. and the distance between the stereo camera 6 is derived. Then, the stereo camera 6 substitutes the derived distance for the value of each pixel of the basic image to generate distance information in a two-dimensional array, and outputs the distance information to the control unit 1 . Therefore, hereinafter, the distance information in the two-dimensional array by the stereo camera 6 is compared with the input image and the output image of the camera 2 and called an input distance image and an output distance image. Also, the resolution (the number of pixels) of the input range image and the output range image of the stereo camera 6 may be the same as or different from the resolution of the input image and the output image of the camera 2 . Also, one or more pixels in the input image or output image of the camera 2 may be associated in advance with one or more pixels in the input range image or output range image of the stereo camera 6 . Also, the input distance image may be generated by the control unit 1 based on a plurality of images output by each imaging mechanism of the stereo camera 6 .

Also, the input image planes of the plurality of imaging mechanisms that constitute the stereo camera 6 are preferably parallel to each other, and most preferably form the same plane. This is because the processing for deriving the distance from the parallax can be performed easily and with high accuracy.

Moreover, the stereo camera 6 may be attached to a position other than the right side and the rear side of the upper rotating body 63 (for example, the front side and the left side), similarly to the camera 2. Alternatively, a fisheye lens may be attached.

Similarly to the camera 2 , the stereo camera 6 acquires an input distance image according to a control signal from the control unit 1 and outputs the acquired input distance image to the control unit 1 . Note that, as with the camera 2, when the stereo camera 6 acquires an input distance image using a fisheye lens or a wide-angle lens, the stereo camera 6 corrects the apparent distortion and tilt caused by using these lenses to obtain a corrected input distance image. Although the image is output to the control unit 1, the input distance image without correction of apparent distortion and tilt may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

In addition, the image generating apparatus 100 generates an image to be processed based on the input image, and performs image conversion processing on the image to be processed so that the positional relationship and sense of distance to the surrounding features can be intuitively grasped. The output image may be generated and then presented to the operator.

The image generation device 100 performs similar processing on the input range image. In that case, the image to be processed is replaced with the distance image to be processed. The same applies to the following description.

The “image to be processed” is an image to be subjected to image transformation processing (for example, scale transformation, affine transformation, distortion transformation, viewpoint transformation processing, etc.) generated based on the input image. is an input image from a camera that captures from above and includes a horizontal image (for example, a sky part) due to its wide angle of view. The input image is projected onto a given spatial model so as not to be displayed unnaturally (e.g., the sky is not treated as being on the ground), and then the projection image projected onto that spatial model is projected onto another two-dimensional model. It is an image suitable for image transformation processing obtained by reprojecting onto a dimensional plane. Note that the processing target image may be used as an output image as it is without performing image conversion processing.

The "spatial model" is at least a plane or curved surface other than the processing target image plane, which is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane, or an angle between the processing target image plane It is a projection target of an input image, which is composed of one or a plurality of planes or curved surfaces.

Note that the image generating apparatus 100 may generate an output image by performing image conversion processing on a projection image projected onto the spatial model without generating a processing target image. Also, the projection image may be used as the output image as it is without image conversion processing.

FIG. 3 is a diagram showing an example of the space model MD on which the input image is projected, and FIG. 3A shows the relationship between the excavator 60 and the space model MD when the excavator 60 is viewed from the side. 3B shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and has a planar region R1 inside its bottom surface and a curved surface region R2 inside its side surface.

Also, FIG. 4 is a diagram showing an example of the relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the space model MD. For clarity, FIG. 4 shows the space model MD in a cylindrical shape instead of a semi-cylindrical shape as shown in FIG. may be The same applies to subsequent figures. Further, as described above, the processing target image plane R3 may be a circular area that includes the plane area R1 of the space model MD, or may be an annular area that does not include the plane area R1 of the space model MD.

Next, various means that the control unit 1 has will be described.

The coordinate association means 10 is means for associating the coordinates on the input image plane where the input image captured by the camera 2 is located, the coordinates on the space model MD, and the coordinates on the processing target image plane R3, For example, various parameters related to the camera 2, such as the optical center of the camera 2, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, and the projection method, which are preset or input via the input unit 3 and the predetermined mutual positional relationship among the input image plane, the space model MD, and the processing target image plane R3, the coordinates on the input image plane, the coordinates on the space model MD, and the processing target image coordinates on the plane R3, and the correspondence relationship is stored in the input image/spatial model correspondence map 40 and the space model/process target image correspondence map 41 of the storage unit 4. FIG.

Note that, when the image to be processed is not generated, the coordinate association means 10 associates the coordinates on the space model MD with the coordinates on the image plane R3 to be processed, and the spatial model/processing object of the correspondence relationship. Storing in the image correspondence map 41 is omitted.

The coordinate association means 10 also performs similar processing on the input range image output by the stereo camera. In that case, the camera, the input image plane, and the image plane to be processed can be read as the stereo camera, the input range image plane, and the range image plane to be processed. The same applies to the following description.

The image generating means 11 is a means for generating an output image. For example, by applying scale transformation, affine transformation, or distortion transformation to the image to be processed, the coordinates on the image plane R3 to be processed and the position of the output image are changed. and the coordinates on the plane of the output image, the correspondence relationship is stored in the processing object image/output image correspondence map 42 of the storage unit 4, and the coordinate correspondence means 10 stores the values of the input image/space model correspondence While referring to the map 40 and the spatial model/image-to-be-processed correspondence map 41, the value of each pixel in the output image (e.g., luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image. to generate an output image.

Further, the image generating means 11 uses the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, etc., which are set in advance or are input via the input unit 3 . Based on the various parameters, the coordinates on the processing target image plane R3 and the coordinates on the output image plane where the output image is located are associated with each other, and the correspondence relationship is stored in the processing target image/output image correspondence map 42 of the storage unit 4. While referring to the input image/spatial model correspondence map 40 and the space model/processing target image correspondence map 41 in which the values are stored by the coordinate association means 10, the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image to generate an output image.

Note that the image generating means 11 may generate an output image by changing the scale of the image to be processed without using the concept of the virtual camera.

Further, when the image to be processed is not generated, the image generating means 11 associates the coordinates on the space model MD with the coordinates on the output image plane in accordance with the applied image transformation processing, and performs correspondence between the input image and the space model. While referring to the map 40, the output image is generated by associating the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) with the value of each pixel in the input image. In this case, the image generating means 11 omits the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane, and the storage of the corresponding relationship in the processing target image/output image correspondence map 42. .

The image generating means 11 also performs similar processing on the input distance image or the processing target distance image. In that case, the output image plane is read as the output range image plane. The same applies to the following description.

The pixel extracting means 12 is means for extracting pixels satisfying a predetermined condition from the input range image of the stereo camera 6 . For example, the pixel extracting means 12 extracts pixels satisfying a predetermined condition from pixels having pixel values representing the distance between the features around the shovel 60 and the stereo camera 6 in the input range image.

Specifically, the pixel extracting means 12 extracts the distance between the plane on which the excavator 60 is positioned (hereinafter referred to as “installation surface”) and the stereo camera 6 (hereinafter referred to as “installation surface distance”) and A pixel whose difference is equal to or greater than a predetermined distance is extracted. Note that the installation surface may be an inclined surface. Also, the installation surface distance is set in advance for each pixel in the input distance image of the stereo camera 6 based on the installation position and installation angle of the stereo camera 6 . Then, the pixel extracting means 12 replaces the pixel values of the unextracted pixels with a predetermined value (for example, zero as the minimum value), and maintains the pixel values of the extracted pixels as they are, thereby extracting the input range image. to correct. Details of the pixel extraction means 12 will be described later.

The distance correction means 13 is means for correcting distance information acquired by the stereo camera. For example, the distance correction means 13 corrects the pixel values extracted by the pixel extraction means 12 . Specifically, the distance correction means 13 adjusts the pixel value representing the distance between the feature around the excavator 60 and the stereo camera 6 to the distance between the reference outside the stereo camera 6 and the feature. Correct to a value that represents the distance. Further, the distance correction means 13 may correct the pixel values in the input distance image output by the stereo camera 6 before extraction by the pixel extraction means 12 . The details of the distance correction means 13 will be described later.

The distance image synthesizing means 14 is means for synthesizing an image relating to the camera and an image relating to the stereo camera. For example, the distance image synthesizing means 14 synthesizes the output image based on the input image of the camera 2 and the output distance image based on the input distance image of the stereo camera 6 , which are generated by the image generating means 11 . The output distance image is an output distance image after at least one of extraction by the pixel extraction means 12 and correction by the distance correction means 13 is performed. However, the output distance image may be an output distance image before being extracted by the pixel extraction means 12 and corrected by the distance correction means 13 . Details of the distance image synthesizing means 14 will be described later.

The driving support means 15 is means for supporting driving of the excavator 60 . For example, the driving support unit 15 executes a function of supporting driving of the excavator 60 when it is determined that a predetermined feature exists in the moving direction of the excavator 60 .

The "predetermined feature" is, for example, a person such as a worker who is likely to collide with the excavator 60, an obstacle that hinders the movement of the excavator 60, a cliff that may cause the excavator 60 to slide, fall, or get stuck, or a depression. , slopes, holes, etc.

Functions that support driving (hereinafter referred to as “driving support functions”) include, for example, outputting an alarm, displaying a warning on the display unit 5, turning on/off a warning light, decelerating the excavator 60, and driving the excavator 60 at a traveling speed. , stop the excavator 60 from traveling, and the like.

Specifically, when the driving support means 15 determines that there is a cliff in the moving direction (backward) of the excavator 60 while the excavator 60 is moving (retreating), the driving support means 15 informs the display unit 5 of the existence of the cliff. Show warning. Note that the moving direction of the shovel 60 is detected based on, for example, the operation content of the operation lever (for example, the operation direction, the operation amount, etc.).

Further, the driving support means 15 may reduce the movement (retreat) speed of the excavator 60 as the distance between the excavator 60 and the cliff becomes shorter, and when the distance between the excavator 60 and the cliff becomes equal to or less than a predetermined distance. The movement (retraction) of the shovel 60 may be stopped when the shovel 60 is in the state of being.

Moreover, the driving support means 15 determines whether or not a predetermined feature exists based on the output of the stereo camera 6 . For example, when the stereo camera 6 detects a feature having a depth of a predetermined distance or more from the installation surface, the driving support means 15 determines that there is a depression that may cause the shovel 60 to slide down. Further, the driving support means 15 causes the excavator 60 to slide down only when at least one of the actual depth, opening area, width, and depth of the depressed portion calculated based on the input range image satisfies a predetermined condition. It may be determined that there is a possible recession. In addition, the driving support means 15 determines whether there is a depressed portion that may cause the excavator 60 to slide down, based on the increasing gradient of the pixel values along the predetermined direction (moving direction of the excavator 60) in the input range image. You can judge.

Alternatively, the driving support means 15 may determine that there is an obstacle that may collide with the excavator 60 when the stereo camera 6 detects a feature having a height equal to or greater than a predetermined distance from the installation surface. . Further, the driving support means 15 determines whether or not there is an obstacle that may collide with the excavator 60 based on the decreasing gradient of the pixel values along the predetermined direction (moving direction of the excavator 60) in the input range image. You can judge.

Next, an example of specific processing by the coordinate matching means 10 and the image generating means 11 will be described.

The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, Hamilton's quaternion.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. It is assumed that the space model is expressed as a solid plane in the XYZ orthogonal coordinate system.

First, the coordinate correspondence means 10 converts the coordinates on the space model (coordinates on the XYZ coordinate system) to the coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is set to the optical After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" indicates that the direction is opposite. This is because the UVW coordinate system has the +W direction in front of the camera, and the XYZ coordinate system has the -Z direction in the vertical downward direction.

If there are a plurality of cameras 2, each camera 2 has its own UVW coordinate system. It will be moved and rotated.

In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, then the Z axis is rotated to coincide with the -W axis, and the X axis is changed to the U Since it is realized by rotating to coincide with the axis, the coordinate correspondence means 10 can integrate these two rotations into one rotation operation by describing this transformation in Hamilton's quaternion. can.

By the way, the rotation for matching a vector A with another vector B corresponds to the process of rotating by the angle formed by the vector A and the vector B about the normal line of the plane spanned by the vector A and the vector B. , and that angle is θ, from the inner product of vector A and vector B, the angle θ is

で表されることとなる。

will be represented by

Also, the unit vector N of the normal to the surface spanned by vector A and vector B is obtained from the outer product of vector A and vector B as

で表されることとなる。

will be represented by

In addition, the quaternion, when i, j, and k are imaginary units, is

を満たす超複素数であり、本実施例において、四元数Ｑは、実成分をｔ、純虚成分をａ、ｂ、ｃとして、

is a hyper-complex number that satisfies

で表されるものとし、四元数Ｑの共役四元数は、

and the conjugate quaternion of the quaternion Q is

で表されるものとする。

shall be represented by

The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero). , c can also represent a rotational movement around an arbitrary vector.

Furthermore, the quaternion Q can be expressed as a single rotation operation by integrating multiple consecutive rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as A point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

ここで、本実施例において、Ｚ軸を－Ｗ軸に一致させる回転を表す四元数をＱｚとすると、ＸＹＺ座標系におけるＸ軸上の点Ｘは、点Ｘ'に移動させられるので、点Ｘ'は、

Here, in this embodiment, let Qz be a quaternion representing the rotation that aligns the Z-axis with the -W-axis. X' is

で表されることとなる。

will be represented by

In this embodiment, if Qx is a quaternion representing a rotation that aligns the line connecting the point X' on the X axis and the origin with the U axis, then "Align the Z axis with the -W axis, and further , the rotation that aligns the X axis with the U axis.

で表されることとなる。

will be represented by

From the above, the coordinates P′ when arbitrary coordinates P on the space model (XYZ coordinate system) are expressed by coordinates on the input image plane (UVW coordinate system) are

で表されることとなり、四元数Ｒがカメラ２のそれぞれで不変であることから、座標対応付け手段１０は、以後、この演算を実行するだけで空間モデル（ＸＹＺ座標系）上の座標を入力画像平面（ＵＶＷ座標系）上の座標に変換することができる。

Since the quaternion R is invariant for each of the cameras 2, the coordinate correspondence means 10 will thereafter obtain the coordinates on the space model (XYZ coordinate system) simply by executing this calculation. It can be transformed into coordinates on the input image plane (UVW coordinate system).

After converting the coordinates on the space model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate correspondence means 10 aligns the optical center C of the camera 2 (coordinates on the UVW coordinate system) with the space An incident angle α formed by a line segment CP′ connecting an arbitrary coordinate P on the model with a coordinate P′ expressed in the UVW coordinate system and the optical axis G of the camera 2 is calculated.

Further, the coordinate association means 10 establishes an intersection point E between the plane H and the optical axis G and the coordinate P' in the plane H parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and including the coordinate P'. and the declination angle φ formed by the U' axis on the plane H and the length of the line segment EP' are calculated.

In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f. , stereoscopic projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/2)), equidistant projection (h=fα), etc., and select an appropriate projection method to determine the image height Calculate h.

After that, the coordinate correspondence means 10 decomposes the calculated image height h into the U component and the V component on the UV coordinate system by the deflection angle φ, and uses a numerical value corresponding to the pixel size per pixel of the input image plane R4. By division, the coordinates P(P') on the space model MD can be associated with the coordinates on the input image plane R4.

If the pixel size of one pixel in the U-axis direction of the input image plane R4 is _aU , and the pixel size of one pixel in the V-axis direction of the input image plane R4 is _aV , the coordinate P on the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P') are

で表されることとなる。

will be represented by

In this way, the coordinate associating means 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD and camera identifiers. , and coordinates on the input image plane R4 are stored in the input image/space model correspondence map 40 in association with each other.

In addition, since the coordinate correspondence unit 10 uses quaternions to calculate coordinate conversion, it has the advantage of not causing gimbal lock unlike the case of calculating coordinate conversion using Euler angles. . However, the coordinate associating means 10 is not limited to calculating coordinate conversion using quaternions, and may use Euler angles to calculate coordinate conversion.

In addition, when it is possible to associate coordinates on a plurality of input image planes R4, the coordinate association means 10 converts the coordinates P(P') on the space model MD to the input image corresponding to the camera with the smallest incident angle α. It may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

Next, a process of reprojecting the coordinates on the curved surface region R2 (coordinates having a component in the Z-axis direction) among the coordinates on the space model MD onto the processing target image plane R3 on the XY plane will be described.

6A and 6B are diagrams for explaining the correspondence between coordinates by the coordinate correspondence means 10. FIG. 6A shows, as an example, the input image plane R4 of the camera 2 adopting the normal projection (h=ftanα). 2 is a diagram showing the correspondence relationship between the above coordinates and coordinates on the space model MD. Both coordinates are made to correspond so that each line segment connecting the two coordinates passes through the optical center C of the camera 2 .

In the example of FIG. 6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane region R1 of the space model MD, and the input image plane R4 of the camera 2 The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2. FIG.

Note that when the camera 2 adopts a projection method other than normal projection (for example, orthographic projection, stereographic projection, equisolid angle projection, equidistant projection, etc.), the coordinate correspondence means 10 uses Correspondingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are made to correspond to the coordinates L1 and L2 on the space model MD.

Specifically, the coordinate association means 10 uses a predetermined function (for example, orthogonal projection (h=f sinα), stereographic projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/2)), 2)), equidistant projection (h=fα), etc.), the coordinates on the input image plane and the coordinates on the space model MD are associated with each other. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2. FIG.

FIG. 6B is a diagram showing the correspondence relationship between the coordinates on the curved surface region R2 of the space model MD and the coordinates on the processing target image plane R3. and a group of parallel lines PL forming an angle β with the plane R3 of the image to be processed. Both the coordinates on the plane R3 are associated with one of the parallel line group PL.

In the example of FIG. 6B, the coordinate associating means 10 assumes that the coordinate L2 on the curved surface region R2 of the space model MD and the coordinate M2 on the processing target image plane R3 lie on a common parallel line, make it correspond.

Note that the coordinate associating means 10 can associate the coordinates on the planar region R1 of the space model MD with the coordinates on the processing target image plane R3 using the group of parallel lines PL in the same way as the coordinates on the curved surface region R2. However, in the example of FIG. 6B, since the plane region R1 and the processing target image plane R3 are a common plane, the coordinate L1 on the plane region R1 of the space model MD and the processing target image plane R3 has the same coordinate value as the coordinate M1 of .

In this manner, the coordinate associating means 10 associates the coordinates on the space model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the space model MD with the coordinates on the processing target image plane R3. are stored in the spatial model/image to be processed correspondence map 41 .

FIG. 6C is a diagram showing the correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting normal projection (h=ftanα) as an example. , the image generating means 11 determines that each line segment connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates passes through the optical center CV of the virtual camera 2V. In this way, both coordinates are associated with each other.

In the example of FIG. 6C, the image generating means 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane area R1 of the space model MD), The coordinates N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

Note that when the virtual camera 2V adopts a projection method other than normal projection (for example, orthographic projection, stereographic projection, equisolid angle projection, equidistant projection, etc.), the image generating means 11 is adapted to each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the image generation means 11 uses a predetermined function (for example, orthogonal projection (h=fsinα), stereographic projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/2 )), equidistant projection (h=fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

In this manner, the image generating means 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3. While referring to the input image/spatial model correspondence map 40 and the space model/processing target image correspondence map 41 stored in the processing target image/output image correspondence map 42 and stored by the coordinate association means 10, each An output image is generated by associating the pixel values with the values of each pixel in the input image.

FIG. 6(D) is a combination of FIGS. 6(A) to 6(C), and includes the camera 2, the virtual camera 2V, the planar region R1 and the curved surface region R2 of the space model MD, and the processing object. 3 shows the positional relationship of image planes R3 to each other.

Next, referring to FIG. 7, the action of the group of parallel lines PL introduced by the coordinate matching means 10 for matching the coordinates on the space model MD and the coordinates on the processing target image plane R3 will be described.

FIG. 7A is a diagram when an angle β is formed between the group of parallel lines PL positioned on the XZ plane and the processing target image plane R3, and FIG. It is a diagram when an angle β1 (β1>β) is formed between a group of parallel lines PL and an image plane R3 to be processed. Further, coordinates La to Ld on the curved surface region R2 of the space model MD in FIGS. 7A and 7B respectively correspond to coordinates Ma to Md on the processing target image plane R3, It is assumed that the intervals between the coordinates La to Ld in FIG. 7A are equal to the intervals between the coordinates La to Ld in FIG. 7B. Although the group of parallel lines PL is assumed to exist on the XZ plane for the purpose of explanation, in reality, it exists so as to radially extend from all points on the Z axis toward the processing target image plane R3. It shall be. Note that the Z-axis in this case will be referred to as a "reprojection axis".

As shown in FIGS. 7A and 7B, the intervals between the coordinates Ma to Md on the processing target image plane R3 are such that the angle between the group of parallel lines PL and the processing target image plane R3 is It decreases linearly as it increases (it decreases uniformly regardless of the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md). On the other hand, the coordinate group on the plane region R1 of the space model MD is not transformed into the coordinate group on the processing target image plane R3 in the example of FIG. 7, so the interval between the coordinate groups does not change. .

The change in the interval between these coordinate groups is such that, of the image portions on the output image plane R5 (see FIG. 6), only the image portion corresponding to the image projected onto the curved surface region R2 of the space model MD is linearly enlarged or means to be scaled down.

Next, with reference to FIG. 8, an alternative example of the group of parallel lines PL introduced by the coordinate associating means 10 for associating the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described.

FIG. 8A is a diagram in which all of the auxiliary lines AL positioned on the XZ plane extend from the starting point T1 on the Z axis toward the processing target image plane R3, and FIG. It is a diagram when all of the line group AL extends from the starting point T2 (T2>T1) on the Z axis toward the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the space model MD in FIGS. 8A and 8B corresponds to each of the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are outside the area of the processing target image plane R3), and the intervals between the coordinates La to Ld in FIG. 8(B) are equal to the respective intervals of the coordinates La to Ld. Although the auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, in reality, it exists so as to radially extend from an arbitrary point on the Z axis toward the processing target image plane R3. It shall be. As in FIG. 7, the Z-axis in this case will be referred to as the "reprojection axis".

As shown in FIGS. 8A and 8B, the intervals between the coordinates Ma to Md on the processing target image plane R3 are the distances (high ) increases non-linearly (the greater the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md, the greater the decrease in each interval). On the other hand, in the example of FIG. 8, the coordinate group on the plane region R1 of the space model MD is not transformed into the coordinate group on the processing target image plane R3, so the interval between the coordinate groups does not change. .

The change in the interval of these coordinate groups corresponds to the image projected onto the curved surface region R2 of the space model MD among the image portions on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image part is non-linearly scaled up or down.

In this way, the image generation device 100 can reproduce the spatial model MD without affecting the image portion (for example, the road surface image) of the output image corresponding to the image projected onto the planar region R1 of the spatial model MD. Since the image portion of the output image corresponding to the image projected onto the curved surface area R2 (for example, a horizontal image) can be linearly or non-linearly enlarged or reduced, the road surface image near the excavator 60 can be reduced. (virtual image when looking at the excavator 60 from directly above). Moreover, it is possible to flexibly expand or contract, and the visibility of the blind area of the shovel 60 can be improved.

Next, referring to FIG. 9, a process of generating an image to be processed by the image generating apparatus 100 (hereinafter referred to as "image to be processed generating process"), and an output image is generated using the generated image to be processed. The process of generating (hereinafter referred to as "output image generation process") will be described. FIG. 9 is a flow chart showing the flow of the process target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Also, it is assumed that the arrangement of the camera 2 (input image plane R4), the space model (the plane region R1 and the curved surface region R2), and the processing target image plane R3 is determined in advance.

First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD by means of the coordinate association means 10 (step S1).

Specifically, the coordinate association unit 10 acquires the angle formed between the group of parallel lines PL and the processing target image plane R3, and obtains the angle formed between the group of parallel lines PL extending from one coordinate on the processing target image plane R3. A point where one intersects with the curved surface region R2 of the space model MD is calculated, and the coordinates on the curved surface region R2 corresponding to the calculated point are converted to the coordinates on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived, and the corresponding relationship is stored in the space model/image to be processed correspondence map 41 . Note that the angle formed between the group of parallel lines PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It can be the value you enter.

Further, when one coordinate on the processing target image plane R3 matches one coordinate on the plane region R1 of the space model MD, the coordinate association unit 10 converts the one coordinate on the plane region R1 to the processing target image. One coordinate corresponding to that one coordinate on the plane R3 is derived, and the corresponding relationship is stored in the space model/image to be processed correspondence map 41 .

After that, the control unit 1 associates one coordinate on the space model MD derived by the above processing with the coordinate on the input image plane R4 by the coordinate association means 10 (step S2).

Specifically, the coordinate association means 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h=ftanα), is a line segment extending from one coordinate on the space model MD, and the optical center The point where the line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are calculated on the input image plane R4 corresponding to the one coordinate on the space model MD. , and the corresponding relationship is stored in the input image/spatial model correspondence map 40 .

After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 have been associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3). are not associated (NO in step S3), the processing in steps S1 and S2 is repeated.

On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 terminates the processing target image generation processing and starts the output image generation processing. , coordinates on the processing target image plane R3 and coordinates on the output image plane R5 (step S4).

Specifically, the image generation means 11 generates an output image by applying scale transformation, affine transformation, or distortion transformation to the image to be processed, and the content of the applied scale transformation, affine transformation, or distortion transformation determines The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image/output image correspondence map .

Alternatively, when generating the output image using the virtual camera 2V, the image generating means 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence may be stored in the processing target image/output image correspondence map 42 .

Alternatively, when the image generating means 11 generates an output image using a virtual camera 2V that employs normal projection (h=ftanα), after acquiring the coordinates of the optical center CV of the virtual camera 2V, the output Calculate the point where a line segment extending from one coordinate on the image plane R5 and passing through the optical center CV intersects the processing target image plane R3, and the coordinates on the processing target image plane R3 corresponding to the calculated point may be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the corresponding relationship may be stored in the processing target image/output image correspondence map .

After that, the control unit 1 causes the image generating means 11 to refer to the input image/spatial model correspondence map 40, the space model/processing target image correspondence map 41, and the processing target image/output image correspondence map 42, and generate the input image plane R4. Correspondence between the above coordinates and coordinates on the space model MD, correspondence between coordinates on the space model MD and coordinates on the processing target image plane R3, and coordinates on the processing target image plane R3 and on the output image plane R5 , and obtains the values (for example, luminance value, hue value, saturation value, etc.) of the coordinates on the input image plane R4 corresponding to the coordinates on the output image plane R5. , the obtained values are adopted as the values of the corresponding coordinates on the output image plane R5 (step S5). Note that when a plurality of coordinates on a plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generating means 11 calculates the coordinates of the plurality of input image planes R4. A statistic based on the values (e.g., mean, maximum, minimum, median, etc.) may be derived and the statistic taken as the value of that one coordinate on the output image plane R5. .

After that, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 have been associated with the coordinate values on the input image plane R4 (step S6). If it is determined not to be attached (NO in step S6), the processing of steps S4 and S5 is repeated.

On the other hand, when determining that all coordinate values have been associated (YES in step S6), the control unit 1 generates an output image and terminates this series of processing.

Note that the image generation apparatus 100 omits the processing target image generation processing when not generating the processing target image, and sets the “coordinates on the processing target image plane” in step S4 in the output image generation processing to “on the space model. shall be read as "coordinates".

With the above configuration, the image generation device 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the features around the excavator 60 and the excavator 60. .

In addition, the image generating apparatus 100 associates the coordinates from the processing target image plane R3 to the input image plane R4 via the space model MD, so that each coordinate on the processing target image plane R3 is mapped to the input image plane. One or a plurality of coordinates on R4 can be reliably matched, and compared to the case of executing coordinate matching in the order from the input image plane R4 to the processing target image plane R3 via the space model MD (in this case can ensure that each coordinate on the input image plane R4 corresponds to one or more coordinates on the processed image plane R3, but some of the coordinates on the processed image plane R3 correspond to the input image plane R4 In some cases, it may not be possible to correspond to any of the coordinates above, and in that case, it is necessary to perform interpolation processing etc. on some of the coordinates on the processing target image plane R3). Can be generated quickly.

Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generation device 100 changes the angle formed between the group of parallel lines PL and the processing target image plane R3. The desired enlargement or reduction can be realized without rewriting the contents of the input image/spatial model correspondence map 40 by rewriting only the portion related to the curved surface region R2 in the space model/process target image correspondence map 41. .

Further, in order to change the appearance of the output image, the image generating apparatus 100 only needs to change the values of various parameters related to scale transformation, affine transformation, or distortion transformation to rewrite the processing target image/output image correspondence map 42. , a desired output image (scale-transformed image, affine-transformed image, or distortion-transformed image) can be generated without rewriting the contents of the input image/spatial model correspondence map 40 and the spatial model/process target image correspondence map 41 .

Similarly, when the viewpoint of the output image is to be changed, the image generation apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image/output image correspondence map 42 so that the input image/space An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the spatial model/processing target image correspondence map 41 .

FIG. 10 is a display example when an output image generated using the input images of the two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

The image generation device 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and then re-projects them onto the processing target image plane R3 to generate the processing target image. Then, the generated image to be processed is subjected to image transformation processing (for example, scale transformation, affine transformation, distortion transformation, viewpoint transformation processing, etc.) to generate an output image, and the vicinity of the excavator 60 is viewed from above. An image looking down (an image in the plane region R1) and an image looking down on the surroundings in the horizontal direction from the excavator 60 (an image in the processing target image plane R3) are displayed simultaneously.

When the image generation device 100 does not generate the image to be processed, the output image is generated by subjecting the image projected onto the space model MD to image conversion processing (for example, viewpoint conversion processing). shall be

In addition, the output image is trimmed into a circle so that the image when the excavator 60 performs a turning motion can be displayed without discomfort. It is generated to be on the axis PV, and is displayed so as to rotate about its center CTR as the excavator 60 turns. In this case, the central axis of the cylinder of the space model MD may or may not coincide with the reprojection axis.

The radius of the space model MD is, for example, 5 meters, and the angle formed by the parallel line group PL with the processing target image plane R3 or the starting point height of the auxiliary line group AL is When there is a feature (for example, a worker) at a position away from the center by the maximum reaching distance (for example, 12 meters) of the excavation attachment E, the feature is sufficiently large on the display unit 5 (for example, , is greater than or equal to 7 millimeters.) can be set to be displayed.

Furthermore, the CG image of the excavator 60 may be arranged so that the front of the excavator 60 coincides with the top of the screen of the display unit 5 and the turning center coincides with the center CTR. This is to make the positional relationship between the shovel 60 and the feature appearing in the output image easier to understand. It should be noted that a frame image including various information such as orientation may be arranged around the output image.

Next, details of the output image generated by the image generating apparatus 100 will be described with reference to FIGS. 11 to 18. FIG.

FIG. 11 is a top view of excavator 60 on which image generating device 100 is mounted. In the embodiment shown in FIG. 11, the excavator 60 has three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three stereo cameras 6 (left side stereo camera 6L, right side stereo camera 6L). A camera 6R and a rear stereo camera 6B). Areas CL, CR, and CB indicated by dashed-dotted lines in FIG. 11 indicate imaging ranges of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate imaging ranges of the left stereo camera 6L, right stereo camera 6R, and rear stereo camera 6B, respectively.

In this embodiment, the imaging range of the stereo camera 6 is narrower than the imaging range of the camera 2, but the imaging range of the stereo camera 6 may be the same as the imaging range of the camera 2, or may be wider than the imaging range of the camera 2. good too. Also, the imaging range of the stereo camera 6 is positioned near the excavator 60 within the imaging range of the camera 2 , but may be located in a region farther from the excavator 60 .

FIG. 12 is a diagram showing input images of each of the three cameras 2 mounted on the excavator 60 and output images generated using these input images.

The image generation device 100 projects the input images of the three cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and then re-projects them onto the processing target image plane R3 to generate the processing target image. . Further, the image generation apparatus 100 generates an output image by performing image transformation processing (for example, scale transformation, affine transformation, distortion transformation, viewpoint transformation processing, etc.) on the generated image to be processed. As a result, the image generation device 100 generates an image looking down on the vicinity of the excavator 60 from the sky (image in the plane area R1) and an image looking down on the surroundings in the horizontal direction from the excavator 60 (image in the processing target image plane R3). display at the same time. Note that the image displayed in the center of the output image is the CG image 60CG of the excavator 60 .

In FIG. 12, the input image of the right side camera 2R and the input image of the rear camera 2B each capture a person within the overlapping area between the imaging range of the right side camera 2R and the imaging range of the rear camera 2B (right side camera 2R and rear camera 2B). See area R10 surrounded by a two-dot chain line in the input image of the front camera 2R and area R11 surrounded by a two-dot chain line in the input image of the rear camera 2B).

However, if the coordinates on the output image plane were to be mapped to the coordinates on the input image plane for the camera with the smallest angle of incidence, the output image would lose the person in the overlap region (one point in the output image). See region R12 surrounded by dashed lines.).

Therefore, in the output image portion corresponding to the overlapping region, the image generation device 100 associates the region to which the coordinates on the input image plane of the rear camera 2B are associated with the coordinates on the input image plane of the right side camera 2R. area and prevent objects in the overlapping area from disappearing.

FIG. 13 is a diagram for explaining image disappearance prevention processing for preventing the disappearance of an object in the overlapping area of the imaging ranges of the two cameras.

FIG. 13(A) is a diagram showing an output image portion corresponding to the overlapping area between the imaging range of the right side camera 2R and the imaging range of the rear camera 2B, which corresponds to the rectangular area R13 indicated by the dotted line in FIG.

In FIG. 13A, a region PR1 filled in gray is an image region in which the input image portion of the rear camera 2B is arranged, and each coordinate on the output image plane corresponding to the region PR1 is the rear camera 2B coordinates on the input image plane are associated.

On the other hand, the region PR2 filled in white is the image region where the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the region PR2 is the input image plane of the right side camera 2R. The upper coordinates are associated.

In this embodiment, the regions PR1 and PR2 are arranged to form a striped pattern, and the boundary line between the regions PR1 and PR2 alternately arranged in a striped pattern is centered on the turning center of the shovel 60. Defined by concentric circles on the horizontal plane.

FIG. 13B is a top view showing the situation of the space area obliquely to the right rear of the excavator 60, showing the current situation of the space area captured by both the rear camera 2B and the right side camera 2R. Also, FIG. 13B shows that a stick-shaped three-dimensional object OB exists obliquely behind the shovel 60 to the right.

FIG. 13(C) shows part of an output image generated based on an input image obtained by actually capturing the spatial region shown in FIG. 13(B) with the rear camera 2B and the right side camera 2R.

Specifically, the image OB1 is the image of the three-dimensional object OB in the input image of the rear camera 2B that is extended in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by viewpoint conversion for generating the road surface image. represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

The image OB2 is obtained by extending the image of the three-dimensional object OB in the input image of the right camera 2R in the extending direction of the line connecting the right camera 2R and the three-dimensional object OB by viewpoint conversion for generating the road surface image. represents something. That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the right side camera 2R.

In this way, the image generation device 100 creates a region PR1 in which the coordinates on the input image plane of the rear camera 2B are associated and a region PR2 in which the coordinates on the input image plane of the right camera 2R are associated in the overlap region. mix. As a result, the image generation device 100 displays both the two images OB1 and OB2 related to one three-dimensional object OB on the output image, thereby preventing the three-dimensional object OB from disappearing from the output image.

14A and 14B are comparison diagrams showing the difference between the output image in FIG. 12 and the output image obtained by applying the image loss prevention processing to the output image in FIG. 14B shows an output image after applying the image loss prevention process. The person disappears in the region R12 surrounded by the dashed line in FIG. 14A, while the person is displayed without disappearing in the region R14 surrounded by the dashed line in FIG. 14B.

Next, an output range image generated by the image generation device 100 will be described with reference to FIG. 15 . FIG. 15 is a diagram showing input range images of each of the three stereo cameras 6 mounted on the shovel 60 and output range images generated using these input range images.

The image generation device 100 projects the input range images of the three stereo cameras 6 onto the plane area R1 and the curved area R2 of the space model MD, and then reprojects them onto the range image plane R3 to be processed. Generate a range image. The image generating apparatus 100 also generates an output range image by performing image transformation processing (for example, scale transformation, affine transformation, distortion transformation, viewpoint transformation processing, etc.) on the generated processing target distance image. Then, the image generation device 100 generates a distance image (distance image in the plane area R1) looking down on the vicinity of the excavator 60 from the sky, and a distance image (image in the processing target distance image plane R3) when looking at the surroundings in the horizontal direction from the excavator 60. ) and are displayed at the same time. Note that the image displayed in the center of the output distance image is the CG image 60CG of the excavator 60 .

Note that the input range image and the output range image in FIG. 15 are displayed so that the smaller the pixel value (the distance from the stereo camera 6), the whiter (lighter) the pixel value, and the larger the pixel value, the darker the color (darker). Note that the pixel values (distances) of the portions corresponding to the road surface and the upper revolving body 63 are set to the maximum value (infinite distance) in order to improve the visibility of the output distance image of features existing around the excavator 60. be done. That is, the portions corresponding to the road surface and the upper rotating body 63 are shown in black.

In FIG. 15, the input distance image of the right stereo camera 6R and the input distance image of the rear stereo camera 6B are each an overlap region between the imaging range of the right stereo camera 6R and the imaging range of the rear stereo camera 6B. (see area R15 surrounded by a two-dot chain line in the input range image of the right stereo camera 6R and area R16 surrounded by a two-dot chain line in the input range image of the rear stereo camera 6B).

However, if the coordinates on the output range image plane are associated with the coordinates on the input range image plane for the closest stereo camera, the output range image will cause the person in the overlap region to disappear ( See region R17 surrounded by a dashed line in the output range image.).

Therefore, in the output range image portion corresponding to the overlap region, the image generation device 100, among the coordinates on the input range image plane of the rear stereo camera 6B and the coordinates on the input range image plane of the right stereo camera 6R, The coordinates with smaller pixel values (distances) are associated with the coordinates on the output range image plane. As a result, the image generation device 100 displays two distance images regarding one three-dimensional object on the output distance image, thereby preventing the three-dimensional object from disappearing from the output distance image. Note that, hereinafter, this process for preventing the disappearance of an object in the overlapping area of the imaging ranges of the two stereo cameras is referred to as distance image disappearance prevention process.

16A and 16B are comparison diagrams showing the difference between the output distance image of FIG. 15 and the output distance image obtained by applying the distance image disappearance prevention process to the output distance image of FIG. FIG. 15 shows the output image, and FIG. 16B shows the output distance image after applying the distance image disappearance prevention process. In the area R17 surrounded by the dashed line in FIG. 16A, the person disappears, while in the area R17 surrounded by the dashed line in FIG. 16B, the person is displayed without disappearing.

Next, referring to FIGS. 17 and 18, the process of synthesizing the output distance image with the output image by the image generating apparatus 100 (hereinafter referred to as "distance image synthesizing process") will be described.

FIG. 17 is a diagram showing an example of a synthesized output image obtained by synthesizing an output distance image with an output image.

For example, as shown in FIG. 16B, the distance image synthesizing means 14 of the image generating device 100 converts the pixel values (distance) of the extracted pixels extracted by the pixel extracting means 12 into luminance values, hue values, saturation values, and the like. , the extracted pixels are superimposed on the output image shown in FIG. 14(B). The superimposed and displayed extracted pixels have, for example, colors corresponding to the distance from the stereo camera 6, and as the distance increases, the color changes stepwise or steplessly, such as red, yellow, green, and blue. change. It should be noted that the superimposed and displayed extracted pixels may have brightness corresponding to the distance from the stereo camera 6, and the brightness may be decreased stepwise or steplessly as the distance increases. Areas EX1 to EX5 in FIG. 17 are areas formed by these extracted pixels, and are hereinafter referred to as feature areas.

Further, the distance image synthesizing means 14 may superimpose the distance information of the feature areas EX1 to EX5 formed by a predetermined number or more of adjacent extracted pixels on the output image. Specifically, the distance image synthesizing means 14 calculates, for example, the minimum value, the maximum value, the average value, the median value, etc. of the pixel values (distances) of the pixels forming the feature area EX1 as the distance to the feature area EX1. is superimposed on the output image as a representative value representing . Note that the distance image synthesizing means 14 may superimpose the representative value on the output image only when the representative value is less than a predetermined value. This is to facilitate recognition of features relatively close to the shovel 60, that is, features with a relatively high possibility of contact.

FIG. 18 is a diagram showing another example of a synthesized output image obtained by synthesizing an output distance image with an output image.

In FIG. 18, the range image synthesizing means 14 superimposes frames RF1 to RF5 corresponding to the feature areas EX1 to EX5 in FIG. 17, instead of changing the color and brightness of the extracted pixels and superimposing them. The line type of the frame is arbitrary including dotted line, solid line, dashed line, dashed line, etc., and the shape of the frame is also arbitrary including rectangle, circle, ellipse and polygon.

Further, the distance image synthesizing means 14 may superimpose and display the distance information corresponding to each of the frames RF1 to RF5 on the output image. Specifically, the distance image synthesizing means 14, for example, calculates the minimum value, maximum value, average value, median value, etc. of the pixel values (distances) of the pixels forming the feature area EX1 corresponding to the frame RF1. is superimposed on the output image as a representative value of the distance corresponding to . Note that the distance image synthesizing means 14 may superimpose the representative value on the output image only when the representative value is less than a predetermined value. This is to facilitate recognition of features relatively close to the shovel 60, that is, features with a relatively high possibility of contact.

Further, the distance image synthesizing means 14 may change the color of the frame according to the corresponding distance information. Specifically, the distance image synthesizing means 14 changes the color of the frame stepwise or steplessly, such as red, yellow, green, and blue, as the distance increases. Note that the distance image synthesizing means 14 may reduce the brightness of the frame stepwise or steplessly as the distance increases.

The operator of the excavator 60 can more easily recognize the position of the feature existing around the excavator 60 and the distance to the feature by viewing the post-composite output image as described above. In addition, the operator of the shovel 60 can more easily recognize the position of a feature existing in a blind spot for the operator and the distance to the feature.

With the above configuration, the image generation device 100 synthesizes the output distance image generated based on the input distance image captured by the stereo camera 6 on the output image generated based on the input image captured by the camera 2. . Therefore, the image generating device 100 can measure the distance to each feature existing in a relatively wide range around the excavator 60 with relatively high resolution. As a result, the image generating apparatus 100 can generate a post-combination output image with high reliability.

Also, the camera 2 and the stereo camera 6 are arranged so that their imaging ranges surround the shovel 60 . Therefore, the image generation device 100 can generate a post-combination output image that shows the surroundings of the excavator 60 when viewed from above. As a result, the operator of the excavator 60 can more easily recognize the position of the feature existing around the excavator 60 and the distance to the feature.

In addition, the image generation device 100 performs the same image conversion processing as the image conversion processing performed on the processing target image on the processing target range image. Therefore, the image generation device 100 can easily associate the coordinates on the output image plane with the coordinates on the output range image plane.

In addition, the image generation device 100 changes the color of the extracted pixels or the frame stepwise or steplessly according to the distance from the stereo camera 6, that is, according to the distance from the shovel 60. FIG. Therefore, the operator of the excavator 60 can intuitively determine whether or not there is a risk of contact.

The image generation device 100 also acquires distance information in a two-dimensional array using the stereo camera 6 . Therefore, the image generating apparatus 100 uses a heat source, a light source, a light-absorbing material, etc. in the imaging range, compared to the case of acquiring distance information in a two-dimensional array using a distance image sensor configured by a light-projecting LED and a light-receiving CCD. less likely to be affected by

Next, referring to FIGS. 19 to 23, the image generating apparatus 100 extracts pixels satisfying a predetermined condition from the input range image and corrects the input range image (hereinafter referred to as “first pixel extraction processing”). ) will be explained. In this embodiment, it is assumed that the pixel value in the input range image captured by the rear stereo camera 6B is equal to or smaller than the installation plane distance.

FIG. 19 is a partial right side view of excavator 60 on which image generating device 100 is mounted. In the embodiment shown in FIG. 19, the excavator 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). A region ZB indicated by a dotted line in FIG. 19 indicates the imaging range of the rear stereo camera 6B.

FIG. 19 also shows that behind the excavator 60 there is an object P3 extending vertically upward from the ground and a protuberance P4 on the ground. Furthermore, FIG. 19 shows that the rear stereo camera 6B has detected the distance E1a to the object P3 and the distance E2a to the protuberance P4.

FIG. 19 also shows that if the object P3 and the raised portion P4 did not exist, the rear stereo camera 6B would have detected the installation surface distances E1 and E2 instead of the distances E1a and E2a. .

FIG. 19 also shows that the difference (E1-E1a) between the installation surface distance E1 and the distance E1a is the distance (hereinafter, the difference when the installation surface distance E1 is greater than the distance E1a is referred to as the "first differential distance") E1b. , indicating that the difference (E2-E2a) between the installation surface distance E2 and the distance E2a is the first differential distance E2b.

20 is an enlarged view of the raised portion P4 in FIG. 19. FIG. As shown in FIGS. 19 and 20, a line segment connecting a point on the raised portion P4 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle with the installation surface (hereinafter, "installation surface angle"). ) to form γ4. In this case, the height (hereinafter referred to as "estimated height") H1b of the raised portion P4 from the installation surface is represented by the first differential distance E2b×sin(γ4). The installation plane angle is set in advance for each pixel in the input range image of the rear stereo camera 6B based on the installation position and installation angle of the rear stereo camera 6B, similar to the installation plane distance.

Also, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, by multiplying the first differential distance, which is the difference between the value of each pixel in the input range image and the installation surface distance corresponding to each pixel, by the sine of the installation surface angle corresponding to each pixel, the An estimated height can be obtained. In this embodiment, the estimated height H1b of the raised portion P4 can be obtained by multiplying the first differential distance E2b by sin(γ4).

By the way, the pixel value in the input range image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B, as described above. Therefore, when the output range image is generated by applying the process of changing the color or brightness according to the distance from the rear stereo camera 6B to the input range image, the visibility of the output range image can be improved by using the pixel values in the input range image as they are. may be damaged. Specifically, the output distance image represents the ground (installation surface) with a color or brightness that changes stepwise as the distance from the shovel 60 increases, which may make it difficult to recognize the existence of the object P3 and the raised portion P4. .

Therefore, the pixel extracting means 12 of the image generation device 100 extracts, for example, pixels satisfying a predetermined condition from the input range image of the rear stereo camera 6B, and corrects the input range image based on the extraction result. Specifically, the pixel extracting means 12 extracts pixels having a pixel value smaller than the installation surface distance corresponding to each pixel among the pixels in the input distance image. Then, the pixel extracting means 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction unit 12 maintains the pixel values of pixels having pixel values smaller than the installation surface distance. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

Further, the pixel extracting means 12 may extract pixels whose first difference distance is equal to or greater than a predetermined distance among the pixels in the input distance image. In this case, the pixel extraction unit 12 replaces the pixel values of the pixels whose first difference distance is less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and replaces the pixel values of the pixels whose first difference distance is greater than or equal to the predetermined distance pixel values are kept as they are. For example, as shown in FIG. 20, the pixel extraction means 12 sets the pixel value of the pixel corresponding to the first difference distance E2b to zero because the first difference distance E2b is less than the predetermined distance E2b1. Note that the predetermined distance E2b1 is a distance corresponding to the predetermined height H1. Alternatively, as shown in FIG. 20, the pixel extracting means 12 may determine that the installation surface height H1b is less than the predetermined height H1, and replace the pixel value of the pixel corresponding to the installation surface height H1b with zero. good. Alternatively, the pixel extraction unit 12 determines that the ratio of the first difference distance E2b to the installation surface distance E2 (hereinafter, the ratio when the installation surface distance E2 is greater than the distance E2a is referred to as the "first difference ratio") is a predetermined ratio. may be determined to be less than and the pixel value of the corresponding pixel may be replaced with zero. The predetermined distance, predetermined height, predetermined ratio, etc. are set in advance for each pixel, pixel row, or pixel column in the input distance image of the rear stereo camera 6B based on the installation position and installation angle of the rear stereo camera 6B. be done. Also, the predetermined height and the predetermined ratio may be common for all pixels in the input range image.

FIG. 21 is a diagram showing an example of an output range image of the rear stereo camera 6B. FIG. 21(A) shows an output range image when the pixel values of all pixels in the input range image are used as they are without extraction by the pixel extracting means 12 . FIG. 21(B) shows an output range image when pixels having pixel values smaller than the installation surface distance corresponding to each pixel are extracted from the pixels in the input range image by the pixel extracting means 12 . In FIG. 21C, the pixel extracting means 12 detects, among the pixels in the input range image, the pixels whose first difference distance is equal to or greater than a predetermined distance, or the pixels whose estimated height is equal to or greater than a predetermined height. 2 shows the output range image when extracted.

As shown in FIG. 21A, when pixel extraction is not performed, the output range image represents the installation surface in a manner in which the luminance decreases stepwise as the distance from the shovel 60 increases, and the object P3 and the protuberance P4 obscure its existence.

On the other hand, as shown in FIG. 21B, when extracting pixels having pixel values smaller than the installation plane distance among the pixels in the input range image, the output range image is Improve visibility. Specifically, in the output distance image of FIG. 21B, the pixel values of all pixels having pixel values equal to the installation surface distance are replaced with zero. Uniform color and brightness. As a result, in the output distance image of FIG. 21(B), the distance image of the object P3 and the protuberance P4 can be distinguished from the distance image of the installation surface, and the visibility can be improved.

Furthermore, as shown in FIG. 21(C), when extracting pixels whose first difference distance is equal to or greater than a predetermined distance among the pixels in the input range image, the output range image has a predetermined height of the installation surface. Improve the visibility of the above features. Specifically, in the output distance image of FIG. 21C, the pixel values of all pixels whose first difference distance is less than the predetermined distance are set to zero. To uniform the color and brightness of features (including installation surfaces) whose height is less than a predetermined height. As a result, in the output range image of FIG. 21C, the range image of the ridge P4 whose installation surface height H1b is less than the predetermined height H1 is combined with the range image of the installation surface into a single color and brightness. Then, in the output distance image of FIG. 21C, only the distance image of the object P3 whose installation surface height is equal to or higher than the predetermined height H1 can be distinguished from the other distance images, and the visibility can be improved.

Note that the output distance image in FIG. 21C is obtained by setting the pixel value of the pixels corresponding to the root portion of the object P3 (the portion where the height of the installation surface is less than the predetermined height H1) to zero and setting the distance image of that portion. Make it a single color and intensity with the range image of the surface. However, even if the first difference distance is less than the predetermined distance or the installation surface height is less than the predetermined height H1, the pixels corresponding to the portion that is determined to be a part of the object P3 are Pixel values in the input range image may be maintained as they are. Whether or not a particular pixel belongs to a part of the range image of the object P3 is determined, for example, by the first differential distance or the installation surface height of the surrounding pixels and the first differential distance or the installation surface height of the specific pixel. can be determined based on

FIG. 22 is a diagram showing an example of an output image of the rear camera 2B. FIG. 22(A) shows an output image when the output range image of FIG. 21(A) is superimposed. FIG. 22(B) shows an output image when the output range image of FIG. 21(B) is superimposed. FIG. 22(C) shows an output image when the output range image of FIG. 21(C) is superimposed.

As shown in FIG. 22A, the output image superimposed with the output distance image of FIG. The presence of P4 cannot be emphasized.

On the other hand, as shown in FIG. 22B, the output image obtained by superimposing the output distance image in which the color and brightness of the installation surface are uniform regardless of the distance from the shovel 60 is the image of the object P3 and the protuberance P4. To stand out from the image of the installation surface and improve its visibility. Also, the output image of FIG. 22(B) superimposes the value of the distance between each of the object P3 and the raised portion P4 and the rear stereo camera 6B. This makes it possible to further emphasize the presence of the object P3 and the protuberance P4. For the value of the distance between the object P3 and the rear stereo camera 6B, for example, a value corrected by the distance correction means 13 is adopted. Also, the value of the distance between the object P3 and the rear stereo camera 6B may be superimposed and displayed only when it is equal to or less than a predetermined value. This is to make the presence of the feature near the excavator 60 more conspicuous. Also, the brightness or color of the value of the distance (pixel value) is determined according to the magnitude of the distance (pixel value). For example, the color of the distance (pixel value) value changes from purple to blue to green to yellow to red as the distance (pixel value) decreases. Further, the value of the distance may be the minimum value among the pixel values of the pixels included in the distance image area of the object P3 in the input distance image extracted by the pixel extraction means 12 . The same applies to the value of the distance between the raised portion P4 and the rear stereo camera 6B.

Furthermore, as shown in FIG. 22(C), regardless of the distance from the excavator 60, the output distance in which the color and brightness of features (including the installation surface) whose installation surface height is less than a predetermined height are made uniform The output image on which the image is superimposed improves the visibility of the object P3. Specifically, in the output image, only the upper portion of the image of the object P3 whose installation surface height is greater than or equal to a predetermined height is made stand out from the other images, thereby improving its visibility. Also, in the output image of FIG. 22(C), only the value of the distance between the object P3 whose installation surface height is equal to or higher than the predetermined height and the rear stereo camera 6B is superimposed and displayed. This makes it possible to further highlight only the presence of the object P3 whose installation surface height is equal to or higher than the predetermined height.

Further, in the above description, the image generation device 100 superimposes the output distance image of the rear stereo camera 6B on the output image of the rear camera 2B and displays the superimposed image on the display unit 5 . However, the image generation device 100 may cause the display unit 5 to display only the output distance image, or may cause the display unit 5 to display the output distance image and the output image side by side.

18 instead of or in addition to displaying the distance values between each of the object P3 and the protuberance P4 and the rear stereo camera 6B. A frame like this may be displayed.

In addition, the image generating apparatus 100 displays an index (not shown) such as a bar-shaped indicator with a distance scale representing the relationship between distance (pixel value) and color or brightness at the edge of the output distance image or the output image. may

Next, the flow of the first pixel extraction process will be described with reference to FIG. Note that FIG. 23 is a flowchart showing the flow of the first pixel extraction process. The pixel extraction means 12 of the image generation device 100 executes this first pixel extraction process, for example, each time an input range image is acquired.

First, the pixel extracting means 12 reads and acquires the installation surface distance for one pixel (hereinafter referred to as "target pixel") in the input distance image from the ROM or the like (step S11). Note that the pixel extracting means 12 may read and acquire the installation surface angle from a ROM or the like.

After that, the pixel extraction means 12 calculates the first difference distance by subtracting the pixel value of the target pixel from the installation surface distance (step S12). Note that the pixel extraction unit 12 may calculate the first difference ratio based on the calculated first difference distance and the installation surface distance, and the estimated height based on the calculated first difference distance and installation surface angle. may be calculated.

After that, the pixel extracting means 12 compares the calculated first difference distance with the predetermined distance stored in the ROM or the like (step S13). The pixel extracting means 12 may compare the calculated first difference ratio with a predetermined ratio stored in the ROM or the like, or may compare the calculated estimated height with the predetermined height stored in the ROM or the like. good.

If the first difference distance is equal to or greater than the predetermined distance (YES in step S13), the pixel extraction unit 12 extracts the target pixel as a pixel that maintains the current pixel value. The same applies when the first difference ratio is equal to or greater than a predetermined ratio, or when the estimated height is equal to or greater than a predetermined height.

On the other hand, if the first difference distance is less than the predetermined distance (NO in step S13), the pixel extraction unit 12 does not extract the target pixel as a pixel that maintains the current pixel value. Specifically, the pixel extraction unit 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S14). The same applies when the first difference ratio is less than the predetermined ratio or when the estimated height is less than the predetermined height.

In the above-described embodiment, the process of replacing the pixel values of target pixels that have not been extracted with a predetermined value is performed when it is determined whether extraction is necessary for each target pixel. It may be performed all at once after judging the necessity of extraction.

After that, the pixel extracting means 12 determines whether or not all the pixels in the acquired input distance image have been determined as to whether or not they need to be extracted (step S15).

If it is determined that all pixels have not yet been determined as to whether or not they need to be extracted (NO in step S15), the pixel extracting means 12 performs the processes from step S11 onward for another target pixel.

If it is determined that all pixels have been determined as to whether or not they need to be extracted (YES in step S15), the pixel extracting means 12 completes the current first pixel extraction process.

With the above configuration, the image generation device 100 can improve the visibility of the distance image related to the feature that the shovel 60 may come into contact with by omitting the detailed display of the distance image related to the installation surface.

In addition, the image generation apparatus 100 omits the detailed display of features whose installation surface height is less than a predetermined height, thereby increasing the visibility of features (for example, workers) whose installation surface height is greater than or equal to a predetermined height. can be improved.

As a result, the image generating device 100 can more reliably prevent the excavator 60 from colliding with the feature.

Next, with reference to FIGS. 24 to 28, another example of processing in which the image generating apparatus 100 extracts pixels satisfying a predetermined condition from the input range image and corrects the input range image (hereinafter referred to as "second pixels"). “extraction processing”) will be described. In this embodiment, it is assumed that the pixel value in the input distance image captured by the rear stereo camera 6B is equal to or greater than the installation plane distance.

FIG. 24 is a partial right side view of excavator 60 on which image generating device 100 is mounted, and corresponds to FIG. In the embodiment shown in FIG. 24, the excavator 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). A region ZB indicated by a dotted line in FIG. 24 indicates the imaging range of the rear stereo camera 6B.

FIG. 24 also shows that behind the excavator 60 there are a shallow depression PK1 and a deep depression (cliff) PK2 in the ground. Further, FIG. 24 shows that the rear stereo camera 6B has detected the distance E3a to the bottom of the shallow depression PK1 and the detection of the distance E4a to the bottom of the deep depression PK2.

Further, in FIG. 24, if the shallow recessed portion PK1 and the deep recessed portion PK2 did not exist, the rear stereo camera 6B would detect the installation surface distances E3 and E4 instead of the distances E3a and E4a. indicates that

FIG. 24 also shows that the difference (E3a-E3) between the distance E3a and the installation surface distance E3 is the distance (hereinafter, the difference when the installation surface distance E3 is smaller than the distance E3a is referred to as the "second differential distance") E3b. , indicating that the difference (E4a-E4) between the distance E4a and the installation surface distance E4 is the second difference distance E4b.

25 is an enlarged view of the shallow depression PK1 in FIG. 24. FIG. As shown in FIGS. 24 and 25, a line segment connecting the bottom point of the shallow depression PK1 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an installation surface angle γ5 with the installation surface. . In this case, the depth (hereinafter referred to as "estimated depth") DP1b from the installation surface of the shallow depression PK1 is represented by the second differential distance E3b×sin(γ5).

Also, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, by multiplying the second difference distance, which is the difference between the value of each pixel in the input range image and the installation plane distance corresponding to each pixel, by the sine of the installation plane angle corresponding to each pixel, the An estimated depth can be determined. In this embodiment, the estimated depth DP1b of the shallow depression PK1 can be obtained by multiplying the second differential distance E3b by sin(γ5).

By the way, the pixel value in the input range image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B, as described above. Therefore, when the output range image is generated by applying the process of changing the color or brightness according to the distance from the rear stereo camera 6B to the input range image, the visibility of the output range image can be improved by using the pixel values in the input range image as they are. may be damaged. Specifically, the output distance image represents the ground (installation surface) with a color or brightness that changes stepwise as the distance from the excavator 60 increases, making it difficult to recognize the presence of the shallow depression PK1 and the deep depression PK2. There is a risk.

Therefore, the pixel extracting means 12 of the image generation device 100 extracts, for example, pixels satisfying a predetermined condition from the input range image of the rear stereo camera 6B, and corrects the input range image based on the extraction result. Specifically, the pixel extracting means 12 extracts pixels having a pixel value greater than the installation surface distance corresponding to each pixel among the pixels in the input distance image. Then, the pixel extracting means 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction unit 12 maintains the pixel values of pixels having pixel values greater than the installation surface distance. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

Further, the pixel extracting means 12 may extract pixels whose second difference distance is equal to or greater than a predetermined distance among the pixels in the input distance image. In this case, the pixel extraction unit 12 replaces the pixel values of the pixels whose second difference distance is less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and replaces the pixel values of the pixels whose second difference distance is greater than or equal to the predetermined distance pixel values are kept as they are. For example, as shown in FIG. 25, the pixel extraction means 12 sets the pixel value of the pixel corresponding to the second difference distance E3b to zero because the second difference distance E3b is less than the predetermined distance E3b1. Note that the predetermined distance E3b1 is a distance corresponding to the predetermined depth DP1. Alternatively, as shown in FIG. 25, the pixel extraction means 12 may determine that the installation surface depth DP1b is less than the predetermined depth DP1, and replace the pixel value of the pixel corresponding to the installation surface depth DP1b with zero. good. Alternatively, the pixel extracting unit 12 determines that the ratio of the second difference distance E3b to the distance E3a (hereinafter, the ratio when the installation surface distance E3 is smaller than the distance E3a is referred to as the "second difference ratio") is less than a predetermined ratio. It may be determined that there is, and the pixel value of the corresponding pixel may be replaced with zero. The predetermined distance, predetermined depth, predetermined ratio, etc. are set in advance for each pixel, pixel row, or pixel column in the input distance image of the rear stereo camera 6B based on the installation position and installation angle of the rear stereo camera 6B. be done. Also, the predetermined depth and the predetermined ratio may be common to all pixels in the input range image.

FIG. 26 is a diagram showing an example of an output range image of the rear stereo camera 6B. FIG. 26(A) shows an output range image when the pixel values of all pixels in the input range image are used as they are without extraction by the pixel extracting means 12 . FIG. 26(B) shows an output range image when pixels having pixel values larger than the installation surface distance corresponding to each pixel are extracted from the pixels in the input range image by the pixel extracting means 12 . FIG. 26(C) shows that, among the pixels in the input range image, the pixel extracting means 12 detects pixels whose second differential distance is equal to or greater than a predetermined distance, or pixels whose estimated depth is equal to or greater than a predetermined depth. 2 shows the output range image when extracted.

As shown in FIG. 26(A), when pixels are not extracted, the output range image represents the installation surface in such a manner that the luminance decreases stepwise as the distance from the shovel 60 increases, and the shallow depression PK1 and the deep depression PK1 are displayed. It obscures the existence of part PK2.

On the other hand, as shown in FIG. 26B, when extracting pixels having a pixel value larger than the installation plane distance from among the pixels in the input range image, the output range image is a recessed portion existing under the installation plane. Improve visibility. Specifically, in the output distance image of FIG. 26B, the pixel values of all pixels having pixel values equal to the installation surface distance are replaced with zero. Uniform color and brightness. As a result, in the output distance image of FIG. 26(B), the distance images of the shallow depression PK1 and the deep depression PK2 can be distinguished from the distance image of the installation surface, and visibility can be improved.

Furthermore, as shown in FIG. 26(C), when extracting pixels whose second difference distance is equal to or greater than a predetermined distance among the pixels in the input range image, the output range image has an installation surface depth equal to or greater than the predetermined depth. Improve the visibility of the above features. Specifically, in the output distance image of FIG. 26C, the pixel values of all pixels whose second difference distance is less than the predetermined distance are set to zero. To uniform the color and brightness of features (including installation surfaces) whose depth is less than a predetermined depth. As a result, in the output range image of FIG. 26(C), the range image of the shallow depression PK1 whose installation surface depth DP1b is less than the predetermined depth DP1 is combined with the range image of the installation surface into a single color and brightness. In the output range image of FIG. 26(C), only the range image of the deep recessed portion PK2 whose installation surface depth is equal to or greater than the predetermined depth DP1 is distinguished from the other range images, and the visibility can be improved. can.

Note that the output distance image of FIG. 26(C) is obtained by setting the pixel values of the pixels corresponding to the edge portion of the deep recessed portion PK2 (the portion where the installation surface depth is less than the predetermined depth DP1) to zero. to a single color and intensity with the range image of the installation surface. However, a pixel corresponding to a portion determined to be part of the deep recessed portion PK2 will be detected even if the second difference distance is less than the predetermined distance, or even if the installation surface depth is less than the predetermined depth DP1. Alternatively, the pixel values in the input range image may be maintained as they are. Whether or not a particular pixel belongs to a part of the range image of the deep recessed portion PK2 can be determined, for example, by the second differential distance or installation surface depth for the surrounding pixels and the second differential distance or installation surface for the specific pixel. can be determined based on depth.

FIG. 27 is a diagram showing an example of an output image of the rear camera 2B. FIG. 27(A) shows an output image when the output range image of FIG. 26(A) is superimposed. FIG. 27(B) shows an output image when the output range image of FIG. 26(B) is superimposed. Also, FIG. 27(C) shows an output image when the output range image of FIG. 26(C) is superimposed.

As shown in FIG. 27A, the output image superimposed with the output distance image of FIG. The presence of the deep depression PK2 cannot be emphasized.

On the other hand, as shown in FIG. 27B, the output image superimposed with the output distance image in which the color and luminance of the installation surface are uniform regardless of the distance from the shovel 60 is a shallow depression portion PK1 and a deep depression portion PK2. To make the image of the image stand out from the image of the installation surface and improve its visibility. In addition, the output image of FIG. 27(B) superimposes the value of the distance between each of the shallow recessed portion PK1 and the deep recessed portion PK2 and the rear stereo camera 6B. This makes it possible to further emphasize the presence of the shallow recessed portion PK1 and the deep recessed portion PK2. For the value of the distance between the shallow depression PK1 and the rear stereo camera 6B, for example, a value corrected by the distance correction means 13 is adopted. Also, the value of the distance between the shallow depression PK1 and the rear stereo camera 6B may be superimposed and displayed only when it is less than a predetermined value. This is to further emphasize the existence of the depressed portion near the shovel 60 . Also, the brightness or color of the value of the distance (pixel value) is determined according to the magnitude of the distance (pixel value). For example, the color of the distance (pixel value) value changes from purple to blue to green to yellow to red as the distance (pixel value) decreases. Moreover, the value of the distance may be the minimum value among the pixel values of the pixels included in the range image region of the shallow depression PK1 in the input range image extracted by the pixel extraction means 12 . The same applies to the value of the distance between the deep depression PK2 and the rear stereo camera 6B.

Furthermore, as shown in FIG. 27(C), regardless of the distance from the excavator 60, the output distance in which the color and brightness of features (including the installation surface) whose installation surface depth is less than a predetermined depth are made uniform The output image on which the image is superimposed improves the visibility of the deep depression PK2. Specifically, in the output image, only the image of the deep recessed portion PK2 whose installation surface depth is greater than or equal to a predetermined depth is distinguished from the other images, thereby improving the visibility. Also, the output image of FIG. 27(C) superimposes only the value of the distance between the deep recessed portion PK2 whose installation surface depth is equal to or greater than a predetermined depth and the rear stereo camera 6B. As a result, it is possible to further highlight only the existence of the deep recessed portion PK2 whose installation surface depth is equal to or greater than the predetermined depth.

Further, in the above description, the image generation device 100 superimposes the output distance image of the rear stereo camera 6B on the output image of the rear camera 2B and displays the superimposed image on the display unit 5 . However, the image generation device 100 may cause the display unit 5 to display only the output distance image, or may cause the display unit 5 to display the output distance image and the output image side by side.

In addition, instead of displaying the distance values between each of the shallow recessed portion PK1 and the deep recessed portion PK2 and the rear stereo camera 6B, or in addition to displaying the distance values, the image generation device 100 may A frame as shown in 18 may be displayed.

Next, the flow of the second pixel extraction process will be described with reference to FIG. Note that FIG. 28 is a flowchart showing the flow of the second pixel extraction process. The pixel extracting means 12 of the image generating device 100 executes this second pixel extracting process, for example, each time an input range image is acquired.

First, the pixel extracting means 12 reads and acquires the installation surface distance for one pixel (hereinafter referred to as "target pixel") in the input distance image from the ROM or the like (step S21). Note that the pixel extracting means 12 may read and acquire the installation surface angle from a ROM or the like.

After that, the pixel extracting means 12 calculates a second differential distance by subtracting the installation surface distance from the pixel value of the target pixel (step S22). Note that the pixel extracting means 12 may calculate the second difference ratio based on the calculated second difference distance and the pixel value of the target pixel. An estimated depth may be calculated.

After that, the pixel extracting means 12 compares the calculated second difference distance with the predetermined distance stored in the ROM or the like (step S23). The pixel extracting means 12 may compare the calculated second difference ratio with a predetermined ratio stored in the ROM or the like, or compare the calculated estimated depth with the predetermined depth stored in the ROM or the like. good.

If the second difference distance is equal to or greater than the predetermined distance (YES in step S23), the pixel extraction unit 12 extracts the target pixel as a pixel that maintains the current pixel value. The same applies when the second difference ratio is equal to or greater than the predetermined ratio, or when the estimated depth is equal to or greater than the predetermined depth.

On the other hand, if the second difference distance is less than the predetermined distance (NO in step S23), the pixel extraction unit 12 does not extract the target pixel as a pixel that maintains the current pixel value. Specifically, the pixel extraction unit 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S24). The same applies when the second difference ratio is less than the predetermined ratio or when the estimated depth is less than the predetermined depth.

Note that the process of replacing the pixel values of target pixels that have not been extracted with a predetermined value may be performed collectively after determining whether extraction is necessary for a plurality of or all target pixels.

After that, the pixel extracting means 12 determines whether or not all the pixels in the acquired input range image have been determined as to whether or not they need to be extracted (step S25).

If it is determined that all pixels have not yet been determined as to whether or not they need to be extracted (NO in step S25), the pixel extracting means 12 performs the processes from step S21 onward for another target pixel.

If it is determined that the extraction necessity has been determined for all pixels (YES in step S25), the pixel extracting means 12 completes the current second pixel extraction process.

With the above configuration, the image generating apparatus 100 omits the detailed display of the distance image related to the installation surface, so that the depth image of the recessed portion (for example, the road shoulder, the cliff, etc.) that may cause the excavator 60 to slide down. Visibility can be improved.

In addition, the image generation device 100 can improve visibility of recessed portions whose installation surface depth is greater than or equal to a predetermined depth by omitting detailed display of recessed portions whose installation surface depth is less than a predetermined depth. .

As a result, the image generating device 100 can more reliably prevent the excavator 60 from sliding down into the depression.

Also, in the above description, the first pixel extraction process and the second pixel extraction process are described separately, but the two pixel extraction processes may be combined.

Next, with reference to FIG. 29, a process for executing the driving support function by the driving support means 15 (hereinafter referred to as "driving support process") will be described. Note that FIG. 29 is a flowchart showing the flow of the driving assistance process, and the driving assistance means 15 repeatedly executes this driving assistance process at predetermined intervals while the excavator 60 is running.

First, the driving support means 15 detects the moving direction of the excavator 60 based on the operating direction of the travel control lever (not shown) of the excavator 60 (step S31).

After that, the driving support means 15 determines whether or not a predetermined feature exists in the moving direction of the shovel 60 (step S32). In the present embodiment, the driving support means 15 determines whether or not there is a depression behind the retreating excavator 60, which may cause the excavator 60 to slide down, based on the input distance image acquired by the rear stereo camera 6B. judge.

When it is determined that there is a depression that may cause the shovel 60 to slide down (YES in step S32), the driving assistance means 15 executes the driving assistance function (step S33). In the present embodiment, the driving support means 15 sounds an alarm and displays a warning message on the display unit 5 to notify that there is a depression. Further, the driving support means 15 may reduce the retreating speed of the excavator 60 as the distance to the depression becomes shorter, and may stop the excavator 60 when the distance to the depression is below a predetermined value. .

On the other hand, when it is determined that there is no depression that may cause the excavator 60 to slide down (NO in step S32), the driving assistance means 15 ends the current driving assistance process without executing the driving assistance function.

With the above configuration, the image generation device 100 automatically determines the presence or absence of a predetermined feature based on the output of the stereo camera 6, and executes the driving support function as necessary, thereby improving the operation of the excavator 60. Safety can be further improved.

Next, referring to FIGS. 30 to 33, the process of correcting the pixel value (distance) of the input distance image by the image generating apparatus 100 (hereinafter referred to as "distance correction process") will be described.

30A and 30B are diagrams showing a shovel 60 on which the image generation device 100 is mounted, FIG. 30A being a top view thereof, and FIG. 30B being a partial right side view thereof. In the embodiment shown in FIG. 30, the excavator 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). A region CB indicated by a dashed dotted line in FIG. 30 indicates the imaging range of the rear camera 2B. A region ZB indicated by a dotted line in FIG. 30 indicates the imaging range of the rear stereo camera 6B.

As shown in FIG. 30, behind the shovel 60, there are two objects P1 and P2 extending vertically upward from the ground. The two objects P1 and P2 are rod-shaped objects with different heights and exist in different directions when viewed from the rear stereo camera 6B. Specifically, as shown in FIG. 30B, the height of the object P1 is lower than the height of the object P2. However, the shortest distances between each of the two objects P1 and P2 and the rear stereo camera 6B are equal to the distance D. This is because the lower object P1 exists closer to the excavator 60 than the higher object P2.

Further, as shown in FIG. 30B, a line segment connecting the closest point on the object P1 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle γ1 with the horizontal plane. Similarly, a line segment connecting the closest point on the object P2 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle γ2 with the horizontal plane. In this case, the horizontal distance D1 between the object P1 and the plane containing the rear surface of the excavator 60 is represented by D×cos(γ1), and the horizontal distance D2 between the object P2 and the plane containing the rear surface of the excavator 60 is , D×cos(γ2). Note that the horizontal distance D2 is greater than the horizontal distance D1.

The pixel value in the input range image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, when the pixel values in the input distance image are used as they are, in the process of changing the display color or brightness according to the distance from the rear stereo camera 6B, the top of the object P1 and the top of the object P2 have the same color or the same brightness. will be displayed. This display causes the operator to misunderstand that the objects P1 and P2 are at the same distance from the rear surface of the excavator 60 even when the object P1 is closer to the excavator 60 than the object P2 as described above. There is a risk of

Such a situation arises because the rear stereo camera 6</b>B is attached to the top of the upper rotating body 63 and captures an obliquely downward image. That is, the distance recognized by the operator is the horizontal distance from the excavator 60 , whereas the distance acquired by the stereo camera 6 is the straight line distance from the stereo camera 6 . The rear stereo camera 6B is attached to the upper part of the upper revolving body 63 so as to protect the sensor from contact with surrounding features.

Therefore, the distance correction means 13 of the image generation device 100, for example, converts the pixel values representing the distances between the objects P1 and P2 and the rear stereo camera 6B to the distances including the respective objects P1 and P2 and the rear surface of the excavator 60. Correct the pixel value to represent the horizontal distance from the plane.

Specifically, the distance correction means 13 calculates the distance between the rear stereo camera 6B and an arbitrary point on the object P1 from the coordinates of the pixel in the input distance image corresponding to the arbitrary point on the object P1 viewed from the rear stereo camera 6B. Derive the angle γ that the line segment connecting Since the mounting position and mounting angle of the rear stereo camera 6B are known, the distance correction means 13 can derive the angle γ from the coordinates of the pixel.

After that, the distance correction means 13 converts the cosine cos(γ) of the angle γ derived from the coordinates of the pixel corresponding to an arbitrary point on the feature to the pixel value of that pixel, that is, the rear stereo camera 6B and the object P1. Multiply the distance between any point above. In this way, the distance correction means 13 calculates the horizontal distance between the rear stereo camera 6B and an arbitrary point on the object P1, and converts the pixel value of the pixel corresponding to the arbitrary point to the calculated horizontal distance. replace with

Similarly, the distance correction means 13 derives the angle γ formed between the horizontal plane and the line segment connecting the rear stereo camera 6B and any point on the object P2. Then, the cosine cos(γ) of the angle γ is multiplied by its pixel value, ie, the distance between the rear stereo camera 6B and any point on the object P2. Then, the horizontal distance between the rear stereo camera 6B and an arbitrary point on the object P2 is calculated, and the pixel value of the pixel corresponding to the arbitrary point is replaced with the calculated horizontal distance.

Here, the effect of correction by the distance correction means 13 will be described with reference to FIGS. 31 and 32. FIG.

FIG. 31 shows an output range image based on an input range image that has not been subjected to range correction processing (hereinafter referred to as an “output range image before correction”) and an output range based on an input range image that has been subjected to range correction processing. 31(A) shows an output range image before correction, and FIG. 31(B) shows a corrected output range image. indicates In this embodiment, pixel extraction processing has already been performed on the input distance image.

Also, FIG. 32 shows an output image after synthesis (hereinafter referred to as "output image after synthesis before correction") obtained by synthesizing the output range image before correction with the output image generated based on the input image of the rear camera 2B; FIG. 32A is a comparison diagram showing the difference between the output image after synthesis and the output image after synthesis by synthesizing the output image with the corrected output distance image (hereinafter referred to as "output image after correction after synthesis"), and FIG. The post-output image is shown, and FIG. 32B shows the output image after correction and synthesis.

As shown in FIGS. 31A and 32A, in the pre-correction output distance image and the pre-correction post-combination output image, the object P1 and the object P2 are represented in similar color schemes. Specifically, both the object P1 and the object P2 are displayed whiter (lighter) closer to the top and blacker (darker) closer to the bottom. Therefore, it is difficult for the operator to determine which object is closer to the excavator 60 .

On the other hand, as shown in FIGS. 31(B) and 32(B), in the corrected output range image and the corrected combined output image, the object P1 and the object P2 are represented in different colors. Specifically, the object P1 is displayed lightly overall, and the object P2 is displayed dark overall. Therefore, the operator can easily recognize that the object P1 is closer to the excavator 60 than the object P2.

FIG. 33 is another example of a comparison diagram showing the difference between the post-combination output image before correction and the post-correction post-combination output image, and shows an example in which frames are superimposed.

As shown in FIG. 33A, in the pre-correction post-combination output image, when the minimum value of the pixel values of the pixels included in the feature area is adopted as the representative value of the feature area, the object P1 and the object P2 are displayed as being at a distance of 1 meter from the shovel 60 .

On the other hand, as shown in FIG. 33B, in the corrected combined output image, horizontal distances D1 and D2 between the objects P1 and P2 and the plane including the rear surface of the excavator 60 are displayed. That is, it is displayed that the object P1 is at a distance of 1 meter from the rear surface of the excavator 60 and the object P2 is at a distance of 1.5 meters from the rear surface of the excavator 60 . Therefore, the operator can easily recognize that the object P1 is closer to the excavator 60 than the object P2.

With the above configuration, the image generation device 100 corrects the pixel values in the input distance image captured by the stereo camera 6 so that the pixel values represent the horizontal distance from the excavator 60 . As a result, the image generation device 100 can inform the operator of the surrounding features in a more comprehensible manner.

Further, in the above-described embodiment, the distance correction means 13 individually corrects the pixel values of the pixels extracted by the pixel extraction means 12 . However, the invention is not limited to this. For example, the distance correcting means 13 does not calculate the horizontal distance using the cosine as described above, but collects predetermined corrections for each pixel row, each pixel column, or each pre-divided area in the input distance image plane. You can also use This is to obtain the same effect as in the case of calculating the horizontal distance using the cosine by a simple correction while suppressing the calculation load. Specifically, the distance correcting means 13 is configured, for example, so that the pixel rows located above the input range image plane are subject to greater distance increase correction, or the pixel rows located below the input range image plane are subjected to distance decrease correction. A predetermined correction amount may be collectively executed so that the correction becomes large.

Further, in the above-described embodiment, the distance correction means 13 performs correction when the rear stereo camera 6B is attached to the upper rear surface of the upper rotating body 63 and images obliquely downward. However, the invention is not limited to this. For example, the distance correction means 13 may perform correction when the stereo camera captures images in the horizontal direction, or may perform correction when the stereo camera captures obliquely upward images.

Further, in the above-described embodiment, the distance correction means 13 uses the plane including the rear surface of the excavator 60 as a reference, and the pixel values of the pixels included in the feature area are the distance between the feature and the plane including the rear surface of the excavator 60 . Correct the pixel value to represent the horizontal distance between However, the invention is not so limited. For example, the distance correction means 13 uses the turning axis PV of the excavator 60 as a reference, and converts the pixel values of the pixels included in the feature area into pixel values representing the horizontal distance between the feature and the turning axis PV of the excavator 60 . can be corrected to

Further, the distance correction means 13 may correct the pixel values of the pixels included in the feature area to pixel values representing the shortest horizontal distance between the feature and the excavator 60 . In this case, the distance correction means 13 selects a reference from a plane including the right side of the excavator 60, a plane including the left side of the excavator 60, a plane including the rear surface of the excavator 60, etc., according to the actual position of the feature. .

The shortest horizontal distance may be the shortest horizontal distance in the moving direction of the shovel 60 , that is, in the direction parallel to the front-rear direction of the lower traveling body 61 . In this case, the pixel values for features not in the direction of travel of excavator 60 are corrected to the maximum value (infinity) even if the feature is near excavator 60 . As a result, the operator can more easily recognize features in the moving direction of the excavator 60 .

Although preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. can be added.

For example, in the above-described embodiment, the image generation device 100 employs the cylindrical space model MD as the space model, but may employ a space model having other columnar shapes such as a polygonal column. A space model composed of two side surfaces may be employed, or a space model having only side surfaces may be employed.

In addition, the image generation device 100 is mounted on an excavator that is self-propelled while having movable members such as a bucket, an arm, a boom, and a turning mechanism, together with a camera and a stereo camera. An operation support system is configured to support movement and operation of those movable members. However, the image generating apparatus 100 is applied to work machines that do not have a turning mechanism, such as forklifts and asphalt finishers, or work machines that have movable members but do not self-propelled, such as industrial machines or fixed cranes. An operation support system may be installed together with the camera and the stereo camera to support the operation of the work machine.

Further, in the above-described embodiment, the distance image synthesizing means 14 generates the output distance image and then synthesizes the generated output distance image with the output image. However, the invention is not limited to this. The distance image synthesizing means 14, for example, synthesizes the input image and the input distance image to generate a post-synthesis input image, generates a post-synthesis processing target image in which information about the distance image is superimposed, and then generates a post-synthesis An output image may be generated. Further, the distance image synthesizing means 14 may, for example, synthesize the processing target image and the processing target distance image to generate a post-synthesis processing target image, and then generate the post-synthesis output image.

Also, in the above-described embodiment, the image generation device 100 includes the camera 2 and the stereo camera 6 separately. Then, the image generation device 100 generates one output image (viewpoint-transformed image) based on a plurality of input images from the camera 2, and one output image based on a plurality of input range images from the stereo camera 6. A distance image (viewpoint conversion distance image) is generated. Then, the image generation device 100 synthesizes the output distance image with the output image. However, the invention is not limited to this configuration. For example, the image generation device 100 generates an output image (viewpoint conversion image) based on three images output by one imaging mechanism of each of the left stereo camera 6L, the rear stereo camera 6B, and the right stereo camera 6R. and one output range image (viewpoint conversion range image) may be generated based on the three input range images of the three stereo cameras. In this case, camera 2 can be omitted. In this case, the left side stereo camera 6L, the rear side stereo camera 6B, and the right side stereo camera 6R are installed approximately in the center of each of the left side, the rear side, and the right side of the upper rotating body 63 so that the imaging range is not biased. It is desirable that Therefore, a plurality of sets of imaging mechanisms in each stereo camera are arranged side by side at predetermined intervals in the vertical direction. This is because it is easy to install the stereo cameras almost at the center of each surface regardless of the size of the juxtaposition interval. Also, if a plurality of sets of imaging mechanisms are arranged side by side, depending on the juxtaposition interval, it will be difficult to install the stereo camera substantially at the center of each surface.

Further, in the above-described embodiment, the stereo camera 6 derives the distance between the stereo camera 6 and the object captured in each pixel of the basic image based on the parallax between the images output by the two imaging mechanisms. Then, the stereo camera 6 substitutes the derived distance for the value of each pixel of the basic image to generate an input distance image as distance information in a two-dimensional array, and sends the generated input distance image to the control unit 1. Output. Then, the control unit 1 uses a plurality of input range images from a plurality of stereo cameras 6 to generate one output range image. However, the invention is not limited to this configuration. For example, the image generation device 100 generates a first output image (first viewpoint-transformed image) based on three images output by the first imaging mechanism of each of the three stereo cameras, and A second output image (second viewpoint conversion image) is generated based on the three images output by the second imaging mechanism in each of . Based on the parallax between the first viewpoint-transformed image and the second viewpoint-transformed image, the image generation device 100 calculates the distance between the object appearing in each pixel of the first viewpoint-transformed image and a predetermined reference point. An output range image may be generated without using the input range image by deriving .

The surroundings monitoring device of Patent Document 1 generates an illustration image of the excavator and its surroundings from the sky, and displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. Further, the surroundings monitoring device of Patent Literature 1 displays stationary obstacles and moving obstacles in different illustrations. In this manner, the surroundings monitoring device of Patent Document 1 can inform the driver of the situation of obstacles around the excavator in an easy-to-understand manner. However, the laser radar described in Patent Document 1 irradiates an object with one laser beam and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between itself and an extremely limited portion of the object surface, and the detection result is used to roughly determine the position of the illustration representing the object. is only used for Therefore, the output image presented by the surroundings monitoring device described in Patent Document 1 is still insufficient to convey to the driver the situation of obstacles around the excavator in an easy-to-understand manner.

DESCRIPTION OF SYMBOLS 1... Control part 2... Camera 2L... Left side camera 2R... Right side camera 2B... Rear camera 3... Input part 4... Storage part 5... Display part 6... Stereo camera 6L... Left side stereo camera 6R... Right side stereo camera 6B... Rear stereo camera 10... Coordinate correspondence means 11... Image generation means 12... Pixel extraction means 13. Distance correction means 14 Distance image synthesizing means 15 Driving support means 40 Input image/space model correspondence map 41 Space model/processing target image correspondence map 42 Processing target image Output image correspondence map 60 Excavator 61 Lower traveling body 62 Turning mechanism 63 Upper turning body 64 Cab 100 Image generating device

Claims (11)

a lower running body;
an upper rotating body rotatably mounted on the lower traveling body;
a plurality of stereo cameras arranged on the upper revolving body,
Each of the plurality of stereo cameras includes a plurality of imaging mechanisms in one housing ,
Further comprising a display device for displaying the image acquired by the imaging mechanism,
Displaying frames for cliffs, depressions, slopes, or holes as the ground, and people or obstacles displayed on the display device ;
Excavator. each of the plurality of stereo cameras has a first imaging mechanism and a second imaging mechanism,
displaying one viewpoint conversion image generated based on a plurality of images output by the first imaging mechanism in each of the plurality of stereo cameras;
Shovel according to claim 1 . generating one viewpoint-transformed distance image based on a plurality of distance images output by the plurality of stereo cameras;
Shovel according to claim 1 . The first stereo camera is arranged behind the upper rotating body,
The second stereo camera is arranged on the side of the upper rotating body,
A shovel according to any one of claims 1 to 3. The stereo camera is arranged facing obliquely downward on the upper surface of the upper revolving body,
A shovel according to any one of claims 1 to 4. The stereo camera does not protrude from the side surface of the upper rotating body,
A shovel according to any one of claims 1 to 5. The imaging ranges of the plurality of stereo cameras have areas that overlap with each other.
A shovel according to any one of claims 1 to 6. the plurality of stereo cameras detect the distance between each of the plurality of stereo cameras and the ground;
Shovel according to claim 1 . The plurality of stereo cameras detect the distance between each of the plurality of stereo cameras and the ground, and the distance between each of the plurality of stereo cameras and a person or an obstacle,
Shovel according to claim 8. The line type of the frame is either a dotted line, a solid line, a broken line, or a one-dot chain line,
The shape of the frame is either rectangular, circular, elliptical, or polygonal,
Shovel according to claim 1 . calculating the height of the raised portion or the depth of the depressed portion from the installation surface, which is the plane on which the shovel is located, based on the detected distance from the ground;
Shovel according to claim 8.

Images