JP2020127217A - Shovel - Google Patents

Abstract

To provide a shovel by which a distance and a direction with respect to the shovel of an object existing in a wider area around the shovel.SOLUTION: A shovel 60 includes: a lower traveling body 61; an upper pivoting body 63 pivotally mounted on the lower traveling body 61; and a plurality of stereo cameras 6 arranged on the upper pivoting body 63.SELECTED DRAWING: Figure 1

Description

The present invention relates to excavators.

2. Description of the Related Art Conventionally, there is known an ambient monitoring device that monitors a surroundings while acquiring a distance and a direction of an object existing around the shovel with respect to the shovel by using a laser radar attached to an upper swing body of the shovel (for example, Patent Document 1). See 1.).

The surroundings monitoring apparatus determines that the object is a stationary obstacle when the distance and the direction of the object with respect to the shovel do not change over a predetermined time, and determines the object as a moving obstacle when the distance and the direction change.

JP, 2008-163719, A

However, the laser radar described in Patent Document 1 irradiates an object with one laser beam and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between the extremely limited part of the object surface and itself.

In view of the above points, the present invention has an object to provide a shovel capable of acquiring the distance and the direction of an object existing in a wider range around the shovel with respect to the shovel.

In order to achieve the above-mentioned object, an excavator according to an embodiment of the present invention includes a lower traveling body, an upper revolving structure rotatably mounted on the lower traveling body, and a plurality of upper revolving structures. And a stereo camera.

By the means described above, the present invention can provide a shovel capable of acquiring the distance and direction of an object existing in a wider range around the shovel with respect to the shovel.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of the shovel in which an image generator is mounted. It is a figure which shows an example of the space model where an input image is projected. It is a figure which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane and the coordinate on a space model. It is a figure for demonstrating matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect|action of a parallel line group. It is a figure for explaining operation of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a top view (1) of the shovel in which the image generation device is mounted. It is a figure which shows each input image of three cameras mounted in the shovel, and the output image produced|generated using these input images. It is a figure for demonstrating the image disappearance prevention process which prevents the disappearance of the object in the overlapping area of each imaging range of two cameras. FIG. 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12. It is a figure which shows each input distance image of three stereo cameras mounted in the shovel, and the output distance image produced|generated using these input distance images. 16 is a comparison diagram showing the difference between the output distance image shown in FIG. 15 and the output distance image obtained by applying the distance image loss prevention processing to the output distance image of FIG. 15. It is a figure which shows an example of the after-synthesis output image obtained by synthesize|combining an output distance image with an output image. It is a figure which shows another example of the after-synthesis output image obtained by synthesize|combining an output distance image with an output image. It is a partial right side view of the shovel which mounts an image generation device. FIG. 20 is an enlarged view of a raised portion in FIG. 19. It is a figure which shows the example of an output distance image. It is a figure which shows the example of an output image. It is a flow chart which shows a flow of the 1st pixel extraction processing. It is a partial right side view of the shovel which mounts an image generation device. FIG. 25 is an enlarged view of the shallow depression portion in FIG. 24. It is a figure which shows the example of an output distance image. It is a figure which shows the example of an output image. It is a flow chart which shows the flow of the 2nd pixel extraction processing. It is a flowchart which shows the flow of a driving assistance process. It is a figure which shows the shovel in which an image generation apparatus is mounted. FIG. 6 is a comparison diagram showing a difference between an uncorrected output distance image and a corrected output distance image. FIG. 6 is a comparison diagram (No. 1) showing a difference between a corrected combined output image and a corrected combined output image. FIG. 7 is a comparison diagram (part 2) showing the difference between the pre-correction combined output image and the post-correction combined output image.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generating apparatus 100 according to an embodiment of the present invention.

The image generating apparatus 100 is an example of a work machine periphery monitoring device that monitors the periphery of a work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and a stereo camera 6. To be done. Specifically, the image generating apparatus 100 generates an output image based on the input image captured by the camera 2 mounted on the work machine and the input distance image output by the stereo camera 6, and outputs the output image to the operator. Present.

FIG. 2 is a diagram showing a configuration example of an excavator 60 as a working machine on which the image generating apparatus 100 is mounted. The excavator 60 is provided on a crawler-type lower traveling body 61, and an upper portion via a turning mechanism 62. The swing body 63 is mounted so as to be swingable around the swing axis PV. The shovel 60 uses the distance information of the two-dimensional array to generate an output image.

The upper revolving structure 63 is provided with a cab (driver's cab) 64 on the front left side thereof, an excavation attachment E on the front center thereof, and a camera 2 (right side camera 2R, rear camera 2B) on the right and rear surfaces thereof. ) And a stereo camera 6 (right side stereo camera 6R, rear stereo camera 6B). The display unit 5 is installed at a position in the cab 64 that is easily visible to the operator.

Next, each component of the image generating apparatus 100 will be described.

The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), etc. Programs corresponding to the attaching unit 10, the image generating unit 11, the pixel extracting unit 12, the distance correcting unit 13, the distance image synthesizing unit 14, and the driving support unit 15 are stored in the ROM or NVRAM, and the RAM is used as a temporary storage area. The CPU is caused to execute the process corresponding to each means while using the.

The camera 2 is a device for acquiring an input image showing the periphery of the shovel 60, and is attached to, for example, the right side surface and the rear surface of the upper swing body 63 so as to capture an area of the cab 64 that is a blind spot of the operator. (See FIG. 2), a right side camera 2R and a rear camera 2B provided with image pickup devices such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side face and the rear face of the upper swing body 63 (for example, the front face and the left side face), and a wide-angle lens or a fish-eye lens is attached so that a wide range can be imaged. May be.

The camera 2 also acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fish-eye lens or a wide-angle lens, the camera 2 outputs to the control unit 1 a corrected input image in which apparent distortion and tilt caused by using these lenses are corrected. The output image may be output, but the input image whose apparent distortion and tilt are not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

The input unit 3 is a device that allows an operator to input various information to the image generating apparatus 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

The storage unit 4 is a device for storing various kinds of information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

The display unit 5 is a device for displaying image information, and is, for example, a liquid crystal display or a projector installed in the cab 64 (see FIG. 2) of the shovel 60, and various types output by the control unit 1. Display the image.

The stereo camera 6 is a device for acquiring a two-dimensional array of distance information of objects existing around the shovel 60, and for example, the upper swing body 63 can capture an image of an area of the operator in the cab 64 which is a blind spot. Is attached to the right side surface and the rear surface (see FIG. 2). The stereo camera 6 may be attached to any one of the front surface, the left side surface, the right side surface, and the rear surface of the upper swing body 63, or may be attached to all the surfaces.

The stereo camera 6 is a device having an imaging function of a plurality of cameras. In the present embodiment, the stereo camera 6 is a device that integrally has an image pickup function of two cameras, and includes two sets of image pickup mechanisms (a combination of an image pickup element and a lens mechanism) and one shared shutter mechanism. It is provided in one housing. However, the stereo camera 6 may be composed of a plurality of separate and independent cameras.

In addition, the plurality of sets of image pickup mechanisms in the stereo camera 6 are arranged at a predetermined interval from each other. In this embodiment, the two sets of image pickup mechanisms are juxtaposed in the vertical direction (vertical direction) with a predetermined interval. The two sets of image pickup mechanisms may be arranged side by side at a predetermined interval in the horizontal direction (horizontal direction), and at predetermined intervals in the vertical direction (vertical direction) and the horizontal direction (horizontal direction). It may be juxtaposed.

Further, the stereo camera 6 acquires the distance information for each pixel, whereas the camera 2 acquires the luminance, the hue value, the saturation value, and the like for each pixel. In the present embodiment, the stereo camera 6 has each pixel of an image (hereinafter, referred to as a “basic image”) output by one imaging mechanism based on the parallax between the images output by each of the two imaging mechanisms. The distance between the object shown in and the stereo camera 6 is derived. Then, the stereo camera 6 substitutes the derived distance into the value of each pixel of the basic image to generate distance information of a two-dimensional array, and outputs the distance information to the control unit 1. Therefore, in the following, the distance information of the two-dimensional array by the stereo camera 6 will be referred to as an input distance image and an output distance image in comparison with the input image and the output image of the camera 2. The resolution (the number of pixels) of the input distance image and the output distance image of the stereo camera 6 may be the same as or different from the resolution of the input image and the output image of the camera 2. Further, one or a plurality of pixels in the input image and the output image of the camera 2 may be associated with one or a plurality of pixels in the input distance image and the output distance image of the stereo camera 6 in advance. Further, the input distance image may be generated by the control unit 1 based on a plurality of images output by each imaging mechanism of the stereo camera 6.

Further, the input image planes of the plurality of image pickup mechanisms constituting the stereo camera 6 are preferably parallel to each other, and most preferably form the same plane. This is because the process for deriving the distance from the parallax can be performed easily and with high accuracy.

Also, the stereo camera 6 may be attached to a position other than the right side surface and the rear surface of the upper swing body 63 (for example, the front surface and the left side surface), similarly to the camera 2, and a wide-angle lens so that a wide range can be imaged. Alternatively, a fisheye lens may be attached.

Also, the stereo camera 6 acquires an input distance image according to a control signal from the control unit 1 and outputs the acquired input distance image to the control unit 1, as with the camera 2. Note that, like the camera 2, when the stereo camera 6 acquires an input range image using a fisheye lens or a wide-angle lens, the stereo camera 6 corrects the corrected input distance by correcting the apparent distortion and tilt caused by using those lenses. Although the image is output to the control unit 1, the input distance image in which the apparent distortion or tilt is not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

Further, the image generation device 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding features and the sense of distance can be intuitively grasped. The output image may be generated and then the output image may be presented to the operator.

The image generating apparatus 100 performs the same process on the input distance image. In that case, the processing target image is replaced with the processing target distance image. The same applies to the following description.

The “processing target image” is an image which is generated based on the input image and which is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.), and for example, the ground surface. When an input image that is input by a camera that captures an image from above and that includes a horizontal image (for example, a sky portion) due to its wide angle of view is used in the image conversion process, the horizontal image is To prevent it from being displayed unnaturally (for example, the sky is not treated as if it were on the surface of the earth), the input image is projected onto a predetermined space model, and then the projected image projected onto the space model is divided into two parts. It is an image suitable for image conversion processing obtained by reprojecting onto a three-dimensional plane. The image to be processed may be used as it is as an output image without performing image conversion processing.

The “spatial model” is at least a plane or a curved surface other than the processing target image plane which is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane, or an angle with the processing target image plane). Which is a flat surface or a curved surface to be formed), and is a projection target of the input image configured by one or a plurality of flat surfaces or curved surfaces.

The image generating apparatus 100 may generate the output image by performing the image conversion process on the projection image projected on the spatial model without generating the processing target image. Further, the projected image may be used as it is as an output image without performing image conversion processing.

FIG. 3 is a diagram showing an example of the space model MD onto which the input image is projected, and FIG. 3A shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from the side. 3B shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and has a flat surface region R1 inside the bottom surface and a curved surface region R2 inside the side surface.

FIG. 4 is a diagram showing an example of the relationship between the spatial model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the planar region R1 of the spatial model MD. For the sake of clarity, FIG. 4 shows the space model MD as a cylindrical shape instead of the semi-cylindrical shape as shown in FIG. 3, but the space model MD has either a semi-cylindrical shape or a cylindrical shape. May be The same applies to the subsequent figures. Further, the processing target image plane R3 may be a circular area including the plane area R1 of the space model MD or an annular area not including the plane area R1 of the space model MD as described above.

Next, various means included in the control unit 1 will be described.

The coordinate associating unit 10 is a unit for associating the coordinates on the input image plane where the input image captured by the camera 2 is located, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3, For example, various parameters relating to the camera 2, such as an optical center of the camera 2, a focal length, a CCD size, an optical axis direction vector, a camera horizontal direction vector, and a projection method, which are set in advance or input through the input unit 3. And the predetermined positional relationship among the input image plane, the spatial model MD, and the processing target image plane R3, the coordinates on the input image plane, the coordinates on the spatial model MD, and the processing target image. The coordinates on the plane R3 are associated with each other, and their correspondence is stored in the input image/spatial model correspondence map 40 and the spatial model/processing target image correspondence map 41 of the storage unit 4.

In addition, when the processing target image is not generated, the coordinate associating unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model/processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

The coordinate associating means 10 also performs the same processing on the input distance image output by the stereo camera. In that case, the camera, the input image plane, and the processing target image plane are replaced with the stereo camera, the input distance image plane, and the processing target distance image plane. The same applies to the following description.

The image generation unit 11 is a unit for generating an output image, and for example, by performing scale conversion, affine transformation, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image are positioned. Corresponding to the coordinates on the output image plane, the correspondence is stored in the processing target image/output image correspondence map 42 of the storage unit 4, and the coordinate matching means 10 stores the values to correspond to the input image/space model. With reference to the map 40 and the spatial model/process target image correspondence map 41, the value of each pixel in the output image (for example, the luminance value, the hue value, the saturation value, etc.) and the value of each pixel in the input image And an output image is generated.

Further, the image generating means 11 is set in advance or is inputted via the input unit 3, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, etc. On the basis of various parameters of the processing target image plane R3 and the coordinates on the output image plane where the output image is located, and the correspondence is stored in the processing target image/output image correspondence map 42 of the storage unit 4. While referring to the input image/spatial model correspondence map 40 and the spatial model/processing target image correspondence map 41 that have been stored and the coordinate correspondence means 10 has stored the values, the value of each pixel in the output image (for example, the brightness value, Hue value, saturation value, etc.) and the value of each pixel in the input image are associated with each other to generate an output image.

Note that the image generation unit 11 may change the scale of the processing target image to generate the output image without using the concept of the virtual camera.

Further, when the image to be processed is not generated, the image generation means 11 associates the coordinates on the spatial model MD with the coordinates on the output image plane according to the image conversion processing performed, and associates the input image with the spatial model. With reference to the map 40, the output image is generated by associating the value of each pixel in the output image (for example, the luminance value, the hue value, the saturation value, etc.) with the value of each pixel in the input image. In this case, the image generation unit 11 omits the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane, and the storage of the correspondence relationship in the processing target image/output image correspondence map 42. ..

The image generation means 11 also performs the same processing on the input distance image or the processing target distance image. In that case, the output image plane is replaced by the output range image plane. The same applies to the following description.

The pixel extracting means 12 is means for extracting pixels satisfying a predetermined condition from the input distance image of the stereo camera 6. For example, the pixel extraction unit 12 extracts pixels satisfying a predetermined condition from pixels having a pixel value indicating the distance between the feature around the shovel 60 and the stereo camera 6 in the input distance image.

Specifically, the pixel extraction unit 12 has a distance (hereinafter, referred to as “installation surface distance”) between the stereo camera 6 and a plane on which the shovel 60 is located (hereinafter, referred to as “installation surface”). Pixels whose difference is equal to or greater than a predetermined distance are extracted. The installation surface may be an inclined surface. The installation surface distance is set in advance for each pixel in the input distance image of the stereo camera 6, based on the installation position and the installation angle of the stereo camera 6. Then, the pixel extraction unit 12 replaces the pixel value of the pixel that has not been extracted with a predetermined value (for example, zero as the minimum value), and maintains the pixel value of the extracted pixel as it is, thereby obtaining the input distance image. to correct. The details of the pixel extraction means 12 will be described later.

The distance correction unit 13 is a unit for correcting the distance information acquired by the stereo camera. For example, the distance correction unit 13 corrects the pixel value extracted by the pixel extraction unit 12. Specifically, the distance correction unit 13 sets the value of a pixel representing the distance between the feature around the shovel 60 and the stereo camera 6 between the reference outside the stereo camera 6 and the feature. Correct to a value that represents distance. Further, the distance correction unit 13 may correct the pixel value in the input distance image output by the stereo camera 6 before the extraction by the pixel extraction unit 12. The details of the distance correction means 13 will be described later.

The distance image synthesizing unit 14 is a unit for synthesizing the image regarding the camera and the image regarding the stereo camera. For example, the distance image synthesizing unit 14 synthesizes the output image generated by the image generating unit 11 based on the input image of the camera 2 and the output distance image based on the input distance image of the stereo camera 6. The output distance image is an output distance image that has been subjected to at least one of extraction by the pixel extraction unit 12 and correction by the distance correction unit 13. However, the output distance image may be an output distance image before the extraction by the pixel extraction unit 12 and the correction by the distance correction unit 13. The details of the distance image synthesizing means 14 will be described later.

The driving support means 15 is means for supporting driving of the shovel 60. For example, the driving support unit 15 executes a function of supporting driving of the shovel 60 when it is determined that a predetermined feature exists in the moving direction of the shovel 60.

The “predetermined feature” is, for example, a person such as a worker who may collide with the shovel 60, an obstacle that blocks the movement of the shovel 60, a cliff or a depressed portion that may cause the shovel 60 to slide, fall, or get stuck. , Slopes, holes, etc.

The function for supporting driving (hereinafter, referred to as “driving support function”) includes, for example, outputting an alarm, displaying a warning on the display unit 5, turning on/blinking a warning light, decelerating the shovel 60, and traveling speed of the shovel 60. Restrictions, stop of traveling of the shovel 60, and the like.

Specifically, when the driving support means 15 determines that there is a cliff in the moving direction (backward) of the shovel 60 while the shovel 60 is moving (retracting), it informs the display unit 5 of the existence of the cliff. Display a warning. The moving direction of the shovel 60 is detected, for example, based on the operation content of the operation lever (for example, the operation direction, the operation amount, etc.).

Further, the driving support means 15 may decrease the moving (retracting) speed of the shovel 60 as the distance between the shovel 60 and the cliff becomes shorter, and the distance between the shovel 60 and the cliff becomes equal to or less than a predetermined distance. When it becomes, the movement (retraction) of the shovel 60 may be stopped.

In addition, the driving support means 15 determines whether or not a predetermined feature exists based on the output of the stereo camera 6. For example, when the stereo camera 6 detects a feature having a depth equal to or greater than a predetermined distance from the installation surface, the driving support unit 15 determines that there is a depression that may slide the shovel 60. Further, the driving support means 15 causes the shovel 60 to slide down only when at least one of the actual depth, opening area, width, and depth of the depression calculated based on the input distance image satisfies a predetermined condition. It may be determined that there is a feared depression. In addition, the driving support unit 15 determines whether or not there is a depression that may slide the shovel 60 based on the increasing gradient of the pixel value along the predetermined direction (moving direction of the shovel 60) in the input distance image. You may judge.

Alternatively, the driving support unit 15 may determine that there is an obstacle that may collide with the shovel 60 when the stereo camera 6 detects a feature having a height of a predetermined distance or more from the installation surface. .. Further, the driving support unit 15 determines whether or not there is an obstacle that may collide with the shovel 60 based on the decreasing gradient of the pixel value along the predetermined direction (moving direction of the shovel 60) in the input distance image. You may judge.

Next, an example of specific processing by the coordinate association unit 10 and the image generation unit 11 will be described.

The coordinate associating unit 10 can associate the coordinates on the input image plane with the coordinates on the spatial model using, for example, a Hamiltonian quaternion.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the spatial model. The input image plane of the camera 2 is a UVW orthogonal coordinate whose origin is the optical center C of the camera 2. The spatial model is represented as one plane in the system, and the spatial model is represented as a three-dimensional surface in the XYZ rectangular coordinate system.

First, the coordinate associating means 10 converts the coordinates on the spatial model (coordinates on the XYZ coordinate system) into the coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is optically converted. After parallel translation to the center C (the origin of the UVW coordinate system), make sure that the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" is in the opposite direction). This means that the UVW coordinate system has the +W direction in the front of the camera and the XYZ coordinate system has the -Z direction in the vertical downward direction.

When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system, and therefore the coordinate associating unit 10 parallelizes the XYZ coordinate system to each of the plurality of UVW coordinate systems. It will be moved and rotated.

In the above conversion, after the XYZ coordinate system is moved in parallel so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, the Z axis is rotated so as to coincide with the −W axis, and further, the X axis is moved to Since it is realized by rotating so as to coincide with the axis, the coordinate associating means 10 can describe these conversions by Hamilton's quaternion to combine these two rotations into one rotation calculation. it can.

By the way, the rotation for matching a certain vector A with another vector B corresponds to the processing for rotating the vector A and the vector B by an angle formed by the normal line of the surface between the vector A and the vector B as an axis. , And the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

Will be represented by

Further, the unit vector N of the normal line of the surface stretched by the vector A and the vector B is calculated from the outer product of the vector A and the vector B.

Will be represented by

It should be noted that the quaternion is, when i, j, and k are imaginary units,

In the present embodiment, the quaternion Q has a real component t, and pure imaginary components a, b, and c.

, And the conjugate quaternion of the quaternion Q is

Shall be represented by.

The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while the real component t is 0 (zero), and t, a, b , C, it is possible to express a rotational motion around an arbitrary vector as an axis.

Furthermore, the quaternion Q can be expressed as one rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S(sx, sy, sz) can be expressed as an arbitrary unit vector. A point D(ex, ey, ez) when rotated by an angle θ while using C(l, m, n) as an axis can be expressed as follows.

Here, in the present embodiment, if a quaternion representing rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X′. X'is

Will be represented by

Further, in the present embodiment, if Qx is a quaternion that represents a rotation that causes the line connecting the point X′ on the X axis and the origin to coincide with the U axis, “Z axis coincides with −W axis, and further, , The rotation that makes the X-axis coincide with the U-axis” is

Will be represented by

As described above, the coordinate P′ when the arbitrary coordinate P on the spatial model (XYZ coordinate system) is expressed by the coordinate on the input image plane (UVW coordinate system) is

Since the quaternion R is invariable in each of the cameras 2, the coordinate associating means 10 thereafter calculates the coordinates on the space model (XYZ coordinate system) only by executing this calculation. It can be converted into coordinates on the input image plane (UVW coordinate system).

After converting the coordinates on the space model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate associating means 10 and the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP′ connecting an arbitrary coordinate P on the model with a coordinate P′ represented by the UVW coordinate system and the optical axis G of the camera 2 is calculated.

Further, the coordinate associating means 10 defines an intersection E of the plane H and the optical axis G and a coordinate P′ on a plane H parallel to the input image plane R4 (for example, CCD surface) of the camera 2 and including the coordinate P′. The declination angle φ formed by the line segment EP′ connecting the two and the U′ axis on the plane H, and the length of the line segment EP′ are calculated.

In the optical system of the camera, since the image height h is usually a function of the incident angle α and the focal length f, the coordinate associating means 10 has a normal projection (h=ftan α) and an orthogonal projection (h=fsin α). , 3D projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/2)), equidistant projection (h=fα), etc. Calculate h.

After that, the coordinate associating means 10 decomposes the calculated image height h into a U component and a V component on the UV coordinate system by the argument φ, and uses a numerical value corresponding to the pixel size per pixel of the input image plane R4. By performing the division, the coordinates P(P′) on the spatial model MD and the coordinates on the input image plane R4 can be associated with each other.

When the pixel size per pixel in the U-axis direction of the input image plane R4 is a _U and the pixel size per pixel in the V-axis direction of the input image plane R4 is a _V , the coordinates P on the spatial model MD are shown. The coordinates (u,v) on the input image plane R4 corresponding to (P′) are

Will be represented by

In this way, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on the one or more input image planes R4 existing for each camera, and coordinates on the space model MD, camera identifier. , And the coordinates on the input image plane R4 are associated and stored in the input image/space model correspondence map 40.

Further, since the coordinate associating unit 10 calculates the coordinate conversion by using the quaternion, it has an advantage that the gimbal lock is not generated unlike the case where the coordinate conversion is calculated by using the Euler angle. .. However, the coordinate associating unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may calculate the coordinate conversion using the Euler angle.

When it is possible to associate the coordinates with the plurality of input image planes R4, the coordinate associating unit 10 inputs the coordinates P(P′) on the spatial model MD with respect to the camera having the smallest incident angle α. The coordinates may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

Next, of the coordinates on the space model MD, a process of reprojecting the coordinates on the curved surface region R2 (coordinates having a component in the Z-axis direction) onto the processing target image plane R3 on the XY plane will be described.

FIG. 6 is a diagram for explaining the association between the coordinates by the coordinate associating means 10. FIG. 6A shows the input image plane R4 of the camera 2 that adopts the normal projection (h=ftan α) as an example. It is a figure which shows the correspondence between the above coordinate and the coordinate on space model MD, and coordinate matching means 10 is on the space model MD corresponding to the coordinate on the input image plane R4 of the camera 2. The two coordinates are made to correspond to each other so that each line segment connecting with the coordinates of ‘a’ passes through the optical center C of the camera 2.

In the example of FIG. 6A, the coordinate associating unit 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane region R1 of the spatial model MD, and inputs the image 2 plane R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

When the camera 2 adopts a projection method other than the normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate associating means 10 is adapted to each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are made to correspond to the coordinates L1 and L2 on the spatial model MD.

Specifically, the coordinate associating means 10 has a predetermined function (for example, orthographic projection (h=fsinα), stereoscopic projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/ 2)), equidistant projections (h=fα), etc.) are used to associate the coordinates on the input image plane with the coordinates on the spatial model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

FIG. 6B is a diagram showing a correspondence relationship between the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3, and the coordinate associating means 10 is located on the XZ plane. A parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the spatial model MD and the processing target image corresponding to the coordinates are introduced. The coordinates on the plane R3 are placed on one of the parallel line groups PL so that the two coordinates are associated with each other.

In the example of FIG. 6B, the coordinate associating means 10 assumes that the coordinate L2 on the curved surface region R2 of the spatial model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line, Make it correspond.

The coordinate associating unit 10 can associate the coordinates on the plane area R1 of the spatial model MD with the coordinates on the processing target image plane R3 by using the parallel line group PL similarly to the coordinates on the curved surface area R2. However, in the example of FIG. 6B, since the plane area R1 and the processing target image plane R3 are common planes, the coordinates L1 on the plane area R1 of the spatial model MD and the processing target image plane R3 are The coordinate M1 has the same coordinate value.

In this way, the coordinate associating unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. And stores it in the spatial model/processing target image correspondence map 41.

FIG. 6C is a diagram showing a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting the normal projection (h=ftan α) as an example. In the image generating means 11, each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates passes through the optical center CV of the virtual camera 2V. In this way, both coordinates are associated.

In the example of FIG. 6C, the image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane area R1 of the spatial model MD), The coordinates N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

When the virtual camera 2V adopts a projection method other than the normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the image generation means 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are made to correspond to the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the image generation unit 11 uses a predetermined function (for example, orthographic projection (h=fsinα), stereoscopic projection (h=2ftan(α/2)), equisolid angle projection (h=2fsin(α/2)). )), equidistant projection (h=fα), etc.), the coordinates on the output image plane R5 are associated with the coordinates on the processing target image plane R3. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

In this way, the image generating means 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and sets the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. While referring to the input image/spatial model correspondence map 40 and the spatial model/processing target image correspondence map 41 stored in the processing target image/output image correspondence map 42 in association with each other, each of the output images The output image is generated by associating the pixel value with the value of each pixel in the input image.

Note that FIG. 6D is a diagram in which FIG. 6A to FIG. 6C are combined, and includes the camera 2, the virtual camera 2V, the plane area R1 and the curved surface area R2 of the space model MD, and the processing target. The mutual positional relationship of the image plane R3 is shown.

Next, the operation of the parallel line group PL introduced by the coordinate associating means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG. 7.

FIG. 7A is a diagram in the case where an angle β is formed between the parallel line group PL located on the XZ plane and the processing target image plane R3, and FIG. 7B is on the XZ plane. FIG. 9 is a diagram in the case where an angle β1 (β1>β) is formed between the parallel line group PL located and the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 7A and 7B corresponds to each of the coordinates Ma to Md on the processing target image plane R3, The intervals between the coordinates La to Ld in FIG. 7A are equal to the intervals between the coordinates La to Ld in FIG. 7B. Note that the parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually, it exists so as to extend radially from all points on the Z axis toward the image plane R3 to be processed. It shall be. The Z axis in this case will be referred to as a "reprojection axis".

As shown in FIGS. 7A and 7B, the intervals between the coordinates Ma to Md on the processing target image plane R3 are the angles between the parallel line group PL and the processing target image plane R3. It linearly decreases as it increases (it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md). On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, so that the interval of the coordinate group does not change. ..

The change in the interval of these coordinate groups is linearly enlarged or only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD among the image portions on the output image plane R5 (see FIG. 6). It means to be reduced.

Next, an alternative example of the parallel line group PL introduced by the coordinate associating means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

8A is a diagram in the case where all the auxiliary line groups AL located on the XZ plane extend from the starting point T1 on the Z axis toward the processing target image plane R3, and FIG. It is a figure when all the line groups AL extend from the starting point T2 (T2>T1) on the Z-axis toward the processing target image plane R3. Further, it is assumed that the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 8A and 8B respectively correspond to the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8A, the coordinates Mc and Md are outside the region of the processing target image plane R3, and thus are not shown.), and the respective intervals of the coordinates La to Ld in FIG. It is assumed to be equal to the respective intervals of the coordinates La to Ld in 8(B). Although the auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, it actually exists so as to extend radially from any one point on the Z axis toward the image plane R3 to be processed. It shall be. Note that the Z axis in this case will be referred to as the "reprojection axis" as in FIG.

As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (high) between the starting point of the auxiliary line group AL and the origin O. S) increases non-linearly (the larger the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md, the larger the decrease amount of each interval). On the other hand, since the coordinate group on the plane area R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the coordinate group interval does not change. ..

The change in the interval between the coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6) as in the case of the parallel line group PL. This means that only the image portion is non-linearly enlarged or reduced.

In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the planar region R1 of the spatial model MD, without affecting the spatial model MD. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface region R2 can be linearly or non-linearly enlarged or reduced, the road surface image in the vicinity of the shovel 60 can be obtained. The features located around the shovel 60 (features in the image when looking horizontally around the shovel 60) are swiftly affected without affecting (the virtual image when the shovel 60 is viewed from directly above). Moreover, it can be flexibly enlarged or reduced, and the visibility of the blind spot area of the shovel 60 can be improved.

Next, with reference to FIG. 9, a process in which the image generation apparatus 100 generates a processing target image (hereinafter, referred to as “processing target image generation processing”) and an output image using the generated processing target image. The process of generating (hereinafter, referred to as “output image generating process”) will be described. Note that FIG. 9 is a flowchart showing the flow of the processing target image generation processing (steps S1 to S3) and the output image generation processing (steps S4 to S6). In addition, it is assumed that the arrangement of the camera 2 (input image plane R4), the space model (the plane area R1 and the curved surface area R2), and the processing target image plane R3 are predetermined.

First, the control unit 1 causes the coordinate associating unit 10 to associate the coordinates on the processing target image plane R3 with the coordinates on the spatial model MD (step S1).

Specifically, the coordinate associating unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. A point, one of which intersects the curved surface area R2 of the space model MD, is calculated, and the coordinate on the curved surface area R2 corresponding to the calculated point is set on the curved surface area R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived and the correspondence is stored in the spatial model/processing target image correspondence map 41. The angle formed between the parallel line group PL and the image plane R3 to be processed may be a value stored in advance in the storage unit 4 or the like, and the operator can dynamically change the angle via the input unit 3. It may be a value to enter.

Further, when one coordinate on the processing target image plane R3 matches one coordinate on the plane region R1 of the spatial model MD, the coordinate associating means 10 sets the one coordinate on the plane region R1 to the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the corresponding relation is stored in the spatial model/process target image correspondence map 41.

After that, the control unit 1 causes the coordinate associating unit 10 to associate one coordinate on the space model MD derived by the above-described process with the coordinate on the input image plane R4 (step S2).

Specifically, the coordinate associating unit 10 is a line segment that extends from one coordinate on the spatial model MD and acquires the coordinates of the optical center C of the camera 2 that adopts the normal projection (h=ftan α). A point where a line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are calculated on the input image plane R4 corresponding to the one coordinate on the space model MD. It is derived as one coordinate and the correspondence is stored in the input image/space model correspondence map 40.

After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the spatial model MD and the coordinates on the input image plane R4 (step S3), and all the coordinates still exist. If it is determined that the values are not associated with each other (NO in step S3), the processes in steps S1 and S2 are repeated.

On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 terminates the processing target image generation processing and then starts the output image generation processing, and the image generation means 11 causes The coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine transformation, or distortion conversion on the processing target image, and is determined by the content of the applied scale conversion, affine transformation, or distortion conversion, The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image/output image correspondence map 42.

Alternatively, when generating the output image using the virtual camera 2V, the image generating means 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the projection method adopted, The correspondence may be stored in the processing target image/output image correspondence map 42.

Alternatively, when generating the output image using the virtual camera 2V that adopts the normal projection (h=ftan α), the image generating means 11 acquires the coordinates of the optical center CV of the virtual camera 2V and then outputs the image. It is a line segment extending from one coordinate on the image plane R5, and a point at which the line segment passing through the optical center CV intersects the processing target image plane R3 is calculated, and the coordinates on the processing target image plane R3 corresponding to the calculated point are calculated. May be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the corresponding relationship may be stored in the processing target image/output image correspondence map 42.

After that, the control unit 1 refers to the input image/spatial model correspondence map 40, the spatial model/processing target image correspondence map 41, and the processing target image/output image correspondence map 42 by the image generating means 11, and the input image plane R4. Correspondence between the upper coordinates and the coordinates on the space model MD, correspondence between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the coordinates on the processing target image plane R3 and the output image plane R5. By tracing the correspondence relationship with the coordinates of the input image plane R4 corresponding to the respective coordinates on the output image plane R5, a value (for example, a luminance value, a hue value, a saturation value, etc.) is acquired. , The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). In addition, when a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 causes each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value (for example, an average value, a maximum value, a minimum value, an intermediate value, etc.) is derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. ..

Then, the control unit 1 determines whether all coordinate values on the output image plane R5 are associated with coordinate values on the input image plane R4 (step S6), and all coordinate values are associated. If it is determined that they are not attached (NO in step S6), the processes in steps S4 and S5 are repeated.

On the other hand, when it is determined that the values of all the coordinates are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

If the image generation apparatus 100 does not generate the processing target image, the processing target image generation processing is omitted, and the "coordinates on the processing target image plane" in step S4 in the output image generation processing is set on the space model. It should be read as "coordinates".

With the above configuration, the image generating apparatus 100 can generate the processing target image and the output image that allow the operator to intuitively understand the positional relationship between the features around the shovel 60 and the shovel 60. ..

Further, the image generating apparatus 100 executes the coordinate association so as to trace back from the processing target image plane R3 to the input image plane R4 via the spatial model MD, thereby setting each coordinate on the processing target image plane R3. One or a plurality of coordinates on R4 can be surely associated with each other, and the coordinates are associated in the order from the input image plane R4 through the spatial model MD to the processing target image plane R3 (in this case, Can surely make each coordinate on the input image plane R4 correspond to one or a plurality of coordinates on the processing target image plane R3. However, a part of the coordinates on the processing target image plane R3 is input to the input image plane R4. In some cases, it may not be associated with any of the above coordinates, in which case it is necessary to perform interpolation processing or the like on some of the coordinates on the processing target image plane R3.) Can be generated quickly.

Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the spatial model MD, the image generating device 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. The desired enlargement or reduction can be realized without rewriting the contents of the input image/spatial model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the spatial model/processing target image correspondence map 41. ..

Further, when changing the appearance of the output image, the image generating apparatus 100 simply changes the values of various parameters related to scale conversion, affine transformation, or distortion conversion and rewrites the processing target image/output image correspondence map 42. A desired output image (scale conversion image, affine conversion image, or distortion conversion image) can be generated without rewriting the contents of the input image/spatial model correspondence map 40 and the space model/processing target image correspondence map 41.

Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 only needs to change the values of various parameters of the virtual camera 2V and rewrite the processing target image/output image correspondence map 42 to change the input image/space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model/processing target image correspondence map 41.

FIG. 10 is a display example when an output image generated using the input images of the two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the shovel 60 is displayed on the display unit 5. ..

The image generating apparatus 100 projects the respective input images of the two cameras 2 on the plane region R1 and the curved region R2 of the spatial model MD, and then re-projects them on the process target image plane R3 to generate the process target image. Then, the output image is generated by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image, and the vicinity of the shovel 60 is changed from the sky. An image viewed down (image in the plane area R1) and an image viewed around the shovel 60 in the horizontal direction (image in the image plane R3 to be processed) are simultaneously displayed.

Note that the output image is generated by performing image conversion processing (for example, viewpoint conversion processing) on the image projected on the spatial model MD when the image generation apparatus 100 does not generate the processing target image. I shall.

Further, the output image is trimmed into a circle so that the image when the shovel 60 makes a turning motion can be displayed without a sense of discomfort, and the center CTR of the circle is on the cylinder center axis of the space model MD and the turning of the shovel 60. It is generated so as to be on the axis PV, and is displayed so as to rotate about its center CTR in response to the turning motion of the shovel 60. In this case, the cylinder center axis of the space model MD may or may not match the reprojection axis.

The radius of the space model MD is, for example, 5 meters, and the angle formed by the parallel line group PL with the processing target image plane R3 or the starting point height of the auxiliary line group AL is the turning of the shovel 60. When there is a feature (for example, a worker) at a position away from the center by the maximum reach distance (for example, 12 meters) of the excavation attachment E, the feature is sufficiently large on the display unit 5 (for example, for example). , 7 mm or more.) Can be set to be displayed.

Further, as the output image, the CG image of the shovel 60 may be arranged such that the front of the shovel 60 matches the upper part of the screen of the display unit 5 and the turning center thereof matches the center CTR. This is to make the positional relationship between the shovel 60 and the feature appearing in the output image easier to understand. The output image may be arranged around a frame image including various information such as the orientation.

Next, the details of the output image generated by the image generating apparatus 100 will be described with reference to FIGS. 11 to 18.

FIG. 11 is a top view of the shovel 60 equipped with the image generating apparatus 100. In the embodiment shown in FIG. 11, the shovel 60 comprises three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three stereo cameras 6 (left side stereo camera 6L, right side stereo). The camera 6R and the rear stereo camera 6B) are provided. Note that regions CL, CR, and CB indicated by alternate long and short dash lines in FIG. 11 indicate the imaging ranges of the left camera 2L, the right camera 2R, and the rear camera 2B, respectively. Areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate the imaging ranges of the left stereo camera 6L, the right stereo camera 6R, and the rear stereo camera 6B, respectively.

In addition, in the present embodiment, the imaging range of the stereo camera 6 is narrower than the imaging range of the camera 2, but the imaging range of the stereo camera 6 may be the same as the imaging range of the camera 2 or wider than the imaging range of the camera 2. Good. Further, the imaging range of the stereo camera 6 is located near the shovel 60 in the imaging range of the camera 2, but may be in a region farther from the shovel 60.

FIG. 12 is a diagram showing respective input images of the three cameras 2 mounted on the shovel 60 and output images generated by using the input images.

The image generation apparatus 100 projects the respective input images of the three cameras 2 on the plane area R1 and the curved surface area R2 of the spatial model MD and then re-projects them on the processing target image plane R3 to generate the processing target image. .. Further, the image generating apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generating apparatus 100 generates an image of the vicinity of the shovel 60 looking down from above (image in the plane area R1) and an image of looking around the shovel 60 in the horizontal direction (image in the processing target image plane R3). Display at the same time. The image displayed in the center of the output image is the CG image 60CG of the shovel 60.

In FIG. 12, the input image of the right-side camera 2R and the input image of the rear-side camera 2B each capture a person in the overlapping area of the imaging range of the right-side camera 2R and the rear-side camera 2B (right side). (See a region R10 surrounded by a chain double-dashed line in the input image of the rear camera 2R and a region R11 surrounded by a chain double-dashed line in an input image of the rear camera 2B.).

However, assuming that the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image causes the person in the overlapping area to disappear (one point in the output image is lost). See the area R12 surrounded by the chain line.).

Therefore, in the image generating apparatus 100, in the output image portion corresponding to the overlapping area, the area on the input image plane of the rear camera 2B is associated with the area on the input image plane of the right camera 2R. Areas are mixed to prevent the objects in the overlapping area from disappearing.

FIG. 13 is a diagram for explaining the image disappearance prevention processing for preventing the disappearance of an object in the overlapping area of the imaging ranges of the two cameras.

FIG. 13A is a diagram showing an output image portion corresponding to an overlapping area of the imaging range of the right camera 2R and the imaging range of the rear camera 2B, and corresponds to the rectangular area R13 shown by the dotted line in FIG.

Further, in FIG. 13A, a region PR1 filled with gray is an image region in which the input image portion of the rear camera 2B is arranged, and the rear camera is located at each coordinate on the output image plane corresponding to the region PR1. The coordinates on the input image plane of 2B are associated with each other.

On the other hand, the region PR2 filled with white is an image region in which the input image portion of the right side camera 2R is arranged, and the coordinates on the output image plane corresponding to the region PR2 are at the input image plane of the right side camera 2R. The upper coordinates are associated.

In the present embodiment, the region PR1 and the region PR2 are arranged so as to form a striped pattern, and the boundary line of the portion where the regions PR1 and PR2 are alternately arranged in a striped pattern is centered on the turning center of the shovel 60. It is defined by concentric circles on a horizontal plane.

FIG. 13B is a top view showing the state of the space area diagonally to the right behind the shovel 60, and shows the current state of the space area imaged by both the rear camera 2B and the right side camera 2R. In addition, FIG. 13B shows that a rod-shaped three-dimensional object OB exists in the diagonally right rear of the shovel 60.

FIG. 13C shows a part of the output image generated based on the input image obtained by actually capturing the spatial area shown in FIG. 13B by the rear camera 2B and the right camera 2R.

Specifically, the image OB1 is expanded in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint conversion for generating the road surface image from the image of the three-dimensional object OB in the input image of the rear camera 2B. Represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

Further, the image OB2 is an image of the three-dimensional object OB in the input image of the right side camera 2R, which is expanded in the extension direction of the line connecting the right side camera 2R and the three-dimensional object OB by the viewpoint conversion to generate the road surface image. Represents a thing. That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the right camera 2R.

As described above, in the overlapping area, the image generating apparatus 100 has an area PR1 in which the coordinates on the input image plane of the rear camera 2B are associated with an area PR2 in which the coordinates on the input image plane of the right camera 2R are associated. To mix. As a result, the image generating apparatus 100 displays both the two images OB1 and OB2 regarding one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

FIG. 14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12, and FIG. FIG. 14B shows an output image after the image loss prevention processing is applied. In the area R12 surrounded by the one-dot chain line in FIG. 14A, the person disappears, whereas in the area R14 surrounded by the one-dot chain line in FIG. 14B, the person is displayed without disappearing.

Next, the output distance image generated by the image generating apparatus 100 will be described with reference to FIG. Note that FIG. 15 is a diagram showing an input distance image of each of the three stereo cameras 6 mounted on the shovel 60 and an output distance image generated using the input distance images.

The image generation device 100 projects the input distance images of the three stereo cameras 6 onto the plane area R1 and the curved surface area R2 of the spatial model MD, and then re-projects them on the processing target distance image plane R3. Generate a range image. The image generating apparatus 100 also generates an output distance image by performing image conversion processing (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target distance image. Then, the image generating apparatus 100, the distance image looking down the vicinity of the shovel 60 from above (distance image in the plane area R1), and the distance image looking around the shovel 60 in the horizontal direction (image in the processing target distance image plane R3). ) And are displayed at the same time. The image displayed in the center of the output distance image is the CG image 60CG of the shovel 60.

The input distance image and the output distance image shown in FIG. 15 are displayed so that the smaller the pixel value (distance from the stereo camera 6), the whiter (lighter) the color, and the larger the pixel value, the blacker (darker). The pixel value (distance) of the portion corresponding to each of the road surface and the upper swing body 63 is set to the maximum value (infinity) in order to improve the visibility in the output distance image of the features existing around the shovel 60. To be done. That is, the portions corresponding to the road surface and the upper swing body 63 are represented in black.

In FIG. 15, the input range image of the right side stereo camera 6R and the input range image of the rear stereo camera 6B are in the overlapping area of the image capturing range of the right side stereo camera 6R and the rear stereo camera 6B, respectively. (See a region R15 surrounded by a chain double-dashed line in the input distance image of the right stereo camera 6R and a region R16 surrounded by a chain double-dashed line in the input distance image of the rear stereo camera 6B).

However, if it is assumed that the coordinates on the output distance image plane are associated with the coordinates on the input distance image plane for the closest stereo camera, the output distance image causes the person in the overlapping area to disappear ( (Refer to the region R17 surrounded by the alternate long and short dash line in the output distance image.).

Therefore, in the output distance image portion corresponding to the overlapping area, the image generating device 100 selects, from among the coordinates on the input distance image plane of the rear stereo camera 6B and the coordinates on the input distance image plane of the right stereo camera 6R, The coordinate having the smaller pixel value (distance) is associated with the coordinate on the output distance image plane. As a result, the image generating apparatus 100 displays two distance images related to one three-dimensional object on the output distance image, and prevents the three-dimensional object from disappearing from the output distance image. Note that, hereinafter, this processing of preventing the disappearance of an object in the overlapping area of the imaging ranges of the two stereo cameras is referred to as distance image disappearance prevention processing.

FIG. 16 is a comparison diagram showing the difference between the output distance image of FIG. 15 and the output distance image obtained by applying the distance image loss prevention processing to the output distance image of FIG. 15, and FIG. 15 shows the output image of FIG. 15, and FIG. 16(B) shows the output distance image after the distance image loss prevention processing is applied. While the person disappears in the area R17 surrounded by the alternate long and short dash line in FIG. 16A, the person is displayed without disappearing in the area R17 surrounded by the alternate long and short dash line in FIG. 16B.

Next, with reference to FIG. 17 and FIG. 18, a process in which the image generation apparatus 100 synthesizes an output distance image with an output image (hereinafter, referred to as “distance image synthesis processing”) will be described.

FIG. 17 is a diagram illustrating an example of a combined output image obtained by combining an output distance image with an output image.

The distance image synthesizing unit 14 of the image generating apparatus 100, for example, as shown in FIG. 16B, the pixel value (distance) of the extracted pixel extracted by the pixel extracting unit 12 is a luminance value, a hue value, a saturation value, or the like. Then, the extracted pixels are displayed on the output image shown in FIG. The extracted pixels that are displayed in a superimposed manner have, for example, a color according to the distance from the stereo camera 6, and as the distance increases, the color gradually or steplessly becomes red, yellow, green, and blue. Change. It should be noted that the extracted pixel to be displayed in a superimposed manner has a brightness corresponding to the distance from the stereo camera 6, and the brightness may be reduced stepwise or steplessly as the distance increases. Regions EX1 to EX5 in FIG. 17 are regions formed by the extracted pixels, and are hereinafter referred to as feature regions.

Further, the distance image synthesizing unit 14 may superimpose and display the distance information of the feature areas EX1 to EX5 formed by a predetermined number or more of adjacent extraction pixels on the output image. Specifically, the distance image synthesizing unit 14 determines, for example, the minimum value, the maximum value, the average value, and the intermediate value of the pixel values (distances) of the pixels configuring the feature area EX1 to the feature area EX1. Is displayed on the output image as a representative value representing. The distance image synthesizing unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is for making it easier to recognize features relatively close to the shovel 60, that is, features having a relatively high possibility of contact.

FIG. 18 is a diagram showing another example of the combined output image obtained by combining the output distance image with the output image.

In FIG. 18, the distance image synthesizing unit 14 superimposes and displays the frames RF1 to RF5 corresponding to the feature areas EX1 to EX5 of FIG. 17, instead of superimposing and displaying by changing the color and brightness of the extracted pixel. The line type of the frame is arbitrary including a dotted line, a solid line, a broken line, a one-dot chain line, and the shape of the frame is also arbitrary including a rectangle, a circle, an ellipse, and a polygon.

Further, the distance image synthesizing unit 14 may superimpose and display the distance information corresponding to each of the frames RF1 to RF5 on the output image. Specifically, the distance image synthesizing unit 14 sets, for example, the minimum value, the maximum value, the average value, and the intermediate value of the pixel values (distances) of the pixels forming the feature area EX1 corresponding to the frame RF1 to the frame RF1. Is displayed on the output image as a representative value of the distance corresponding to. The distance image synthesizing unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is for making it easier to recognize features relatively close to the shovel 60, that is, features having a relatively high possibility of contact.

Further, the distance image synthesizing unit 14 may change the color of the frame according to the corresponding distance information. Specifically, the distance image synthesizing unit 14 changes the color of the frame stepwise or steplessly such as red, yellow, green, and blue as the distance increases. The distance image synthesizing unit 14 may decrease the frame brightness stepwise or steplessly as the distance increases.

The operator of the shovel 60 can more easily recognize the position of the feature existing around the shovel 60 and the distance to the feature by looking at the post-composition output image as described above. Further, the operator of the shovel 60 can more easily recognize the position of the feature existing in the area that is a blind spot as seen from the operator and the distance to the feature.

With the above configuration, the image generating apparatus 100 synthesizes the output distance image generated based on the input distance image captured by the stereo camera 6 with the output image generated based on the input image captured by the camera 2. .. Therefore, the image generating apparatus 100 can measure the distance to each feature existing in a relatively wide range around the shovel 60 with relatively high resolution. As a result, the image generating apparatus 100 can generate a highly reliable post-synthesis output image.

Further, the camera 2 and the stereo camera 6 are arranged so that their respective imaging ranges surround the circumference of the shovel 60. Therefore, the image generating apparatus 100 can generate a post-combination output image that represents a state when the periphery of the shovel 60 is viewed from above. As a result, the operator of the shovel 60 can more easily recognize the position of the feature existing around the shovel 60 and the distance to the feature.

Further, the image generation apparatus 100 performs the same image conversion process as the image conversion process performed on the processing target image on the processing target distance image. Therefore, the image generating apparatus 100 can easily associate the coordinates on the output image plane with the coordinates on the output distance image plane.

Further, the image generating apparatus 100 changes the color of the extracted pixel or the frame stepwise or steplessly according to the distance from the stereo camera 6, that is, according to the distance from the shovel 60. Therefore, the operator of the shovel 60 can sensuously determine whether there is a danger of contact.

The image generating apparatus 100 also uses the stereo camera 6 to acquire distance information of a two-dimensional array. Therefore, the image generating apparatus 100 has a heat source, a light source, a light absorbing material, etc. in the imaging range as compared with the case where the distance image sensor configured by the light emitting LED and the light receiving CCD is used to acquire the distance information of the two-dimensional array. Not easily affected by.

Next, with reference to FIGS. 19 to 23, a process in which the image generation device 100 extracts pixels satisfying a predetermined condition from the input distance image and corrects the input distance image (hereinafter, referred to as “first pixel extraction processing”). Will be described). In the present embodiment, the pixel value in the input distance image captured by the rear stereo camera 6B is equal to the installation surface distance or smaller than the installation surface distance.

FIG. 19 is a partial right side view of the shovel 60 equipped with the image generating apparatus 100. In the embodiment shown in FIG. 19, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). The area ZB indicated by the dotted line in FIG. 19 indicates the imaging range of the rear stereo camera 6B.

Further, FIG. 19 shows that an object P3 extending vertically upward from the ground and a raised portion P4 of the ground exist behind the shovel 60. Further, FIG. 19 shows that the distance E1a to the object P3 and the distance E2a to the raised portion P4 are detected by the rear stereo camera 6B.

Further, FIG. 19 shows that if the object P3 and the raised portion P4 were not present, the rear stereo camera 6B detected the installation surface distances E1 and E2 instead of the distances E1a and E2a. ..

Further, in FIG. 19, the difference (E1-E1a) between the installation surface distance E1 and the distance E1a is a distance (hereinafter, the difference when the installation surface distance E1 is larger than the distance E1a is referred to as “first difference distance”) E1b. And the difference (E2-E2a) between the installation surface distance E2 and the distance E2a is the first difference distance E2b.

20 is an enlarged view of the raised portion P4 in FIG. As shown in FIGS. 19 and 20, a line segment connecting a point on the raised portion P4 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle (hereinafter referred to as “installation surface angle”) between the installation surface. ) Form γ4. In this case, the height (hereinafter, “estimated height”) H1b from the installation surface of the raised portion P4 is represented by the first difference distance E2b×sin(γ4). Further, the installation surface angle is set in advance for each pixel in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B, like the installation surface distance.

Also, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, by multiplying the first difference distance, which is the difference between the value of each pixel in the input distance image and the installation surface distance corresponding to each pixel, by the sine of the installation surface angle corresponding to each pixel, The estimated height can be calculated. In the present embodiment, the estimated height H1b of the raised portion P4 can be obtained by multiplying the first difference distance E2b by sin(γ4).

By the way, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B as described above. Therefore, when the output distance image is generated by subjecting the input distance image to the process of changing the color or the brightness according to the distance from the rear stereo camera 6B, if the pixel value in the input distance image is used as it is, the visibility of the output distance image is improved. May be damaged. Specifically, the output distance image represents the ground (installation surface) with a color or brightness that gradually changes as the distance from the shovel 60 increases, which may make it difficult to recognize the existence of the object P3 and the raised portion P4. ..

Therefore, the pixel extracting unit 12 of the image generating apparatus 100 extracts, for example, pixels satisfying a predetermined condition from the input distance image of the rear stereo camera 6B, and corrects the input distance image based on the extraction result. Specifically, the pixel extraction unit 12 extracts, from each pixel in the input distance image, a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel. Then, the pixel extraction unit 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction means 12 maintains the pixel value of the pixel having the pixel value smaller than the installation surface distance as it is. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

In addition, the pixel extraction unit 12 may extract pixels of which the first difference distance is equal to or more than a predetermined distance from the pixels in the input distance image. In this case, the pixel extraction unit 12 replaces the pixel value of the pixel having the first difference distance less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and the pixel having the first difference distance not less than the predetermined distance. The pixel value of is maintained as it is. For example, as shown in FIG. 20, the pixel extraction unit 12 sets the pixel value of the pixel corresponding to the first difference distance E2b to zero because the first difference distance E2b is less than the predetermined distance E2b1. The predetermined distance E2b1 is a distance corresponding to the predetermined height H1. Alternatively, as shown in FIG. 20, the pixel extraction unit 12 determines that the installation surface height H1b is less than the predetermined height H1 and replaces the pixel value of the pixel corresponding to the installation surface height H1b with zero. Good. Alternatively, the pixel extraction unit 12 has a predetermined ratio of the ratio of the first difference distance E2b to the installation surface distance E2 (hereinafter, the ratio when the installation surface distance E2 is larger than the distance E2a is referred to as “first difference ratio”). The pixel value of the corresponding pixel may be replaced with zero when it is determined to be less than. The predetermined distance, the predetermined height, the predetermined ratio, and the like are set in advance for each pixel, each pixel row, or each pixel column in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B. To be done. Further, the predetermined height and the predetermined ratio may be common to all pixels in the input distance image.

FIG. 21 is a diagram showing an example of an output distance image of the rear stereo camera 6B. FIG. 21A shows an output distance image when the pixel values of all the pixels in the input distance image are used as they are without performing the extraction by the pixel extracting means 12. FIG. 21B shows an output distance image when the pixel extracting unit 12 extracts, from the pixels in the input distance image, a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel. In addition, in FIG. 21C, the pixel extraction unit 12 determines that among the pixels in the input distance image, the pixels whose first difference distance is a predetermined distance or more or whose estimated height is a predetermined height or more are displayed. The output distance image when it is extracted is shown.

As shown in FIG. 21A, when the pixels are not extracted, the output distance image represents the installation surface in a manner that the brightness gradually decreases as the distance from the shovel 60 increases, and the output surface of the object P3 and the raised portion P4 is represented. Makes it difficult to understand its existence.

On the other hand, as shown in FIG. 21B, when the pixels having the pixel value smaller than the installation surface distance among the pixels in the input distance image are extracted, the output distance image is a feature existing on the installation surface. Improves visibility. Specifically, in the output distance image of FIG. 21B, the pixel values of all the pixels having the pixel value equal to the installation surface distance are replaced with zero, so that the installation surface of the installation surface is displayed regardless of the distance from the shovel 60. Make color and brightness uniform. As a result, the output distance image of FIG. 21B can make the distance image of the object P3 and the raised portion P4 stand out from the distance image of the installation surface, and improve its visibility.

Further, as shown in FIG. 21C, when the pixels having the first difference distance equal to or larger than the predetermined distance among the pixels in the input distance image are extracted, the output distance image has the installation surface height equal to the predetermined height. The visibility of the above features is improved. Specifically, in the output distance image of FIG. 21C, since the pixel values of all the pixels having the first difference distance less than the predetermined distance are set to zero, the installation surface regardless of the distance from the shovel 60. Make the color and brightness of the features (including the installation surface) whose height is less than the predetermined height uniform. As a result, in the output distance image of FIG. 21C, the distance image of the raised portion P4 whose installation surface height H1b is less than the predetermined height H1 is made into a single color and brightness together with the installation surface distance image. Then, in the output distance image of FIG. 21C, only the distance image of the object P3 whose installation surface height is equal to or higher than the predetermined height H1 is made to stand out from the other distance images, and the visibility can be improved.

In the output distance image of FIG. 21C, the pixel value of the pixel corresponding to the base portion (the portion where the installation surface height is less than the predetermined height H1) of the object P3 is set to zero, and the distance image of that portion is installed. A single color and intensity with the range image of the surface. However, even if the first difference distance is less than the predetermined distance or the installation surface height is less than the predetermined height H1, the pixels corresponding to the portion determined to be a part of the object P3, The pixel value in the input range image may be maintained as it is. Whether or not the specific pixel belongs to a part of the distance image of the object P3 is determined by, for example, the first differential distance or installation surface height of surrounding pixels and the first differential distance or installation surface height of the specific pixel. Can be determined based on.

FIG. 22 is a diagram showing an example of an output image of the rear camera 2B. FIG. 22(A) shows an output image when the output distance image of FIG. 21(A) is superimposed. FIG. 22(B) shows an output image when the output distance image of FIG. 21(B) is superimposed. Further, FIG. 22C shows an output image when the output distance image of FIG. 21C is superimposed.

As shown in FIG. 22(A), the output image on which the output distance image of FIG. 21(A) is superimposed represents the installation surface in such a manner that the brightness gradually decreases as the distance from the shovel 60 increases, and the object P3 and the raised portion are displayed. The presence of P4 cannot be highlighted.

On the other hand, as shown in FIG. 22B, the output image on which the output distance image in which the color and the luminance of the installation surface are made uniform is superposed regardless of the distance from the shovel 60 is the image of the object P3 and the raised portion P4. Make it stand out from the image of the installation surface and improve its visibility. Further, the output image of FIG. 22B superimposes and displays the value of the distance between each of the object P3 and the raised portion P4 and the rear stereo camera 6B. This makes it possible to further emphasize the existence of the object P3 and the raised portion P4. The value of the distance between the object P3 and the rear stereo camera 6B is, for example, a value corrected by the distance correction unit 13. Further, the value of the distance between the object P3 and the rear stereo camera 6B may be superimposed and displayed only when the value is a predetermined value or less. This is because the presence of a feature located near the shovel 60 is further emphasized. The brightness or color of the value of the distance (pixel value) is determined according to the magnitude of the distance (pixel value). For example, the color of the value of the distance (pixel value) changes like purple, blue, green, yellow, and red as the distance (pixel value) decreases. The value of the distance may be the minimum value of the pixel values of the pixels included in the distance image area of the object P3 in the input distance image, which is extracted by the pixel extracting unit 12. The same applies to the value of the distance between the raised portion P4 and the rear stereo camera 6B.

Further, as shown in FIG. 22C, an output distance in which the color and the brightness of the features (including the installation surface) whose installation surface height is less than the predetermined height are uniform regardless of the distance from the shovel 60. The output image on which the image is superimposed improves the visibility of the object P3. Specifically, the output image makes only the upper part of the image of the object P3 whose installation surface height is equal to or higher than the predetermined height stand out from other images, and improves the visibility. Further, the output image in FIG. 22C superimposes and displays only the value of the distance between the object P3 whose installation surface height is equal to or higher than the predetermined height and the rear stereo camera 6B. Thereby, only the existence of the object P3 whose installation surface height is equal to or higher than the predetermined height can be further emphasized.

Further, in the above description, the image generating apparatus 100 displays the output distance image of the rear stereo camera 6B on the display unit 5 by superimposing it on the output image of the rear camera 2B. However, the image generating apparatus 100 may display only the output distance image on the display unit 5, or may display the output distance image and the output image side by side on the display unit 5.

Further, the image generating apparatus 100 is shown in FIG. 18 instead of displaying the value of the distance between each of the object P3 and the raised portion P4 and the rear stereo camera 6B, or in addition to displaying the value of the distance. Such a frame may be displayed.

Further, the image generating apparatus 100 displays an index (not shown) such as a rod-shaped indicator with a distance scale indicating the relationship between the distance (pixel value) and the color or the brightness at the end of the output distance image or the output image. May be.

Next, the flow of the first pixel extraction processing will be described with reference to FIG. Note that FIG. 23 is a flowchart showing the flow of the first pixel extraction processing. The pixel extracting unit 12 of the image generating apparatus 100 executes the first pixel extracting process, for example, every time the input distance image is acquired.

First, the pixel extraction unit 12 reads out and acquires the installation surface distance for one pixel (hereinafter, referred to as “target pixel”) in the input distance image from the ROM or the like (step S11). The pixel extraction means 12 may read and acquire the installation surface angle from a ROM or the like.

Then, the pixel extraction means 12 calculates the first difference distance by subtracting the pixel value of the target pixel from the installation surface distance (step S12). The pixel extraction means 12 may calculate the first difference ratio based on the calculated first difference distance and the installation surface distance, or the estimated height based on the calculated first difference distance and the installation surface angle. May be calculated.

After that, the pixel extracting means 12 compares the calculated first difference distance with the predetermined distance stored in the ROM or the like (step S13). The pixel extracting means 12 may compare the calculated first difference ratio with a predetermined ratio stored in the ROM or the like, or may compare the calculated estimated height with the predetermined height stored in the ROM or the like. Good.

When the first difference distance is equal to or greater than the predetermined distance (YES in step S13), the pixel extraction unit 12 extracts the target pixel as a pixel that maintains the current pixel value as it is. The same applies when the first difference ratio is equal to or higher than the predetermined ratio, or when the estimated height is equal to or higher than the predetermined height.

On the other hand, when the first difference distance is less than the predetermined distance (NO in step S13), the pixel extracting unit 12 does not extract the target pixel as a pixel that maintains the current pixel value as it is. Specifically, the pixel extracting means 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S14). The same applies when the first difference ratio is less than the predetermined ratio or when the estimated height is less than the predetermined height.

In the above-described embodiment, the process of replacing the pixel value of the target pixel that has not been extracted with the predetermined value is performed when the necessity of extraction is determined for each target pixel, but for a plurality of or all target pixels. It may be collectively performed after determining the necessity of extraction.

After that, the pixel extraction unit 12 determines whether or not the extraction necessity is determined for all the pixels in the acquired input distance image (step S15).

If it is determined that the determination as to whether extraction is necessary has not been performed for all pixels (NO in step S15), the pixel extraction unit 12 executes the processing in step S11 and subsequent steps for another target pixel.

When it is determined that the extraction necessity is determined for all the pixels (YES in step S15), the pixel extraction unit 12 completes the first pixel extraction process of this time.

With the above configuration, the image generating apparatus 100 can improve the visibility of the distance image regarding the feature that the shovel 60 may contact by omitting the detailed display of the distance image regarding the installation surface.

In addition, the image generation apparatus 100 omits the detailed display of the feature whose installation surface height is less than the predetermined height, so that the visibility of a feature (e.g., a worker) whose installation surface height is the predetermined height or more. Can be improved.

As a result, the image generating apparatus 100 can more reliably prevent the shovel 60 from colliding with a feature.

Next, with reference to FIGS. 24 to 28, another example of a process in which the image generation device 100 extracts pixels satisfying a predetermined condition from the input distance image and corrects the input distance image (hereinafter, referred to as “second pixel”). Extraction process"). In this embodiment, the pixel value in the input distance image captured by the rear stereo camera 6B is equal to or larger than the installation surface distance.

FIG. 24 is a partial right side view of the shovel 60 equipped with the image generating apparatus 100, and corresponds to FIG. 19. In the embodiment shown in FIG. 24, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). The area ZB indicated by the dotted line in FIG. 24 indicates the imaging range of the rear stereo camera 6B.

Further, FIG. 24 shows that a shallow depression portion PK1 and a deep depression portion (cliff) PK2 are present behind the shovel 60. Further, FIG. 24 shows that the distance E3a to the bottom of the shallow depression PK1 and the distance E4a to the bottom of the deep depression PK2 are detected by the rear stereo camera 6B.

Further, in FIG. 24, if the shallow depression portion PK1 and the deep depression portion PK2 do not exist, the rear stereo camera 6B detects the installation surface distances E3 and E4 instead of the distances E3a and E4a. Indicates that.

Further, in FIG. 24, the difference (E3a−E3) between the distance E3a and the installation surface distance E3 is a distance (hereinafter, the difference when the installation surface distance E3 is smaller than the distance E3a is referred to as “second difference distance”) E3b. And the difference (E4a-E4) between the distance E4a and the installation surface distance E4 is the second difference distance E4b.

FIG. 25 is an enlarged view of the shallow depression PK1 in FIG. As shown in FIGS. 24 and 25, the line segment connecting the bottom point of the shallow depression PK1 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an installation surface angle γ5 with the installation surface. .. In this case, the depth (hereinafter referred to as “estimated depth”) DP1b from the installation surface of the shallow depression PK1 is represented by the second differential distance E3b×sin(γ5).

Also, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, by multiplying the second difference distance, which is the difference between the value of each pixel in the input distance image and the installation surface distance corresponding to each pixel, by the sine of the installation surface angle corresponding to each pixel, The estimated depth can be obtained. In the present embodiment, the estimated depth DP1b of the shallow depression PK1 can be obtained by multiplying the second difference distance E3b by sin(γ5).

By the way, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B as described above. Therefore, when the output distance image is generated by subjecting the input distance image to the process of changing the color or the brightness according to the distance from the rear stereo camera 6B, if the pixel value in the input distance image is used as it is, the visibility of the output distance image is improved. May be damaged. Specifically, the output distance image represents the ground (installation surface) with a color or brightness that gradually changes as the distance from the shovel 60 increases, making it difficult to see the presence of the shallow depression PK1 and the deep depression PK2. There is a risk.

Therefore, the pixel extracting unit 12 of the image generating apparatus 100 extracts, for example, pixels satisfying a predetermined condition from the input distance image of the rear stereo camera 6B, and corrects the input distance image based on the extraction result. Specifically, the pixel extracting unit 12 extracts, from each pixel in the input distance image, a pixel having a pixel value larger than the installation surface distance corresponding to each pixel. Then, the pixel extraction unit 12 replaces the pixel value of the pixel having the pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction unit 12 maintains the pixel value of the pixel having the pixel value larger than the installation surface distance as it is. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

Further, the pixel extraction means 12 may extract pixels of which the second difference distance is equal to or more than a predetermined distance from each pixel in the input distance image. In this case, the pixel extraction unit 12 replaces the pixel value of the pixel having the second difference distance less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and the pixel having the second difference distance not less than the predetermined distance. The pixel value of is maintained as it is. For example, as shown in FIG. 25, the pixel extraction unit 12 sets the pixel value of the pixel corresponding to the second difference distance E3b to zero because the second difference distance E3b is less than the predetermined distance E3b1. The predetermined distance E3b1 is a distance corresponding to the predetermined depth DP1. Alternatively, as shown in FIG. 25, the pixel extraction unit 12 determines that the installation surface depth DP1b is less than the predetermined depth DP1 and replaces the pixel value of the pixel corresponding to the installation surface depth DP1b with zero. Good. Alternatively, the pixel extraction unit 12 has a ratio of the second difference distance E3b to the distance E3a (hereinafter, the ratio when the installation surface distance E3 is smaller than the distance E3a is referred to as “second difference ratio”) is less than a predetermined ratio. It may be determined that there is, and the pixel value of the corresponding pixel may be replaced with zero. The predetermined distance, the predetermined depth, the predetermined ratio, and the like are set in advance for each pixel, each pixel row, or each pixel column in the input distance image of the rear stereo camera 6B based on the installation position and the installation angle of the rear stereo camera 6B. To be done. Further, the predetermined depth and the predetermined ratio may be common to all pixels in the input distance image.

FIG. 26 is a diagram showing an example of an output distance image of the rear stereo camera 6B. FIG. 26A shows an output distance image when the pixel values of all the pixels in the input distance image are used as they are without performing the extraction by the pixel extracting means 12. FIG. 26B shows an output distance image when the pixel extracting unit 12 extracts, from the pixels in the input distance image, a pixel having a pixel value larger than the installation surface distance corresponding to each pixel. In addition, in FIG. 26C, the pixel extraction unit 12 determines that among the pixels in the input distance image, the pixels having the second difference distance equal to or larger than the predetermined distance or the pixels having the estimated depth equal to or larger than the predetermined depth. The output distance image when it is extracted is shown.

As shown in FIG. 26A, when the pixel extraction is not performed, the output distance image represents the installation surface in a manner that the brightness gradually decreases as the distance from the shovel 60 increases, and the shallow depression portion PK1 and the deep depression portion are displayed. Makes it difficult to understand the existence of part PK2.

On the other hand, as shown in FIG. 26B, when the pixels having the pixel value larger than the installation surface distance are extracted from the respective pixels in the input distance image, the output distance image includes the depressions existing below the installation surface. Improves visibility. Specifically, in the output distance image of FIG. 26B, since the pixel values of all the pixels having the pixel value equal to the installation surface distance are replaced with zero, regardless of the distance from the shovel 60, the installation surface image of Make color and brightness uniform. As a result, in the output distance image of FIG. 26B, the distance images of the shallow depression portion PK1 and the deep depression portion PK2 can be distinguished from the distance image of the installation surface, and the visibility can be improved.

Further, as shown in FIG. 26C, when the pixels having the second difference distance equal to or larger than the predetermined distance among the pixels in the input distance image are extracted, the output distance image has the installation surface depth equal to the predetermined depth. The visibility of the above features is improved. Specifically, in the output distance image of FIG. 26C, since the pixel values of all the pixels having the second difference distance less than the predetermined distance are set to zero, regardless of the distance from the shovel 60, the installation surface Make the color and brightness of features (including the installation surface) whose depth is less than the specified depth uniform. As a result, in the output range image of FIG. 26C, the range image of the shallow depression PK1 having the installation surface depth DP1b less than the predetermined depth DP1 is made into a single color and brightness together with the range image of the installation surface. Then, in the output distance image of FIG. 26C, only the distance image of the deep depression PK2 having the installation surface depth equal to or greater than the predetermined depth DP1 can be distinguished from the other distance images, and the visibility can be improved. it can.

In the output distance image of FIG. 26C, the pixel value of the pixel corresponding to the edge portion (the portion where the installation surface depth is less than the predetermined depth DP1) of the deep depression PK2 is set to zero, and the distance image of that portion is obtained. With a range image of the installation surface into a single color and brightness. However, for the pixel corresponding to the portion that is determined to be a part of the deep depression PK2, the second difference distance is less than the predetermined distance, or the installation surface depth is less than the predetermined depth DP1. Alternatively, the pixel value in the input range image may be maintained as it is. Whether or not the specific pixel belongs to a part of the distance image of the deep depression PK2 is determined by, for example, the second differential distance or installation surface depth with respect to surrounding pixels and the second differential distance or installation surface with respect to the specific pixel. It can be judged based on the depth.

FIG. 27 is a diagram showing an example of an output image of the rear camera 2B. FIG. 27(A) shows an output image when the output distance image of FIG. 26(A) is superimposed. FIG. 27(B) shows an output image when the output distance image of FIG. 26(B) is superimposed. 27C shows an output image when the output distance image of FIG. 26C is superimposed.

As shown in FIG. 27(A), the output image on which the output distance image of FIG. 26(A) is superimposed represents the installation surface in a manner in which the brightness gradually decreases as the distance from the excavator 60 increases, and the shallow depression PK1 and The existence of the deep depression PK2 cannot be emphasized.

On the other hand, as shown in FIG. 27B, the output image obtained by superimposing the output distance image in which the color and the luminance of the installation surface are made uniform regardless of the distance from the shovel 60 is the shallow depression portion PK1 and the deep depression portion PK2. Image is distinguished from the image of the installation surface to improve its visibility. Further, the output image of FIG. 27B superimposes and displays the value of the distance between each of the shallow depression portion PK1 and the deep depression portion PK2 and the rear stereo camera 6B. Thereby, the existence of the shallow depression portion PK1 and the deep depression portion PK2 can be further emphasized. For the value of the distance between the shallow depression PK1 and the rear stereo camera 6B, for example, a value corrected by the distance correction unit 13 is adopted. The value of the distance between the shallow depression PK1 and the rear stereo camera 6B may be displayed in a superimposed manner only when the value is less than a predetermined value. This is because the presence of a depressed portion near the shovel 60 is further emphasized. The brightness or color of the value of the distance (pixel value) is determined according to the magnitude of the distance (pixel value). For example, the color of the value of the distance (pixel value) changes like purple, blue, green, yellow, and red as the distance (pixel value) decreases. The value of the distance may be the minimum value of the pixel values of the pixels extracted by the pixel extraction unit 12 and included in the distance image area of the shallow depression PK1 in the input distance image. The same applies to the value of the distance between the deep depression PK2 and the rear stereo camera 6B.

Further, as shown in FIG. 27C, an output distance in which the color and the brightness of the features (including the installation surface) whose installation surface depth is less than the predetermined depth are uniform regardless of the distance from the shovel 60. The output image on which the images are superimposed improves the visibility of the deep depression PK2. Specifically, the output image makes only the image of the deep depression PK2 having the installation surface depth equal to or larger than the predetermined depth stand out from the other images, and improves the visibility. Further, the output image of FIG. 27C displays only the value of the distance between the deep depression PK2 having the installation surface depth equal to or greater than the predetermined depth and the rear stereo camera 6B in a superimposed manner. As a result, only the presence of the deep depression PK2 having the installation surface depth equal to or greater than the predetermined depth can be further emphasized.

Further, in the above description, the image generating apparatus 100 displays the output distance image of the rear stereo camera 6B on the display unit 5 by superimposing it on the output image of the rear camera 2B. However, the image generating apparatus 100 may display only the output distance image on the display unit 5, or may display the output distance image and the output image side by side on the display unit 5.

Further, the image generating apparatus 100 does not display the value of the distance between each of the shallow depression PK1 and the deep depression PK2 and the rear stereo camera 6B, or in addition to the display of the value of the distance, A frame as shown in 18 may be displayed.

Next, the flow of the second pixel extraction processing will be described with reference to FIG. Note that FIG. 28 is a flowchart showing the flow of the second pixel extraction processing. The pixel extraction means 12 of the image generation device 100 executes this second pixel extraction processing, for example, every time an input distance image is acquired.

First, the pixel extraction means 12 reads out and acquires the installation surface distance for one pixel (hereinafter, referred to as “target pixel”) in the input distance image from the ROM or the like (step S21). The pixel extraction means 12 may read and acquire the installation surface angle from a ROM or the like.

Then, the pixel extraction means 12 calculates the second difference distance by subtracting the installation surface distance from the pixel value of the target pixel (step S22). The pixel extraction means 12 may calculate the second difference ratio based on the calculated second difference distance and the pixel value of the target pixel thereof, or based on the calculated second difference distance and the installation surface angle. The estimated depth may be calculated.

After that, the pixel extracting means 12 compares the calculated second difference distance with the predetermined distance stored in the ROM or the like (step S23). The pixel extracting means 12 may compare the calculated second difference ratio with a predetermined ratio stored in the ROM or the like, or may compare the calculated estimated depth with the predetermined depth stored in the ROM or the like. Good.

When the second difference distance is equal to or greater than the predetermined distance (YES in step S23), the pixel extraction unit 12 extracts the target pixel as a pixel that maintains the current pixel value as it is. The same applies when the second difference ratio is greater than or equal to the predetermined ratio, or when the estimated depth is greater than or equal to the predetermined depth.

On the other hand, when the second difference distance is less than the predetermined distance (NO in step S23), the pixel extraction unit 12 does not extract the target pixel as a pixel that maintains the current pixel value as it is. Specifically, the pixel extraction unit 12 replaces the pixel value of the target pixel with a predetermined value stored in the ROM or the like (step S24). The same applies when the second difference ratio is less than the predetermined ratio or when the estimated depth is less than the predetermined depth.

The process of replacing the pixel value of the target pixel that has not been extracted with the predetermined value may be performed collectively after determining whether or not extraction is necessary for a plurality of or all target pixels.

After that, the pixel extraction means 12 determines whether or not the extraction necessity is determined for all the pixels in the acquired input distance image (step S25).

When it is determined that the extraction necessity determination has not been performed for all pixels (NO in step S25), the pixel extraction unit 12 executes the processing in step S21 and subsequent steps for another target pixel.

When it is determined that the necessity of extraction is determined for all the pixels (YES in step S25), the pixel extracting unit 12 completes the second pixel extracting process of this time.

With the above configuration, the image generating apparatus 100 omits the detailed display of the distance image regarding the installation surface, and thus the distance image of the depression (for example, a road shoulder, a cliff, or the like) that may cause the shovel 60 to slide down. The visibility can be improved.

Further, the image generating apparatus 100 can improve the visibility of the depressed portion having the installation surface depth equal to or greater than the predetermined depth by omitting the detailed display of the depressed portion having the installation surface depth less than the predetermined depth. ..

As a result, the image generating apparatus 100 can more reliably prevent the shovel 60 from sliding down into the depressed portion.

Further, in the above description, the first pixel extraction process and the second pixel extraction process are described separately, but two pixel extraction processes may be combined.

Next, with reference to FIG. 29, a process in which the driving support means 15 executes the driving support function (hereinafter, referred to as “driving support process”) will be described. Note that FIG. 29 is a flowchart showing the flow of the driving support process, and the driving support means 15 repeatedly executes the driving support process at a predetermined cycle while the shovel 60 is traveling.

First, the driving support means 15 detects the moving direction of the shovel 60 based on the operating direction of the traveling operation lever (not shown) of the shovel 60 (step S31).

Then, the driving support means 15 determines whether or not a predetermined feature exists in the moving direction of the shovel 60 (step S32). In the present embodiment, the driving support unit 15 determines whether or not there is a recessed portion that may slide the shovel 60 behind behind the shovel 60 that moves backward based on the input distance image acquired by the rear stereo camera 6B. judge.

When it is determined that there is a depression that may slide the shovel 60 (YES in step S32), the driving support unit 15 executes the driving support function (step S33). In the present embodiment, the driving support means 15 sounds an alarm and displays a warning message on the display unit 5 notifying that there is a depression. Further, the driving support means 15 may reduce the retreat speed of the shovel 60 as the distance to the depressed portion decreases, and may stop the shovel 60 when the distance to the depressed portion falls below a predetermined value. ..

On the other hand, when it is determined that there is no depression that may slide the shovel 60 (NO in step S32), the driving support unit 15 ends the current driving support process without executing the driving support function.

With the above configuration, the image generating apparatus 100 automatically determines the presence or absence of a predetermined feature on the basis of the output of the stereo camera 6, and executes the driving support function as necessary, thereby operating the shovel 60. The safety can be further improved.

Next, a process (hereinafter, referred to as “distance correction process”) in which the image generation device 100 corrects a pixel value (distance) of an input distance image will be described with reference to FIGS. 30 to 33.

30 is a diagram showing a shovel 60 equipped with the image generating apparatus 100, FIG. 30(A) is a top view thereof, and FIG. 30(B) is a partial right side view thereof. In the embodiment shown in FIG. 30, the shovel 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). The area CB indicated by the one-dot chain line in FIG. 30 indicates the imaging range of the rear camera 2B. Further, a region ZB indicated by a dotted line in FIG. 30 shows an imaging range of the rear stereo camera 6B.

As shown in FIG. 30, behind the shovel 60, there are two objects P1 and P2 extending vertically upward from the ground. The two objects P1 and P2 are rod-shaped objects having different heights and exist in different directions when viewed from the rear stereo camera 6B. Specifically, as shown in FIG. 30B, the height of the object P1 is lower than the height of the object P2. However, the shortest distance between each of the two objects P1 and P2 and the rear stereo camera 6B is equal to the distance D. This is because the lower object P1 is closer to the shovel 60 than the higher object P2.

Further, as shown in FIG. 30B, a line segment connecting the closest point on the object P1 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle γ1 with the horizontal plane. Similarly, the line segment connecting the closest point on the object P2 viewed from the rear stereo camera 6B and the rear stereo camera 6B forms an angle γ2 with the horizontal plane. In this case, the horizontal distance D1 between the object P1 and the plane including the rear surface of the shovel 60 is represented by D×cos(γ1), and the horizontal distance D2 between the object P2 and the plane including the rear surface of the shovel 60 is , D×cos(γ2). The horizontal distance D2 is larger than the horizontal distance D1.

The pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B. Therefore, when the pixel value in the input distance image is used as it is, in the process of changing the display color or the brightness according to the distance from the rear stereo camera 6B, the top of the object P1 and the top of the object P2 have the same color or the same brightness. Will be displayed. Even if the object P1 is closer to the shovel 60 than the object P2 as described above, this display erroneously recognizes to the operator that the objects P1 and P2 exist at the same distance from the rear surface of the shovel 60. May cause

Such a situation arises because the rear stereo camera 6B is attached to the upper part of the upper revolving structure 63 and images obliquely downward. That is, the distance recognized by the operator is the horizontal distance from the shovel 60, while the distance acquired by the stereo camera 6 is a straight distance from the stereo camera 6. The mounting position of the rear stereo camera 6B is set above the upper swing body 63 so that the sensor can be protected from contact with surrounding features.

Therefore, the distance correction unit 13 of the image generating apparatus 100 includes, for example, the pixel value indicating the distance between each of the objects P1 and P2 and the rear stereo camera 6B, including each of the objects P1 and P2 and the rear surface of the shovel 60. Correct to a pixel value that represents the horizontal distance to the plane.

Specifically, the distance correction unit 13 determines the rear stereo camera 6B and an arbitrary point on the object P1 from the coordinates of pixels in the input distance image corresponding to the arbitrary point on the object P1 viewed from the rear stereo camera 6B. The angle γ formed by the line segment connecting the points with the horizontal plane is derived. Since the mounting position and the mounting angle of the rear stereo camera 6B are known, the distance correcting unit 13 can derive the angle γ from the coordinates of the pixel.

After that, the distance correction unit 13 determines the pixel value of the pixel, that is, the rear stereo camera 6B and the object P1 in the cosine cos(γ) of the angle γ derived from the coordinates of the pixel corresponding to an arbitrary point on the feature. Multiply the distance to any point above. In this way, the distance correction unit 13 calculates the horizontal distance between the rear stereo camera 6B and an arbitrary point on the object P1, and calculates the pixel value of the pixel corresponding to the arbitrary point as the calculated horizontal distance. Replace with.

Similarly, the distance correction unit 13 derives an angle γ formed by the line segment connecting the rear stereo camera 6B and an arbitrary point on the object P2 with the horizontal plane. Then, the cosine cos(γ) of the angle γ is multiplied by the pixel value thereof, that is, the distance between the rear stereo camera 6B and an arbitrary point on the object P2. Then, the horizontal distance between the rear stereo camera 6B and an arbitrary point on the object P2 is calculated, and the pixel value of the pixel corresponding to the arbitrary point is replaced with the calculated horizontal distance.

Here, the effect of the correction by the distance correction unit 13 will be described with reference to FIGS. 31 and 32.

FIG. 31 shows an output distance image based on an input distance image that has not been subjected to distance correction processing (hereinafter referred to as “pre-correction output distance image”) and an output distance based on the input distance image that has been subjected to distance correction processing. FIG. 31A is a comparison diagram showing a difference from an image (hereinafter, referred to as “corrected output distance image”), FIG. 31(A) shows an uncorrected output distance image, and FIG. 31(B) is a corrected output distance image. Indicates. In this embodiment, pixel extraction processing has already been performed on the input distance image.

Further, FIG. 32 shows a post-combination output image in which a pre-correction output distance image is combined with an output image generated based on the input image of the rear camera 2B (hereinafter referred to as “pre-correction post-combination output image”). FIG. 32A is a comparison diagram showing a difference from a post-composition output image in which the corrected output distance image is combined with the output image (hereinafter, referred to as “corrected post-composition output image”), and FIG. 32B shows the post-output image, and FIG. 32B shows the corrected post-synthesis post-output image.

As shown in FIGS. 31A and 32A, the object P1 and the object P2 are represented by the same color scheme in the pre-correction output distance image and the pre-correction post-composition output image. Specifically, both the objects P1 and P2 are displayed whiter (thinner) closer to the top and blacker (darker) closer to the bottom. Therefore, it is difficult for the operator to determine which object is closer to the shovel 60.

On the other hand, as shown in FIGS. 31B and 32B, in the corrected output distance image and the corrected combined output image, the object P1 and the object P2 are represented by different colors. Specifically, the object P1 is displayed lightly on the whole, and the object P2 is displayed dark on the whole. Therefore, the operator can easily recognize that the object P1 is closer to the shovel 60 than the object P2.

FIG. 33 is another example of the comparison diagram showing the difference between the pre-correction combined output image and the corrected post-composition output image, and shows an example in which a frame is displayed in a superimposed manner.

As shown in FIG. 33A, in the pre-correction combined output image, when the minimum pixel value of the pixels included in the feature area is adopted as the representative value of the feature area, the object P1 and the object P2 are Are displayed as being at a distance of 1 meter from the shovel 60.

On the other hand, as shown in FIG. 33B, in the corrected combined output image, the horizontal distances D1 and D2 between the objects P1 and P2 and the plane including the rear surface of the shovel 60 are displayed. That is, it is displayed that the object P1 is at a distance of 1 meter from the rear surface of the shovel 60 and the object P2 is at a distance of 1.5 meters from the rear surface of the shovel 60. Therefore, the operator can easily recognize that the object P1 is closer to the shovel 60 than the object P2.

With the above configuration, the image generating apparatus 100 corrects the pixel value so that the pixel value in the input distance image captured by the stereo camera 6 represents the horizontal distance from the shovel 60. As a result, the image generating apparatus 100 can inform the operator of the situation of the surrounding features more easily.

Further, in the above-described embodiment, the distance correction unit 13 individually corrects the pixel value of the pixel extracted by the pixel extraction unit 12. However, the present invention is not limited to this. For example, the distance correction unit 13 collects predetermined corrections for each pixel row, each pixel column, or each predivided region in the input distance image plane without calculating the horizontal distance using the cosine as described above. You may do it by doing. This is because the same effect as in the case where the horizontal distance is calculated using the cosine is obtained by simple correction while suppressing the calculation load. Specifically, for example, the distance correction unit 13 increases the distance increase correction for pixel rows above the input distance image plane, or decreases the distance for pixel rows below the input distance image plane. In order to increase the correction, the correction with a predetermined correction amount may be collectively executed.

In addition, in the above-described embodiment, the distance correction unit 13 performs the correction when the rear stereo camera 6B is attached to the upper rear surface of the upper swing body 63 and images obliquely downward. However, the present invention is not limited to this. For example, the distance correction unit 13 may perform correction when the stereo camera captures an image in the horizontal direction, or may perform correction when the stereo camera captures an image obliquely above.

Further, in the above-described embodiment, the distance correction unit 13 uses the plane including the rear surface of the shovel 60 as a reference, and determines the pixel value of the pixel included in the feature area between the feature and the plane including the rear surface of the shovel 60. It is corrected to a pixel value that represents the horizontal distance between them. However, the present invention is not limited to this. For example, the distance correction unit 13 uses the turning axis PV of the shovel 60 as a reference, and sets the pixel value of the pixel included in the feature area to a pixel value that represents the horizontal distance between the feature and the turning axis PV of the shovel 60. It may be corrected to.

Further, the distance correction unit 13 may correct the pixel value of the pixel included in the feature area to a pixel value representing the shortest horizontal distance between the feature and the shovel 60. In this case, the distance correction unit 13 selects a reference from a plane including the right side surface of the shovel 60, a plane including the left side surface of the shovel 60, a plane including the rear surface of the shovel 60, and the like according to the actual position of the feature. ..

The shortest horizontal distance may be the shortest horizontal distance in the moving direction of the shovel 60, that is, the direction parallel to the front-rear direction of the lower traveling body 61. In this case, the pixel value of the feature that is not in the moving direction of the shovel 60 is corrected to the maximum value (infinity) even if the feature is near the shovel 60. As a result, the operator can more easily recognize the feature in the moving direction of the shovel 60.

Although the preferred embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various modifications and substitutions may be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

For example, in the above-described embodiment, the image generating apparatus 100 adopts the cylindrical space model MD as the space model, but may adopt a space model having another columnar shape such as a polygonal column, and the bottom surface and A space model composed of two side surfaces may be adopted, or a space model having only side surfaces may be adopted.

The image generating apparatus 100 is mounted on a self-propelled excavator with movable members such as a bucket, an arm, a boom, and a turning mechanism together with a camera and a stereo camera, and presents a surrounding image to the operator of the excavator. An operation support system for supporting movement and operation of those movable members is configured. However, the image generating apparatus 100 is used for a work machine such as a forklift or an asphalt finisher that does not have a turning mechanism, or a work machine that has a movable member but does not run by itself such as an industrial machine or a fixed crane. You may comprise the operation support system mounted with a camera and a stereo camera, and supporting operation of those work machines.

Further, in the above-described embodiment, the distance image synthesizing unit 14 generates the output distance image and then synthesizes the generated output distance image with the output image. However, the present invention is not limited to this. The distance image synthesizing unit 14, for example, synthesizes the input image and the input distance image to generate a post-synthesis input image, generates a post-synthesis processing target image on which information regarding the distance image is displayed in a superimposed manner, and An output image may be generated. Further, the distance image synthesizing unit 14 may generate the post-synthesis output image after generating the post-synthesis processing target image by synthesizing the processing target image and the processing target distance image, for example.

Further, in the above-described embodiment, the image generation device 100 includes the camera 2 and the stereo camera 6 separately. Then, the image generating apparatus 100 generates one output image (viewpoint conversion image) based on the plurality of input images from the camera 2 and outputs one output image based on the plurality of input distance images from the stereo camera 6. A range image (viewpoint conversion range image) is generated. Then, the image generating apparatus 100 combines the output distance image with the output image. However, the present invention is not limited to this configuration. For example, the image generating apparatus 100 generates an output image (viewpoint conversion image) based on three images output by one imaging mechanism in each of the left stereo camera 6L, the rear stereo camera 6B, and the right stereo camera 6R. In addition, one output distance image (viewpoint conversion distance image) may be generated based on the three input distance images of the three stereo cameras. In this case, the camera 2 can be omitted. Further, in this case, the left side stereo camera 6L, the rear side stereo camera 6B, and the right side stereo camera 6R are installed substantially at the center of each of the left side surface, the rear surface, and the right side surface of the upper swing body 63 so that the imaging range is not biased. It is desirable to be done. Therefore, a plurality of sets of image pickup mechanisms in each stereo camera are arranged side by side with a predetermined interval in the vertical direction. This is because, regardless of the size of the juxtaposed space, it is easy to install the stereo camera at the approximate center of each surface. Further, if a plurality of sets of image pickup mechanisms are arranged side by side in the lateral direction, it may be difficult to install the stereo camera substantially at the center of each surface depending on the arrangement intervals.

Further, in the above-described embodiment, the stereo camera 6 derives the distance between the stereo camera 6 and the object shown in each pixel of the basic image based on the parallax between the images output by each of the two imaging mechanisms. Then, the stereo camera 6 substitutes the derived distance into the value of each pixel of the basic image to generate an input distance image as distance information of the two-dimensional array, and outputs the generated input distance image to the control unit 1. Output. Then, the control unit 1 uses the plurality of input distance images from the plurality of stereo cameras 6 to generate one output distance image. However, the present invention is not limited to this configuration. For example, the image generation device 100 generates a first output image (first viewpoint conversion image) based on the three images output by the first imaging mechanism in each of the three stereo cameras, and the three stereo cameras The second output image (second viewpoint conversion image) is generated based on the three images output by the second imaging mechanism in each of the above. Then, the image generation device 100 determines the distance between the object captured in each pixel of the first viewpoint-converted image and the predetermined reference point based on the parallax between the first viewpoint-converted image and the second viewpoint-converted image. The output range image may be generated by deriving the input range image without using the input range image.

The surrounding monitoring device of Patent Document 1 generates an illustration image of the shovel and its surroundings seen from the sky, and displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. In addition, the surrounding monitoring device of Patent Document 1 displays stationary obstacles and moving obstacles in different mode illustrations. In this way, the surroundings monitoring device of Patent Document 1 enables the driver to be easily informed of the status of obstacles around the shovel. However, the laser radar described in Patent Document 1 irradiates an object with one laser beam and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between the extremely limited portion of the object surface and itself, and the detection result roughly determines the position of the illustration representing the object. It is only used for. Therefore, the output image presented by the surroundings monitoring device described in Patent Document 1 is still insufficient to convey the situation of obstacles around the shovel to the driver in an easy-to-understand manner.

1... control unit 2... camera 2L... left side camera 2R... right side camera 2B... rear camera 3... input unit 4... storage unit 5... display unit 6... Stereo camera 6L... Left side stereo camera 6R... Right side stereo camera 6B... Rear stereo camera 10... Coordinates matching means 11... Image generating means 12... Pixel extracting means 13. ..Distance correction means 14...distance image combining means 15...driving support means 40...input image/spatial model correspondence map 41...spatial model/processing target image correspondence map 42...processing target image Output image correspondence map 60... excavator 61... lower traveling body 62... revolving mechanism 63... upper revolving body 64... cab 100... image generating device

In order to achieve the above object, an excavator according to an embodiment of the present invention includes a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, and a plurality of upper revolving bodies. possess a stereo camera, a, each of the plurality of said stereoscopic camera comprises a plurality of imaging mechanism in a single casing.

Claims (7)

An undercarriage,
An upper revolving structure mounted on the lower traveling structure so as to be freely rotatable,
A plurality of stereo cameras arranged on the upper swing body,
Shovel. The first stereo camera is disposed behind the upper swing body,
The second stereo camera is arranged laterally of the upper swing body,
The shovel according to claim 1. The stereo camera is arranged diagonally downward on the upper surface of the upper swing body,
The shovel according to claim 1. The stereo camera does not protrude from the side surface of the upper swing body,
The shovel according to any one of claims 1 to 3. The imaging ranges of the plurality of stereo cameras have areas overlapping with each other,
The shovel according to any one of claims 1 to 4. The stereo camera is a device that integrally has an imaging function of a plurality of cameras,
The shovel according to any one of claims 1 to 5. The stereo camera is composed of a plurality of separate and independent cameras,
The shovel according to any one of claims 1 to 5.