JP6257919B2 - Excavator - Google Patents

Description

The present invention relates to a shovel that generates an output image using distance information of a two-dimensional array.

  2. Description of the Related Art Conventionally, there has been known an ambient monitoring device that monitors the surroundings while acquiring the distance and direction of an object existing around the excavator using a laser radar attached to the upper revolving unit of the excavator (for example, Patent Documents). 1).

  The surrounding monitoring apparatus determines that an object is a stationary obstacle when the distance and direction of the object to the excavator do not change over a predetermined time, and determines that the object is a moving obstacle when the distance and direction change. Then, the surroundings monitoring device generates an illustration image of the excavator and the surroundings viewed from above, and displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. In addition, the surrounding monitoring device displays the stationary obstacle and the moving obstacle with different illustrations.

  In this way, the surrounding monitoring device can inform the driver of the state of obstacles around the excavator in an easy-to-understand manner.

JP 2008-163719 A

  However, the laser radar described in Patent Document 1 irradiates an object with one laser beam and detects the distance and direction of the object from the reflected light. Therefore, the laser radar described in Patent Document 1 only detects the distance between the extremely limited portion of the object surface and itself, and the detection result roughly determines the position of the illustration representing the object. It is only used for Therefore, the output image presented by the surrounding monitoring device described in Patent Document 1 is still insufficient to convey the state of the obstacle around the excavator to the driver in an easily understandable manner.

In view of the above-described points, an object of the present invention is to provide an excavator that can easily convey the situation of surrounding objects to an operator.

In order to achieve the above-described object, an excavator according to an embodiment of the present invention includes a traveling body, a revolving body that is turnably mounted on the traveling body, a plurality of imaging mechanisms attached to the revolving body, Display means for displaying a bird's-eye view image that displays the periphery of the excavator, which is generated based on a lateral image of the revolving structure and an image behind the revolving structure, and based on the outputs of the plurality of imaging mechanisms, Control means for determining the presence or absence of an object having a predetermined height around the body, and the bird's-eye view image of the swivel body is displayed at the same time obliquely behind the swivel body. A side image and a rear image of the revolving structure are combined and displayed, and the control means includes the object at a position where the imaging ranges of the plurality of imaging mechanisms overlap. And determine that the duplicate The position includes a slanting rear side of the swivel body in which an image of a side of the swivel body and an image behind the swivel body are combined, and the control means outputs an alarm when it is determined that an object exists Or display a warning on the overhead image, turn on or blink a warning light, decelerate the shovel, limit the operating speed of the shovel, or operate the shovel. Stop .

With the above-described means, the present invention can provide an excavator that can convey the situation of surrounding objects to the operator in an easy-to-understand manner.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of the shovel mounted with an image generation apparatus. It is a figure which shows an example of the space model on which an input image is projected. It is a figure which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a top view (the 1) of an excavator carrying an image generating device. It is a figure which shows each input image of the three cameras mounted in the shovel, and the output image produced | generated using those input images. It is a figure for demonstrating the image loss prevention process which prevents the loss | disappearance of the object in the overlapping area | region of each imaging range of two cameras. FIG. 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying image loss prevention processing to the output image of FIG. 12. It is a figure which shows each input distance image of the three stereo cameras mounted in the shovel, and the output distance image produced | generated using those input distance images. FIG. 16 is a comparison diagram showing the difference between the output distance image shown in FIG. 15 and the output distance image obtained by applying the distance image disappearance prevention process to the output distance image of FIG. 15. It is a figure which shows an example of the output image after a synthesis | combination obtained by synthesize | combining an output distance image with an output image. It is a figure which shows another example of the output image after a synthesis | combination obtained by synthesize | combining an output distance image with an output image. It is a partial right view of the shovel carrying an image generating apparatus. It is an enlarged view of the protruding part in FIG. It is a figure which shows the example of an output distance image. It is a figure which shows the example of an output image. It is a flowchart which shows the flow of a pixel extraction process. It is a flowchart which shows the flow of a driving assistance process. It is a figure which shows the shovel mounted with an image generation apparatus. It is a contrast diagram showing the difference between the output distance image before correction | amendment, and the corrected output distance image. FIG. 9 is a comparison diagram (part 1) showing a difference between a pre-correction combined output image and a corrected combined output image. FIG. 10 is a comparison diagram (part 2) showing a difference between a pre-correction composite output image and a corrected composite output image.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

  The image generation apparatus 100 is an example of a work machine periphery monitoring apparatus that monitors the periphery of the work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and a stereo camera 6. Is done. Specifically, the image generation device 100 generates an output image based on the input image captured by the camera 2 mounted on the work machine and the input distance image output by the stereo camera 6, and outputs the output image to the operator. Present.

  FIG. 2 is a diagram illustrating a configuration example of an excavator 60 as a work machine on which the image generating apparatus 100 is mounted. The excavator 60 is placed on a crawler-type lower traveling body 61 via a turning mechanism 62. A swing body 63 is mounted so as to be swingable around a swing axis PV.

  The upper swing body 63 includes a cab (operator's cab) 64 on the front left side, a drilling attachment E on the front center, and the camera 2 (right camera 2R, rear camera 2B) on the right and rear surfaces. ) And a stereo camera 6 (right-side stereo camera 6R, rear stereo camera 6B). In addition, the display part 5 is installed in the position in the cab 64 where the operator is easy to visually recognize.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), etc. A program corresponding to each of the attaching means 10, the image generating means 11, the pixel extracting means 12, the distance correcting means 13, the distance image synthesizing means 14, and the driving support means 15 is stored in a ROM or NVRAM as a temporary storage area. The CPU is caused to execute processing corresponding to each means while using the RAM.

  The camera 2 is a device for acquiring an input image that reflects the surroundings of the excavator 60, and is attached to the right side surface and the rear surface of the upper swing body 63 so that, for example, an area that is a blind spot of the operator in the cab 64 can be imaged. (See FIG. 2), a right side camera 2R and a rear camera 2B provided with an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens or a fisheye lens is attached so as to capture a wide range. It may be.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Although it is output, an input image in which the apparent distortion or tilt is not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display, a projector, or the like installed in the cab 64 (see FIG. 2) of the excavator 60. Display an image.

  The stereo camera 6 is a device for obtaining a two-dimensional array of distance information of objects existing around the excavator 60. For example, the upper turning body 63 can capture an image of a blind spot of an operator in the cab 64. Are attached to the right side and the rear side (see FIG. 2). In addition, the stereo camera 6 may be attached to any one of the front surface, the left side surface, the right side surface, and the rear surface of the upper swing body 63, or may be attached to all the surfaces.

  The stereo camera 6 is a device having an imaging function of a plurality of cameras. In this embodiment, the stereo camera 6 is an apparatus that integrally includes the imaging functions of two cameras, and includes two sets of imaging mechanisms (a combination of an imaging element and a lens mechanism) and one shared shutter mechanism. Provided in one housing. However, the stereo camera 6 may be configured by a plurality of separate and independent cameras.

  In addition, the plurality of sets of imaging mechanisms in the stereo camera 6 are arranged at a predetermined interval from each other. In the present embodiment, the two sets of imaging mechanisms are juxtaposed at a predetermined interval in the vertical direction (vertical direction). Note that the two sets of imaging mechanisms may be juxtaposed with a predetermined interval in the horizontal direction (horizontal direction), with a predetermined interval in each of the vertical direction (vertical direction) and the horizontal direction (horizontal direction). They may be juxtaposed.

  In addition, the stereo camera 6 acquires distance information for each pixel, while the camera 2 acquires luminance, hue value, saturation value, and the like for each pixel. In the present embodiment, the stereo camera 6 has each pixel of an image (hereinafter referred to as “basic image”) output by one imaging mechanism based on the parallax between images output by each of the two sets of imaging mechanisms. The distance between the object reflected on the camera and the stereo camera 6 is derived. Then, the stereo camera 6 substitutes the derived distance into the value of each pixel of the basic image to generate two-dimensional array distance information, and outputs the distance information to the control unit 1. Therefore, hereinafter, the distance information of the two-dimensional array by the stereo camera 6 is referred to as an input distance image and an output distance image by comparing with the input image and the output image of the camera 2. Further, the resolution (number of pixels) of the input distance image and output distance image of the stereo camera 6 may be the same as or different from the resolution of the input image and output image of the camera 2. In addition, one or more pixels in the input image and output image of the camera 2 may be associated in advance with one or more pixels in the input distance image and output distance image of the stereo camera 6. Further, the input distance image may be generated by the control unit 1 based on a plurality of images output by each imaging mechanism of the stereo camera 6.

  Also, the input image planes of the plurality of imaging mechanisms constituting the stereo camera 6 are preferably parallel to each other, and most preferably form the same plane. This is because the process for deriving the distance from the parallax can be performed easily and with high accuracy.

  Similarly to the camera 2, the stereo camera 6 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens that can capture a wide range. Or a fisheye lens may be attached.

  Similarly to the camera 2, the stereo camera 6 acquires an input distance image according to a control signal from the control unit 1, and outputs the acquired input distance image to the control unit 1. Similarly to the camera 2, when the stereo camera 6 acquires an input distance image using a fish-eye lens or a wide-angle lens, the corrected input distance corrected for apparent distortion and tilt caused by using these lenses. Although the image is output to the control unit 1, the input distance image that is not corrected for the apparent distortion and tilt may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  In addition, the image generation apparatus 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding features and a sense of distance can be intuitively grasped. An output image to be generated may be generated, and the output image may be presented to the operator.

  The image generating apparatus 100 performs the same process on the input distance image. In this case, the processing target image is replaced with the processing target distance image. The same applies to the following description.

  The “processing target image” is an image that is generated based on an input image and that is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.). When an input image including a horizontal image (for example, an empty portion) is used in an image conversion process with an image captured from above by a camera that captures the image from above, the horizontal image is The input image is projected onto a predetermined spatial model so that it is not unnaturally displayed (for example, the sky part is not treated as being on the ground surface), and then the projected image projected onto the spatial model is changed to another two. It is an image suitable for image conversion processing, which is obtained by reprojecting onto a dimensional plane. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “spatial model” is a plane or curved surface other than the processing target image plane that is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane or an angle with the processing target image plane). A plane or a curved surface to be formed), and a projection target of an input image composed of one or a plurality of planes or curved surfaces.

  Note that the image generation apparatus 100 may generate an output image by performing image conversion processing on the projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image as it is without being subjected to image conversion processing.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected. FIG. 3A illustrates a relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. FIG. 3B shows the relationship between the excavator 60 and the space model MD when the excavator 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and includes a planar region R1 inside the bottom surface and a curved region R2 inside the side surface.

  FIG. 4 is a diagram illustrating an example of a relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is a plane including the plane area R1 of the space model MD, for example. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for the sake of clarity, the space model MD may be either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the subsequent drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the spatial model MD, or may be an annular area not including the plane area R1 of the spatial model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate association means 10 is a means for associating coordinates on the input image plane where the input image captured by the camera 2 is located, coordinates on the space model MD, and coordinates on the processing target image plane R3. For example, various parameters relating to the camera 2 such as optical center, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc., which are set in advance or input via the input unit 3 And the coordinates on the input image plane, the coordinates on the space model MD, and the processing target image based on the predetermined positional relationship among the input image plane, the spatial model MD, and the processing target image plane R3. The coordinates on the plane R3 are associated with each other, and the corresponding relationship is stored in the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 of the storage unit 4. .

  When the processing target image is not generated, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

  In addition, the coordinate association unit 10 performs the same process on the input distance image output from the stereo camera. In this case, the camera, the input image plane, and the processing target image plane are replaced with the stereo camera, the input distance image plane, and the processing target distance image plane. The same applies to the following description.

  The image generation unit 11 is a unit for generating an output image. For example, by performing scale conversion, affine conversion, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image are positioned. Corresponding to the coordinates on the output image plane to be processed, the correspondence relationship is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the coordinate correspondence means 10 stores the value corresponding to the input image / spatial model With reference to the map 40 and the spatial model / processing target image correspondence map 41, the value of each pixel in the output image (for example, the luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image To generate an output image.

  Further, the image generation means 11 is preset or input via the input unit 3, the optical center of the virtual camera, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc. The coordinates on the processing target image plane R3 and the coordinates on the output image plane on which the output image is located are associated with each other on the basis of the various parameters, and the corresponding relationship is stored in the processing target image / output image correspondence map 42 of the storage unit 4. With reference to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored and stored in the coordinate matching means 10, the value of each pixel (for example, luminance value, A hue value, a saturation value, etc.) and the value of each pixel in the input image are associated with each other to generate an output image.

  Note that the image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the image generation unit 11 does not generate the processing target image, the image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane in accordance with the applied image conversion processing, and supports the input image / space model. While referring to the map 40, the output image is generated by associating the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) with the value of each pixel in the input image. In this case, the image generation unit 11 omits the association between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence relationship in the processing target image / output image correspondence map 42. .

  The image generation unit 11 performs the same process on the input distance image or the processing target distance image. In that case, the output image plane is read as the output distance image plane. The same applies to the following description.

  The pixel extraction unit 12 is a unit for extracting pixels that satisfy a predetermined condition from the input distance image of the stereo camera 6. For example, the pixel extraction unit 12 extracts a pixel that satisfies a predetermined condition from pixels having a pixel value representing a distance between a feature around the excavator 60 and the stereo camera 6 in the input distance image.

  Specifically, the pixel extraction means 12 is a distance between the plane (hereinafter referred to as “installation surface”) on which the excavator 60 is located and the stereo camera 6 (hereinafter referred to as “installation surface distance”). Pixels whose difference is equal to or greater than a predetermined distance are extracted. The installation surface may be an inclined surface. The installation plane distance is set in advance for each pixel in the input distance image of the stereo camera 6 based on the installation position and installation angle of the stereo camera 6. Then, the pixel extraction unit 12 replaces the pixel value of the pixel that has not been extracted with a predetermined value (for example, zero as the minimum value), and maintains the pixel value of the extracted pixel as it is, thereby obtaining the input distance image. to correct. Details of the pixel extracting means 12 will be described later.

  The distance correction unit 13 is a unit for correcting the distance information acquired by the stereo camera. For example, the distance correction unit 13 corrects the pixel value extracted by the image extraction unit 12. Specifically, the distance correction means 13 determines the pixel value representing the distance between the feature around the excavator 60 and the stereo camera 6 between the reference outside the stereo camera 6 and the feature. Correct the value to represent the distance. Further, the distance correction unit 13 may correct the pixel value in the input distance image output from the stereo camera 6 before extraction by the image extraction unit 12. Details of the distance correction means 13 will be described later.

  The distance image synthesizing unit 14 is a unit for synthesizing an image related to the camera and an image related to the stereo camera. For example, the distance image synthesis unit 14 synthesizes the output image based on the input image of the camera 2 and the output distance image based on the input distance image of the stereo camera 6 generated by the image generation unit 11. The output distance image is an output distance image that has been subjected to at least one of extraction by the image extraction unit 12 and correction by the distance correction unit 13. However, the output distance image may be an output distance image before extraction by the image extraction unit 12 and correction by the distance correction unit 13. Details of the distance image synthesis means 14 will be described later.

  The driving support means 15 is a means for supporting the driving of the excavator 60. For example, the driving support means 15 performs a function of supporting the driving of the excavator 60 when it is determined that a predetermined feature exists in the moving direction of the excavator 60.

  The “predetermined feature” includes, for example, a person such as a worker who may collide with the excavator 60, an obstacle that prevents the excavator 60 from moving, and the like.

  Functions that support driving (hereinafter referred to as “driving support functions”) include, for example, output of alarms, display of warnings on the display unit 5, lighting / flashing of warning lights, deceleration of the shovel 60, and travel speed of the shovel 60. Restrictions, stoppage of excavator 60 travel, and the like.

  Specifically, when the driving support means 15 determines that there is an obstacle in the moving direction (backward) of the excavator 60 when the excavator 60 is moving (retreating), the presence of the obstacle on the display unit 5 is present. Display a warning to inform you. The moving direction of the excavator 60 is detected based on, for example, the operation content of the operation lever (for example, the operation direction, the operation amount, etc.).

  The driving support means 15 may reduce the moving (retracting) speed of the excavator 60 as the distance between the excavator 60 and the obstacle decreases, and the distance between the excavator 60 and the obstacle is a predetermined distance. The movement (retreat) of the excavator 60 may be stopped when the following occurs.

  The driving support means 15 may determine that there is an obstacle that may collide with the excavator 60 when a feature having a height of a predetermined distance or more from the installation surface is detected by the stereo camera 6. . Further, the driving support means 15 determines whether or not there is an obstacle that may collide with the shovel 60 based on the decreasing gradient of the pixel value along a predetermined direction (moving direction of the shovel 60) in the input distance image. You may judge.

  Next, an example of specific processing by the coordinate association unit 10 and the image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 has UVW orthogonal coordinates with the optical center C of the camera 2 as the origin. The space model is represented as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate association unit 10 converts the coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is optically converted. After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" indicates that the direction is reversed) This means that the XYZ coordinate system is rotated so that the UVW coordinate system coincides with the + W direction in front of the camera and the XYZ coordinate system in the −Z direction vertically below.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate association unit 10 uses an XYZ coordinate system in parallel with each of the plurality of UVW coordinate systems. It will be moved and rotated.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. Since it is realized by rotating to coincide with the axis, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this transformation in Hamilton's quaternion. it can.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the surface extending between the vector A and the vector B as an axis. If the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It will be expressed as

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

The conjugate quaternion of the quaternion Q is

It shall be represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. A point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

It will be expressed as

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , A quaternion R representing "rotation to make the X axis coincide with the U axis"

It will be expressed as

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

Since the quaternion R is invariable in each of the cameras 2, the coordinate association unit 10 thereafter performs the above calculation to obtain the coordinates on the space model (XYZ coordinate system). It can be converted into coordinates on the input image plane (UVW coordinate system).

  After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 determines the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model with a coordinate P ′ represented in the UVW coordinate system and the optical axis G of the camera 2 is calculated.

  In addition, the coordinate association unit 10 includes an intersection E between the plane H and the optical axis G, and a coordinate P ′ in a plane H that is parallel to the input image plane R4 (for example, CCD plane) of the camera 2 and includes the coordinate P ′. Are calculated, and the deviation angle φ formed by the line segment EP ′ connecting the two and the U ′ axis in the plane H, and the length of the line segment EP ′.

  In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate matching means 10 performs normal projection (h = ftanα) and orthographic projection (h = fsinα). Image height by selecting an appropriate projection method such as stereo projection (h = 2 ftan (α / 2)), equisolid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, the coordinates P (P ′) on the space model MD can be associated with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

It will be expressed as

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  Note that, when it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate associating means 10 inputs the coordinates P (P ′) on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with coordinates on the image plane R4, or may be associated with coordinates on the input image plane R4 selected by the operator.

  Next, a process of reprojecting coordinates on the curved surface area R2 (coordinates having a component in the Z-axis direction) among the coordinates on the spatial model MD onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between coordinates by the coordinate association means 10, and FIG. 6A shows an input image plane R4 of the camera 2 that employs a normal projection (h = ftanα) as an example. FIG. 6 is a diagram showing a correspondence relationship between the coordinates on the upper surface and the coordinates on the space model MD, and the coordinate associating means 10 displays the coordinates on the input image plane R4 of the camera 2 and the space model MD corresponding to the coordinates. Both the coordinates are made to correspond to each other so that each of the line segments connecting the coordinates passes through the optical center C of the camera 2.

  In the example of FIG. 6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane area R1 of the space model MD, and inputs the image 2 R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 6B is a diagram showing a correspondence relationship between coordinates on the curved surface region R2 of the space model MD and coordinates on the processing target image plane R3, and the coordinate association unit 10 is positioned on the XZ plane. The parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the spatial model MD and the processing target image corresponding to the coordinates are introduced. The coordinates on the plane R3 are associated with each other such that the coordinates are on one of the parallel line groups PL.

  In the example of FIG. 6B, the coordinate matching means 10 assumes that the coordinate L2 on the curved surface region R2 of the space model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line, and both coordinates are obtained. Make it correspond.

  The coordinate association means 10 can associate the coordinates on the plane area R1 of the spatial model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface area R2. However, in the example of FIG. 6B, since the plane area R1 and the processing target image plane R3 are a common plane, the coordinates L1 on the plane area R1 of the spatial model MD and the processing target image plane R3 The coordinate M1 has the same coordinate value.

  In this way, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. And stored in the space model / processing object image correspondence map 41.

  FIG. 6C is a diagram illustrating a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V that adopts the normal projection (h = ftanα) as an example. In the image generation means 11, each line segment connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates passes through the optical center CV of the virtual camera 2V. In this way, both coordinates are associated.

  In the example of FIG. 6C, the image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane area R1 of the space model MD), The coordinate N2 on the output image plane R5 of the virtual camera 2V is associated with the coordinate M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  Note that when the virtual camera 2V employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the image generating unit 11 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and sets the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. Each of the output images is stored in the processing target image / output image correspondence map 42 in association with each other in the output image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored by the coordinate matching means 10. An output image is generated by associating the pixel value with the value of each pixel in the input image.

  FIG. 6D is a combination of FIGS. 6A to 6C. The camera 2, the virtual camera 2V, the plane area R1 and the curved area R2 of the space model MD, and the processing target. The mutual positional relationship of image plane R3 is shown.

  Next, the operation of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 7A is a diagram in the case where the angle β is formed between the parallel line group PL positioned on the XZ plane and the processing target image plane R3, and FIG. 7B is the diagram on the XZ plane. It is a figure in case angle (beta) 1 ((beta) 1> (beta)) is formed between the parallel line group PL and the process target image plane R3. Also, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 7A and 7B corresponds to the coordinates Ma to Md on the processing target image plane R3, respectively. Assume that the intervals between the coordinates La to Ld in FIG. 7A are equal to the intervals between the coordinates La to Ld in FIG. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. It shall be. In this case, the Z axis is referred to as a “reprojection axis”.

  As shown in FIGS. 7A and 7B, the distance between the coordinates Ma to Md on the processing target image plane R3 is such that the angle between the parallel line group PL and the processing target image plane R3 is the same. As it increases, it decreases linearly (it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md). On the other hand, since the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is such that only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD is linearly enlarged or out of the image portion on the output image plane R5 (see FIG. 6). It means to be reduced.

  Next, an alternative example of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 8A is a diagram in the case where all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3, and FIG. It is a figure in case all the line groups AL extend toward the process target image plane R3 from the starting point T2 (T2> T1) on the Z axis. 8A and 8B, the coordinates La to Ld on the curved surface region R2 of the spatial model MD correspond to the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are outside the region of the processing target image plane R3.) The intervals between the coordinates La to Ld in FIG. It is assumed that the interval between the coordinates La to Ld in 8 (B) is equal. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the processing target image plane R3. It shall be. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (high) between the starting point of the auxiliary line group AL and the origin O. As the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md is larger, the width of reduction of each interval is larger. On the other hand, since the coordinate group on the plane region R1 of the spatial model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image portion is enlarged or reduced nonlinearly.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and does not affect the space model MD. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 can be linearly or nonlinearly enlarged or reduced, the road surface image in the vicinity of the excavator 60 Without affecting the excavator 60 (virtual image when seen from directly above), the features located around the excavator 60 (features in the image when the excavator 60 is seen in the horizontal direction) can be quickly displayed. And it can be expanded or contracted flexibly, and the visibility of the blind spot area of the excavator 60 can be improved.

  Next, referring to FIG. 9, the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. Processing to be generated (hereinafter referred to as “output image generation processing”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Further, it is assumed that the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved surface area R2), and the processing target image plane R3 is determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD using the coordinate association unit 10 (step S1).

  Specifically, the coordinate association unit 10 obtains an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. One point that intersects the curved surface region R2 of the space model MD is calculated, and a coordinate on the curved surface region R2 corresponding to the calculated point is set to one on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived as coordinates, and the corresponding relationship is stored in the space model / processing object image correspondence map 41. It should be noted that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be a value to be entered.

  In addition, when one coordinate on the processing target image plane R3 matches one coordinate on the plane area R1 of the space model MD, the coordinate association unit 10 uses the one coordinate on the plane area R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing object image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate association unit 10 to associate one coordinate on the spatial model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate association unit 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα), is a line segment extending from one coordinate on the space model MD, and the optical center A point where the line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are set on the input image plane R4 corresponding to the one coordinate on the space model MD. And the corresponding relationship is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3), and all the coordinates are still set. Are determined to be not associated (NO in step S3), the processes in steps S1 and S2 are repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation process, starts the output image generation process, and the image generation unit 11 The coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

  Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image, and is determined by the content of the applied scale conversion, affine conversion, or distortion conversion. The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating the output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, when the output image is generated using the virtual camera 2V that employs normal projection (h = ftanα), the image generation unit 11 obtains the coordinates of the optical center CV of the virtual camera 2V and outputs A line segment extending from one coordinate on the image plane R5, a point where a line segment passing through the optical center CV intersects the processing target image plane R3 is calculated, and the coordinates on the processing target image plane R3 corresponding to the calculated point May be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Thereafter, the control unit 1 uses the image generation unit 11 to refer to the input image plane R4 while referring to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. The correspondence between the coordinates on the space model MD and the coordinates on the space model MD, the correspondence between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the coordinates on the processing target image plane R3 and the output image plane R5 And the values of the coordinates on the input image plane R4 corresponding to the coordinates on the output image plane R5 (for example, luminance values, hue values, saturation values, etc.) are obtained. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). Note that when a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 uses each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value (for example, average value, maximum value, minimum value, intermediate value, etc.) may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. .

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6), and all the coordinate values are still associated. If it is determined that it is not attached (NO in step S6), the processes in steps S4 and S5 are repeated.

  On the other hand, if the control unit 1 determines that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

  Note that, when the processing target image is not generated, the image generation apparatus 100 omits the processing target image generation processing, and sets “coordinates on the processing target image plane” in step S4 in the output image generation processing to “on the spatial model”. It shall be read as "coordinates".

  With the above configuration, the image generation apparatus 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the features around the excavator 60 and the excavator 60. .

  Further, the image generation apparatus 100 associates coordinates on the processing target image plane R3 from the processing target image plane R3 to the input image plane R4 through the spatial model MD, thereby obtaining the coordinates on the processing target image plane R3. One or more coordinates on R4 can be reliably associated, and compared with a case where coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD (in this case) Can reliably correspond each coordinate on the input image plane R4 to one or a plurality of coordinates on the processing target image plane R3. However, a part of the coordinates on the processing target image plane R3 is part of the input image plane R4. In some cases, the coordinates may not be associated with any of the above coordinates, and in such a case, it is necessary to perform interpolation processing or the like on a part of the coordinates on the processing target image plane R3). It is possible to rapidly generate.

  Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generating apparatus 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Thus, it is possible to realize a desired enlargement or reduction without rewriting the contents of the input image / space model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the space model / processing object image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters relating to scale conversion, affine transformation, or distortion transformation. The desired output image (scale-converted image, affine-transformed image, or distortion-converted image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing object image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

  The image generation apparatus 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects them onto the processing target image plane R3 to generate a processing target image. Then, an image conversion process (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion process, etc.) is performed on the generated processing target image to generate an output image, and the vicinity of the excavator 60 can be seen from above. An image looking down (image in the plane region R1) and an image looking around in the horizontal direction from the excavator 60 (image in the processing target image plane R3) are simultaneously displayed.

  When the image generation apparatus 100 does not generate the processing target image, the output image is generated by performing image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD. Shall.

  The output image is trimmed in a circle so that the image when the excavator 60 performs the turning motion can be displayed without a sense of incongruity, and the center CTR of the circle is on the cylindrical central axis of the space model MD and the excavator 60 is turned. It is generated so as to be on the axis PV, and is displayed so as to rotate about its center CTR in accordance with the turning operation of the excavator 60. In this case, the cylindrical central axis of the space model MD may or may not coincide with the reprojection axis.

  The radius of the space model MD is, for example, 5 meters, and the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is the turning of the shovel 60. When a feature (for example, a worker) exists at a position separated from the center by the maximum reachable distance (for example, 12 meters) of the excavation attachment E, the feature is sufficiently large (for example, on the display unit 5). , 7 mm or more.) Can be set to be displayed.

  Further, the output image may be a CG image of the excavator 60 arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. This is to make the positional relationship between the excavator 60 and the feature appearing in the output image easier to understand. Note that a frame image including various kinds of information such as an orientation may be arranged around the output image.

  Next, details of an output image generated by the image generation apparatus 100 will be described with reference to FIGS.

  FIG. 11 is a top view of the excavator 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three stereo cameras 6 (left side stereo camera 6L, right side stereo). Camera 6R and rear stereo camera 6B). In addition, areas CL, CR, and CB indicated by alternate long and short dash lines in FIG. 11 indicate imaging ranges of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. In addition, areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate imaging ranges of the left stereo camera 6L, the right stereo camera 6R, and the rear stereo camera 6B, respectively.

  In this embodiment, the imaging range of the stereo camera 6 is narrower than the imaging range of the camera 2, but the imaging range of the stereo camera 6 may be the same as the imaging range of the camera 2 and is wider than the imaging range of the camera 2. Also good. Further, the imaging range of the stereo camera 6 is located in the vicinity of the excavator 60 within the imaging range of the camera 2, but may be in a region farther from the excavator 60.

  FIG. 12 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images.

  The image generation apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the space model MD, and then reprojects them onto the process target image plane R3 to generate a process target image. . The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generating apparatus 100 includes an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1) and an image in which the surroundings are viewed in the horizontal direction from the excavator 60 (an image in the processing target image plane R3). Display at the same time. The image displayed at the center of the output image is a CG image 60CG of the excavator 60.

  In FIG. 12, the input image of the right-side camera 2R and the input image of the rear camera 2B each capture a person in the overlapping area between the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B (right side). (See region R10 surrounded by a two-dot chain line in the input image of the direction camera 2R and region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.)

  However, if the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image loses the person in the overlapping area (one point in the output image). (See region R12 surrounded by a chain line.)

  Therefore, in the output image portion corresponding to the overlapping region, the image generation device 100 associates the region on the input image plane of the rear camera 2B with the coordinate on the input image plane of the right-side camera 2R. The area is mixed to prevent the object in the overlapping area from disappearing.

  FIG. 13 is a diagram for explaining an image disappearance prevention process for preventing the disappearance of an object in the overlapping region of the imaging ranges of the two cameras.

  FIG. 13A is a diagram showing an output image portion corresponding to an overlapping area between the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B, and corresponds to the rectangular area R13 indicated by the dotted line in FIG.

  In FIG. 13A, a gray area PR1 is an image area in which the input image portion of the rear camera 2B is arranged, and each coordinate on the output image plane corresponding to the area PR1 has a rear camera. Coordinates on the 2B input image plane are associated.

  On the other hand, a region PR2 filled with white is an image region in which the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the portion PR2 has an input image plane of the right side camera 2R. The upper coordinates are associated.

  In the present embodiment, the region PR1 and the region PR2 are arranged so as to form a stripe pattern, and the boundary line of the portion where the regions PR1 and the region PR2 are alternately arranged in a stripe shape is centered on the turning center of the excavator 60. Determined by concentric circles on a horizontal plane.

  FIG. 13B is a top view showing the state of the space region diagonally right behind the shovel 60, and shows the current state of the space region imaged by both the rear camera 2B and the right side camera 2R. FIG. 13B shows that a rod-shaped three-dimensional object OB exists behind the excavator 60 diagonally to the right.

  FIG. 13C shows a part of an output image generated based on an input image obtained by actually capturing the spatial region shown in FIG. 13B with the rear camera 2B and the right side camera 2R.

  Specifically, the image OB1 extends in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint transformation for generating the road surface image of the three-dimensional object OB in the input image of the rear camera 2B. Represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

  Further, the image OB2 is expanded in the extension direction of the line connecting the right-side camera 2R and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the right rear camera 2R. Represents a thing. That is, the image OB2 is a part of an image of the three-dimensional object OB displayed when a road surface image in the output image portion is generated using the input image of the right side camera 2R.

  As described above, the image generating apparatus 100 includes the region PR1 in which the coordinates on the input image plane of the rear camera 2B are associated with the region PR2 in which the coordinates on the input image plane of the right-side camera 2R are associated in the overlapping region. Mix. As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 related to one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

  14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12, and FIG. 14 (A) is the output of FIG. An image is shown, and FIG. 14B shows an output image after applying the image disappearance prevention process. In the region R12 surrounded by the alternate long and short dash line in FIG. 14A, the person disappears, whereas in the region R14 surrounded by the alternate long and short dash line in FIG. 14B, the person is displayed without disappearing.

  Next, an output distance image generated by the image generation apparatus 100 will be described with reference to FIG. FIG. 15 is a diagram illustrating input distance images of the three stereo cameras 6 mounted on the excavator 60 and output distance images generated using the input distance images.

  The image generation apparatus 100 projects the input distance images of the three stereo cameras 6 onto the plane area R1 and the curved surface area R2 of the space model MD and then reprojects them onto the processing object distance image plane R3. Generate a distance image. Further, the image generation apparatus 100 generates an output distance image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target distance image. The image generation apparatus 100 then includes a distance image (distance image in the plane region R1) looking down from the vicinity of the excavator 60 and a distance image (image in the processing target distance image plane R3) viewed from the excavator 60 in the horizontal direction. ) Are displayed at the same time. The image displayed at the center of the output distance image is a CG image 60CG of the excavator 60.

  Note that the input distance image and the output distance image in FIG. 15 are displayed so that the smaller the pixel value (distance from the stereo camera 6), the whiter (lighter), and the larger the pixel value, the darker (darker). Note that the pixel values (distances) of the portions corresponding to the road surface and the upper swing body 63 are set to the maximum value (infinity) in order to improve the visibility in the output distance image of the features existing around the excavator 60. Is done. That is, the portions corresponding to the road surface and the upper swing body 63 are represented in black.

  In FIG. 15, the input distance image of the right-side stereo camera 6R and the input distance image of the rear-side stereo camera 6B are respectively in the overlapping area of the imaging range of the right-side stereo camera 6R and the imaging range of the rear-side stereo camera 6B. (See the region R15 surrounded by the two-dot chain line in the input distance image of the right-side stereo camera 6R and the region R16 surrounded by the two-dot chain line in the input distance image of the rear stereo camera 6B.)

  However, if the coordinates on the output distance image plane are associated with the coordinates on the input distance image plane related to the closest stereo camera, the output distance image causes the person in the overlapping region to disappear ( (Refer area | region R17 enclosed with the dashed-dotted line in an output distance image.).

  Therefore, the image generation apparatus 100, in the output distance image portion corresponding to the overlapping region, out of the coordinates on the input distance image plane of the rear stereo camera 6B and the coordinates on the input distance image plane of the right-side stereo camera 6R, The coordinates with the smaller pixel value (distance) are associated with the coordinates on the output distance image plane. As a result, the image generation apparatus 100 displays two distance images related to one solid object on the output distance image, and prevents the solid object from disappearing from the output distance image. Hereinafter, this process for preventing the disappearance of an object in the overlapping area of the imaging ranges of the two stereo cameras is referred to as a distance image disappearance prevention process.

  FIG. 16 is a comparison diagram showing the difference between the output distance image of FIG. 15 and the output distance image obtained by applying the distance image disappearance prevention process to the output distance image of FIG. 15. FIG. The output image of FIG. 15 is shown, and FIG. 16B shows the output distance image after applying the distance image disappearance prevention process. In FIG. 16A, a person disappears in a region R17 surrounded by a one-dot chain line, whereas in a region R17 surrounded by a one-dot chain line in FIG. 16B, a person is displayed without disappearing.

  Next, with reference to FIG. 17 and FIG. 18, processing in which the image generation apparatus 100 combines an output distance image with an output image (hereinafter referred to as “distance image combining processing”) will be described.

  FIG. 17 is a diagram illustrating an example of a combined output image obtained by combining an output distance image with an output image.

  For example, as shown in FIG. 16B, the distance image synthesizing unit 14 of the image generating apparatus 100 uses the pixel value (distance) of the extracted pixel extracted by the pixel extracting unit 12 as a luminance value, a hue value, a saturation value, and the like. Then, the extracted pixel is superimposed on the output image shown in FIG. The extracted pixels displayed in a superimposed manner have, for example, a color corresponding to the distance from the stereo camera 6, and as the distance increases, the color is red or yellow, green, or blue stepwise or steplessly. To change. Note that the extracted pixels displayed in a superimposed manner have a luminance corresponding to the distance from the stereo camera 6, and the luminance may be decreased stepwise or steplessly as the distance increases. Regions EX1 to EX5 in FIG. 17 are regions formed by these extracted pixels, and are hereinafter referred to as feature regions.

  Further, the distance image synthesis unit 14 may superimpose and display the distance information of the feature regions EX1 to EX5 formed by a predetermined number or more of the adjacent extraction pixels on the output image. Specifically, the distance image synthesis means 14 uses, for example, the minimum value, maximum value, average value, intermediate value, etc. of the pixel values (distances) of the pixels constituting the feature region EX1 as the distance to the feature region EX1. Is superimposed on the output image as a representative value. The distance image composition unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is to make it easier to recognize features that are relatively close to the excavator 60, that is, features that have a relatively high possibility of contact.

  FIG. 18 is a diagram illustrating another example of the combined output image obtained by combining the output distance image with the output image.

  In FIG. 18, the distance image composition unit 14 superimposes and displays the frames RF1 to RF5 corresponding to the feature regions EX1 to EX5 in FIG. 17 instead of superimposing and displaying the extracted pixels in different colors and brightness. The line type of the frame is arbitrary including a dotted line, a solid line, a broken line, an alternate long and short dash line, and the shape of the frame is arbitrary including a rectangle, a circle, an ellipse, and a polygon.

  Further, the distance image synthesis unit 14 may superimpose and display distance information corresponding to each of the frames RF1 to RF5 on the output image. Specifically, the distance image composition unit 14 uses, for example, the minimum value, maximum value, average value, intermediate value, and the like of the pixel values (distances) of the pixels constituting the feature region EX1 corresponding to the frame RF1 in the frame RF1. Are superimposed on the output image as a representative value of the distance corresponding to. The distance image composition unit 14 may superimpose and display the representative value on the output image only when the representative value is less than the predetermined value. This is to make it easier to recognize features that are relatively close to the excavator 60, that is, features that have a relatively high possibility of contact.

  Further, the distance image composition unit 14 may change the color of the frame according to the corresponding distance information. Specifically, the distance image synthesizing unit 14 changes the color of the frame stepwise or steplessly such as red, yellow, green, and blue as the distance increases. The distance image synthesis means 14 may decrease the brightness of the frame stepwise or steplessly as the distance increases.

  By looking at the output image after synthesis as described above, the operator of the excavator 60 can more easily recognize the position of the feature existing around the excavator 60 and the distance to the feature. Further, the operator of the excavator 60 can more easily recognize the position of the feature existing in the blind spot as viewed from the operator and the distance to the feature.

  With the above configuration, the image generation apparatus 100 combines the output distance image generated based on the input distance image captured by the stereo camera 6 on the output image generated based on the input image captured by the camera 2. . Therefore, the image generating apparatus 100 can measure the distance to each feature existing in a relatively wide range around the excavator 60 with a relatively high resolution. As a result, the image generation apparatus 100 can generate a highly reliable post-combination output image.

  Further, the camera 2 and the stereo camera 6 are arranged so that the respective imaging ranges surround the excavator 60. Therefore, the image generation apparatus 100 can generate a combined output image that represents a state when the periphery of the excavator 60 is viewed from above. As a result, the operator of the excavator 60 can more easily recognize the position of the feature existing around the excavator 60 and the distance to the feature.

  Further, the image generating apparatus 100 performs the same image conversion process as the image conversion process performed on the processing target image on the processing target distance image. Therefore, the image generation device 100 can easily associate the coordinates on the output image plane with the coordinates on the output distance image plane.

  Further, the image generating apparatus 100 changes the color of the extraction pixel or the frame stepwise or steplessly according to the distance from the stereo camera 6, that is, according to the distance from the excavator 60. Therefore, the operator of the shovel 60 can sensuously determine whether there is a danger of contact.

  In addition, the image generation apparatus 100 acquires distance information of a two-dimensional array using the stereo camera 6. Therefore, the image generation apparatus 100 has a heat source, a light source, a light-absorbing material, and the like in the imaging range as compared with a case where distance information of a two-dimensional array is acquired using a distance image sensor including a light emitting LED and a light receiving CCD. Not easily affected by

  Next, referring to FIGS. 19 to 23, the image generation apparatus 100 extracts pixels that satisfy a predetermined condition from the input distance image and corrects the input distance image (hereinafter referred to as “pixel extraction process”). ). In the present embodiment, the pixel value in the input distance image captured by the rear stereo camera 6B is equal to or smaller than the installation surface distance.

  FIG. 19 is a partial right side view of the excavator 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 19, the excavator 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). Note that a region ZB indicated by a dotted line in FIG. 19 indicates an imaging range of the rear stereo camera 6B.

  Further, FIG. 19 shows that an object P3 and a raised portion P4 of the ground that extend vertically upward from the ground exist behind the excavator 60. Furthermore, FIG. 19 shows that the distance E1a to the object P3 is detected and the distance E2a to the raised portion P4 is detected by the rear stereo camera 6B.

  FIG. 19 shows that if the object P3 and the raised portion P4 are not present, the rear stereo camera 6B detects the installation surface distances E1 and E2 instead of the distances E1a and E2a. .

  In FIG. 19, the difference (E1−E1a) between the installation surface distance E1 and the distance E1a is a distance (hereinafter referred to as “difference distance”) E1b, and the difference (E2) between the installation surface distance E2 and the distance E2a. -E2a) indicates the difference distance E2b.

  FIG. 20 is an enlarged view of the raised portion P4 in FIG. As shown in FIGS. 19 and 20, the line segment connecting the point on the raised portion P4 and the stereo camera 6B viewed from the rear stereo camera 6B is an angle between the installation surface and the installation surface (hereinafter referred to as “installation surface angle”). ) Γ4 is formed. In this case, the height (hereinafter referred to as “estimated height”) H1b of the raised portion P4 from the installation surface is represented by a difference distance E2b × sin (γ4). Similarly to the installation surface distance, the installation surface angle is set in advance for each pixel in the input distance image of the rear stereo camera 6B based on the installation position and installation angle of the rear stereo camera 6B.

  The pixel value in the input distance image captured by the stereo camera 6B represents the distance between the feature and the stereo camera 6B. Therefore, by multiplying the difference distance, which is the difference between the value of each pixel in the input distance image and the installation surface distance corresponding to each pixel, by the sine of the installation surface angle corresponding to each pixel, the estimated height of the feature is obtained. You can ask for it. In the present embodiment, the estimated height H1b of the raised portion P4 can be obtained by multiplying the difference distance E2b by sin (γ4).

  Incidentally, the pixel value in the input distance image captured by the rear stereo camera 6B represents the distance between the feature and the rear stereo camera 6B as described above. Therefore, when an output distance image is generated by performing a process of changing the color or luminance according to the distance from the rear stereo camera 6B on the input distance image, the visibility of the output distance image is obtained by using the pixel value in the input distance image as it is. May be damaged. Specifically, the output distance image represents the ground (installation surface) with a color or brightness that changes stepwise as it moves away from the excavator 60, and may make it difficult to understand the presence of the object P3 and the raised portion P4. .

  Therefore, for example, the pixel extraction unit 12 of the image generation apparatus 100 extracts pixels that satisfy a predetermined condition from the input distance image of the rear stereo camera 6B, and corrects the input distance image based on the extraction result. Specifically, the pixel extraction unit 12 extracts a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel from the pixels in the input distance image. Then, the pixel extraction unit 12 replaces the pixel value of a pixel having a pixel value equal to the installation surface distance with a predetermined value (for example, zero as the minimum value). On the other hand, the pixel extraction means 12 maintains the pixel value of the pixel having a pixel value smaller than the installation surface distance as it is. In this case, the pixel value equal to the installation surface distance may be a value having a predetermined width.

  Moreover, the pixel extraction means 12 may extract the pixel whose difference distance is more than predetermined distance among each pixel in an input distance image. In this case, the pixel extraction unit 12 replaces the pixel value of the pixel whose difference distance is less than the predetermined distance with a predetermined value (for example, zero as the minimum value), and replaces the pixel value of the pixel whose difference distance is the predetermined distance or more. Keep it as it is. For example, as illustrated in FIG. 20, the pixel extraction unit 12 sets the pixel value of the pixel corresponding to the difference distance E2b to zero because the difference distance E2b is less than the predetermined distance E2b1. The predetermined distance E2b1 is a distance corresponding to the predetermined height H1. Alternatively, as illustrated in FIG. 20, the pixel extraction unit 12 determines that the installation surface height H1b is less than the predetermined height H1, and replaces the pixel value of the pixel corresponding to the installation surface height H1b with zero. Good. Alternatively, the pixel extraction unit 12 determines that the ratio of the difference distance E2b to the installation surface distance E2 (hereinafter referred to as “difference ratio”) is less than a predetermined ratio, and replaces the pixel value of the corresponding pixel with zero. May be. The predetermined distance, the predetermined height, the predetermined ratio, and the like are set in advance for each pixel, each pixel row, or each pixel column in the input distance image of the rear stereo camera 6B based on the installation position and installation angle of the rear stereo camera 6B. Is done. Further, the predetermined height and the predetermined ratio may be common for all the pixels in the input distance image.

  FIG. 21 is a diagram illustrating an example of an output distance image of the rear stereo camera 6B. FIG. 21A shows an output distance image when pixel values of all the pixels in the input distance image are used as they are without performing extraction by the pixel extracting means 12. FIG. 21B shows an output distance image when the pixel extraction unit 12 extracts a pixel having a pixel value smaller than the installation surface distance corresponding to each pixel among the pixels in the input distance image. In FIG. 21C, the pixel extraction unit 12 extracts pixels from which the difference distance is equal to or greater than a predetermined distance from among the pixels in the input distance image, or pixels whose estimated height is equal to or greater than the predetermined height. The output distance image is shown.

  As shown in FIG. 21A, when pixel extraction is not performed, the output distance image represents the installation surface in such a manner that the luminance gradually decreases as the distance from the excavator 60 increases, and the object P3 and the raised portion P4 are displayed. Make the existence difficult to understand.

  On the other hand, as shown in FIG. 21B, when extracting a pixel having a pixel value smaller than the installation surface distance among the pixels in the input distance image, the output distance image is a feature of the feature existing on the installation surface. Improve visibility. Specifically, in the output distance image of FIG. 21B, since the pixel values of all the pixels having the pixel value equal to the installation surface distance are replaced with zero, regardless of the distance from the excavator 60, Make the color and brightness uniform. As a result, the output distance image of FIG. 21B can make the distance image of the object P3 and the raised portion P4 stand out from the distance image of the installation surface and improve the visibility.

  Furthermore, as shown in FIG. 21C, when extracting pixels whose difference distance is greater than or equal to a predetermined distance among the pixels in the input distance image, the output distance image has an installation surface height that is greater than or equal to the predetermined height. Improve the visibility of features. Specifically, in the output distance image of FIG. 21C, the pixel values of all the pixels whose difference distance is less than the predetermined distance are set to zero, so that the installation surface height is set regardless of the distance from the excavator 60. Makes the color and brightness of features (including the installation surface) less than a predetermined height uniform. As a result, in the output distance image of FIG. 21C, the distance image of the raised portion P4 whose installation surface height H1b is less than the predetermined height H1 is made into a single color and luminance together with the installation surface distance image. In the output distance image of FIG. 21C, only the distance image of the object P3 whose installation surface height is equal to or higher than the predetermined height H1 can be distinguished from other distance images, and the visibility can be improved.

  Note that the output distance image of FIG. 21C sets the distance image of the pixel P corresponding to the base portion of the object P3 (the portion whose installation surface height is less than the predetermined height H1). A single color and brightness along with the surface distance image. However, the input distance of the pixel corresponding to the part determined to be a part of the object P3 is the input distance even if the difference distance is less than the predetermined distance or the installation surface height is less than the predetermined height H1. You may maintain the pixel value in an image as it is. Whether or not a specific pixel belongs to a part of the distance image of the object P3 is determined based on, for example, a difference distance or installation surface height related to surrounding pixels and a difference distance or installation surface height related to the specific pixel. Can be done.

  FIG. 22 is a diagram illustrating an example of an output image of the rear camera 2B. FIG. 22A shows an output image when the output distance image of FIG. 21A is superimposed. FIG. 22B shows an output image when the output distance image of FIG. 21B is superimposed. FIG. 22C shows an output image when the output distance image of FIG. 21C is superimposed.

  As shown in FIG. 22A, the output image obtained by superimposing the output distance image of FIG. 21A represents the installation surface in such a manner that the luminance gradually decreases as the distance from the excavator 60 increases. The existence of P4 cannot be highlighted.

  On the other hand, as shown in FIG. 22B, an output image obtained by superimposing an output distance image in which the color and brightness of the installation surface are uniform regardless of the distance from the excavator 60 is an image of the object P3 and the raised portion P4. Make it stand out from the image of the installation surface and improve its visibility. Also, the output image of FIG. 22B superimposes and displays the distance value between each of the object P3 and the raised portion P4 and the rear stereo camera 6B. Thereby, the presence of the object P3 and the raised portion P4 can be further emphasized. Note that, for example, a value corrected by the distance correction unit 13 is adopted as the value of the distance between the object P3 and the rear stereo camera 6B. Further, the value of the distance between the object P3 and the rear stereo camera 6B may be displayed in a superimposed manner only when it is equal to or less than a predetermined value. This is to make the presence of the feature near the excavator 60 stand out further. Further, the brightness or color of the distance (pixel value) is determined according to the magnitude of the distance (pixel value). For example, the color of the value of the distance (pixel value) changes as purple, blue, green, yellow, and red as the distance (pixel value) decreases. The distance value may be the minimum value of the pixel values of the pixels included in the distance image area of the object P3 in the input distance image extracted by the pixel extraction unit 12. The same applies to the value of the distance between the raised portion P4 and the rear stereo camera 6B.

  Furthermore, as shown in FIG. 22C, the output distance in which the color and brightness of features (including the installation surface) whose installation surface height is less than the predetermined height are made uniform regardless of the distance from the excavator 60. The output image on which the image is superimposed improves the visibility of the object P3. Specifically, the output image highlights only the upper part of the image of the object P3 whose installation surface height is equal to or higher than a predetermined height from other images, and improves the visibility. Further, the output image of FIG. 22C superimposes and displays only the value of the distance between the object P3 whose installation surface height is equal to or higher than the predetermined height and the rear stereo camera 6B. Thereby, only the presence of the object P3 whose installation surface height is equal to or higher than the predetermined height can be further emphasized.

  In the above description, the image generation apparatus 100 displays the output distance image of the rear stereo camera 6 </ b> B on the display unit 5 by superimposing it on the output image of the rear camera 2. However, the image generating apparatus 100 may display only the output distance image on the display unit 5 or may display the output distance image and the output image side by side on the display unit 5.

  Further, the image generating apparatus 100 is shown in FIG. 18 instead of displaying the distance value between each of the object P3 and the raised portion P4 and the rear stereo camera 6B, or in addition to displaying the distance value. Such a frame may be displayed.

  In addition, the image generation apparatus 100 displays an index (not shown) such as a bar-shaped indicator with a distance scale indicating the relationship between the distance (pixel value) and the color or luminance at the end of the output distance image or the output image. May be.

  Next, the flow of pixel extraction processing will be described with reference to FIG. FIG. 23 is a flowchart showing the flow of pixel extraction processing. For example, the pixel extraction unit 12 of the image generation apparatus 100 executes this pixel extraction process every time an input distance image is acquired.

  First, the pixel extraction unit 12 reads and obtains an installation surface distance relating to one pixel (hereinafter referred to as “target pixel”) in the input distance image from a ROM or the like (step S11). Note that the pixel extraction unit 12 may read and acquire the installation surface angle from a ROM or the like.

  Thereafter, the pixel extraction unit 12 calculates a difference distance by subtracting the pixel value of the target pixel from the installation surface distance (step S12). The pixel extraction unit 12 may calculate the difference ratio based on the calculated difference distance and the installation surface distance, or may calculate the estimated height based on the calculated difference distance and the installation surface angle. .

  Thereafter, the pixel extraction unit 12 compares the calculated difference distance with a predetermined distance stored in the ROM or the like (step S13). The pixel extracting unit 12 may compare the calculated difference ratio with a predetermined ratio stored in the ROM or the like, or may compare the calculated estimated height with a predetermined height stored in the ROM or the like.

  When the difference distance is equal to or greater than the predetermined distance (YES in step S13), the pixel extraction unit 12 extracts the target pixel as a pixel that maintains the current pixel value as it is. The same applies to the case where the difference ratio is greater than or equal to the predetermined ratio or the estimated height is greater than or equal to the predetermined height.

  On the other hand, when the difference distance is less than the predetermined distance (NO in step S13), the pixel extraction unit 12 does not extract the target pixel as a pixel that maintains the current pixel value as it is. Specifically, the pixel extraction unit 12 replaces the pixel value of the target pixel with a predetermined value stored in a ROM or the like (step S14). The same applies when the difference ratio is less than the predetermined ratio or when the estimated height is less than the predetermined height.

  In the above-described embodiment, the process of replacing the pixel value of the target pixel that has not been extracted with a predetermined value is performed when the necessity of extraction is determined for each target pixel, but for a plurality of or all target pixels. It may be performed collectively after the necessity of extraction is determined.

  Thereafter, the pixel extraction unit 12 determines whether or not extraction of all the pixels in the acquired input distance image has been determined (step S15).

  If it is determined that the necessity of extraction has not yet been determined for all the pixels (NO in step S15), the pixel extraction unit 12 performs the processing from step S11 on another target pixel.

  If it is determined that the extraction necessity is determined for all the pixels (YES in step S15), the pixel extraction unit 12 completes the current pixel extraction process.

  With the above configuration, the image generation apparatus 100 can improve the visibility of the distance image regarding the feature that the excavator 60 may contact by omitting the detailed display of the distance image regarding the installation surface.

  Further, the image generation apparatus 100 omits detailed display of features whose installation surface height is less than a predetermined height, so that the visibility of features (for example, workers) whose installation surface height is equal to or higher than the predetermined height is displayed. Can be improved.

  As a result, the image generating apparatus 100 can more reliably prevent the excavator 60 from colliding with the feature.

  Next, a process (hereinafter referred to as “driving support process”) in which the driving support means 15 executes the driving support function will be described with reference to FIG. FIG. 24 is a flowchart showing the flow of the driving support process, and the driving support means 15 repeatedly executes this driving support process at a predetermined cycle while the excavator 60 is operating.

  First, the driving support means 15 detects the operating direction of the excavator 60 (the moving direction, the turning direction, the operating direction of the excavation attachment E, etc.) based on the operating direction of the operating lever (not shown) of the excavator 60. (Step S31).

  Thereafter, the driving support means 15 determines whether or not a predetermined feature exists in the operating direction of the excavator 60 (step S32). In this embodiment, for example, when the driving support means 15 detects that the excavator 60 is moving backward, based on the input distance image acquired by the rear stereo camera 6B, the driving support means 15 It is determined whether there is an obstacle.

  If it is determined that there is an obstacle (YES in step S32), the driving support means 15 executes the driving support function (step S33). In the present embodiment, the driving support means 15 sounds an alarm and displays a warning message notifying that an obstacle exists on the display unit 5. The driving support means 15 may reduce the reverse speed of the shovel 60 as the distance to the obstacle becomes shorter, and may stop the shovel 60 when the distance to the obstacle falls below a predetermined value. . Similarly, when the driving support means 15 detects that the excavator 60 is turning, or when it is detected that the excavation attachment E is operating, the excavator 15 also decreases as the distance to the obstacle decreases. The turning speed of 60 or the operating speed of the excavation attachment E may be reduced, and the turning of the excavator 60 or the operation of the excavation attachment E may be stopped when the distance to the obstacle falls below a predetermined value. In addition, when two or more of the driving, turning, and excavation attachment E operations are performed simultaneously, the driving support means 15 may execute these restrictions individually, or execute them in combination. May be.

  On the other hand, when it is determined that there is no obstacle (NO in step S32), the driving support means 15 ends the current driving support process without executing the driving support function.

  With the above configuration, the image generation apparatus 100 automatically determines the presence / absence of a predetermined feature based on the output of the stereo camera 6 and executes the driving support function as necessary, thereby enabling the operation of the excavator 60 to be performed. Safety can be further improved.

  Next, a process in which the image generation apparatus 100 corrects the pixel value (distance) of the input distance image (hereinafter referred to as “distance correction process”) will be described with reference to FIGS.

  FIG. 25 is a view showing the excavator 60 on which the image generating apparatus 100 is mounted, FIG. 25 (A) is a top view thereof, and FIG. 25 (B) is a partial right side view thereof. In the embodiment shown in FIG. 25, the excavator 60 includes one camera 2 (rear camera 2B) and one stereo camera 6 (rear stereo camera 6B). Note that a region CB indicated by a one-dot chain line in FIG. 25 indicates an imaging range of the rear camera 2B. A region ZB indicated by a dotted line in FIG. 25 indicates the imaging range of the rear stereo camera 6B.

  As shown in FIG. 25, behind the excavator 60, there are two objects P1 and P2 extending vertically upward from the ground. The two objects P1 and P2 are rod-shaped objects having different heights and exist in different directions when viewed from the rear stereo camera 6B. Specifically, as shown in FIG. 25B, the height of the object P1 is lower than the height of the object P2. However, the shortest distance between each of the two objects P1, P2 and the rear stereo camera 6B is equal to the distance D. This is because the lower object P1 is closer to the shovel 60 than the higher object P2.

  Further, as shown in FIG. 25B, a line segment connecting the closest point on the object P1 viewed from the stereo camera 6B and the stereo camera 6B forms an angle γ1 with the horizontal plane. Similarly, the line segment connecting the closest point on the object P2 viewed from the stereo camera 6B and the stereo camera 6B forms an angle γ2 with the horizontal plane. In this case, the horizontal distance D1 between the object P1 and the plane including the rear surface of the shovel 60 is represented by D × cos (γ1), and the horizontal distance D2 between the object P2 and the plane including the rear surface of the excavator 60 is , D × cos (γ2). The horizontal distance D2 is greater than the horizontal distance D1.

  The pixel value in the input distance image captured by the stereo camera 6B represents the distance between the feature and the stereo camera 6B. Therefore, when the pixel value in the input distance image is used as it is, the top of the object P1 and the top of the object P2 are displayed in the same color or the same brightness in the process of changing the display color or brightness according to the distance from the stereo camera 6B. Will be. This display is erroneously recognized by the operator that the object P1 and the object P2 exist at the same distance from the rear surface of the shovel 60 even when the object P1 is closer to the shovel 60 than the object P2 as described above. There is a risk of causing.

  Such a situation occurs because the rear stereo camera 6 </ b> B captures an image of the obliquely lower side while being attached to the upper part of the upper swing body 63. That is, the distance recognized by the operator is the horizontal distance from the excavator 60, whereas the distance acquired by the stereo camera 6 is a linear distance from the stereo camera 6. Note that the mounting position of the stereo camera 6B is the upper part of the upper swing body 63 so that the sensor can be protected from contact with surrounding features.

  Therefore, the distance correction unit 13 of the image generation apparatus 100 includes, for example, a pixel value indicating the distance between each of the objects P1 and P2 and the stereo camera 6B, and a plane including each of the objects P1 and P2 and the rear surface of the excavator 60. To a pixel value representing a horizontal distance between the two.

  Specifically, the distance correction unit 13 connects the stereo camera 6B and an arbitrary point on the object P1 from the coordinates of the pixel in the input distance image corresponding to the arbitrary point on the object P1 viewed from the stereo camera 6B. The angle γ that the line segment forms with the horizontal plane is derived. Since the mounting position and mounting angle of the stereo camera 6B are known, the distance correction means 13 can derive the angle γ from the coordinates of the pixel.

  Thereafter, the distance correction means 13 adds the pixel value of the pixel, that is, the stereo camera 6B and the object P1, to the cosine cos (γ) of the angle γ derived from the coordinates of the pixel corresponding to an arbitrary point on the feature. Multiply the distance between any point. In this way, the distance correction unit 13 calculates the horizontal distance between the stereo camera 6B and an arbitrary point on the object P1, and calculates the pixel value of the pixel corresponding to the arbitrary point by the calculated horizontal distance. replace.

  Similarly, the distance correcting unit 13 derives an angle γ formed between a line segment connecting the stereo camera 6B and an arbitrary point on the object P2 with the horizontal plane. Then, the cosine cos (γ) of the angle γ is multiplied by the pixel value, that is, the distance between the stereo camera 6B and an arbitrary point on the object P2. Then, the horizontal distance between the stereo camera 6B and an arbitrary point on the object P2 is calculated, and the pixel value of the pixel corresponding to the arbitrary point is replaced with the calculated horizontal distance.

  Here, the effect of the correction by the distance correction means 13 will be described with reference to FIGS.

  FIG. 26 shows an output distance image based on an input distance image that has not been subjected to distance correction processing (hereinafter referred to as “pre-correction output distance image”) and an output distance based on an input distance image that has been subjected to distance correction processing. FIG. 26A is a comparison diagram showing a difference from an image (hereinafter referred to as “corrected output distance image”), FIG. 26A shows an output distance image before correction, and FIG. 26B shows a corrected output distance image. Indicates. In this embodiment, pixel extraction processing has already been performed on the input distance image.

  FIG. 27 shows a combined output image obtained by combining a pre-correction output distance image with an output image generated based on the input image of the camera 2B (hereinafter referred to as “pre-correction combined output image”), and its output image. FIG. 27A is a comparison diagram showing a difference from a combined output image obtained by combining a corrected output distance image with an output image (hereinafter referred to as “corrected combined output image”), and FIG. An output image is shown, and FIG. 27B shows a corrected combined output image.

  As shown in FIGS. 26A and 27A, the object P1 and the object P2 are represented in the same color scheme in the output distance image before correction and the output image after combination before correction. Specifically, both of the object P1 and the object P2 are displayed whiter (thin) as it approaches the top, and black (darker) as it approaches the bottom. Therefore, it is difficult for the operator to determine which object is closer to the excavator 60.

  In contrast, as shown in FIGS. 26B and 27B, in the corrected output distance image and the corrected combined output image, the object P1 and the object P2 are represented by different colors. Specifically, the object P1 is displayed as a thin overall, and the object P2 is displayed as a dark overall. Therefore, the operator can easily recognize that the object P1 is closer to the excavator 60 than the object P2.

  FIG. 28 is another example of a comparison diagram showing the difference between the pre-correction combined output image and the corrected combined output image, and shows an example in which frames are displayed in a superimposed manner.

  As shown in FIG. 28A, in the pre-correction synthesized output image, when the minimum pixel value of the pixel included in the feature region is adopted as the representative value of the feature region, the object P1 and the object P2 Are displayed as being at a distance of 1 meter from the excavator 60.

  On the other hand, as shown in FIG. 28B, in the corrected combined output image, horizontal distances D1 and D2 between each of the objects P1 and P2 and a plane including the rear surface of the excavator 60 are displayed. That is, it is displayed that the object P1 is at a distance of 1 meter from the rear surface of the excavator 60 and the object P2 is at a distance of 1.5 meters from the rear surface of the excavator 60. Therefore, the operator can easily recognize that the object P1 is closer to the excavator 60 than the object P2.

  With the above configuration, the image generation apparatus 100 corrects the pixel value so that the pixel value in the input distance image captured by the stereo camera 6 represents the horizontal distance from the shovel 60. As a result, the image generating apparatus 100 can inform the operator of the status of surrounding features in an easy-to-understand manner.

  In the above-described embodiment, the distance correction unit 13 individually corrects the pixel values of the pixels extracted by the pixel extraction unit 12. However, the present invention is not limited to this. For example, the distance correction unit 13 collects predetermined corrections for each pixel row, each pixel column, or each pre-divided area in the input distance image plane without calculating the horizontal distance using the cosine as described above. May be performed. This is to obtain the same effect as the case of calculating the horizontal distance using the cosine by simple correction while suppressing the calculation load. Specifically, the distance correction unit 13 is configured so that, for example, a pixel row located above the input distance image plane increases correction of the distance, or a pixel row located below the input distance image plane decreases the distance. Corrections with a predetermined correction amount may be collectively executed so that the correction becomes large.

  In the above-described embodiment, the distance correction unit 13 performs correction when the stereo camera 6B is attached to the upper part of the rear surface of the upper swing body 63 and picks up an image obliquely downward. However, the present invention is not limited to this. For example, the distance correction unit 13 may perform correction when the stereo camera captures an image in the horizontal direction, or may perform correction when the stereo camera captures an image from above obliquely.

  In the above-described embodiment, the distance correcting unit 13 uses the plane including the rear surface of the excavator 60 as a reference, and calculates the pixel value of the pixel included in the feature region between the feature and the plane including the rear surface of the excavator 60. The pixel value representing the horizontal distance between them is corrected. However, the present invention is not limited to this. For example, the distance correcting unit 13 uses the rotation axis PV of the excavator 60 as a reference, the pixel value of the pixel included in the feature region, and the pixel value representing the horizontal distance between the feature and the rotation axis PV of the excavator 60. You may correct to.

  The distance correction unit 13 may correct the pixel value of the pixel included in the feature region to a pixel value that represents the shortest horizontal distance between the feature and the excavator 60. In this case, the distance correction unit 13 selects a reference from a plane including the right side surface of the excavator 60, a plane including the left side surface of the excavator 60, a plane including the rear surface of the excavator 60, and the like according to the actual position of the feature. .

  The shortest horizontal distance may be the shortest horizontal distance in a direction parallel to the moving direction of the excavator 60, that is, the front-rear direction of the lower traveling body 61. In this case, the value of the pixel related to the feature that is not in the moving direction of the shovel 60 is corrected to the maximum value (infinite) even if the feature is near the shovel 60. As a result, the operator can more easily recognize the feature in the moving direction of the excavator 60.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 employs the cylindrical spatial model MD as the spatial model, but may employ a spatial model having other columnar shapes such as a polygonal column, A spatial model composed of two side surfaces may be adopted, or a spatial model having only side surfaces may be adopted.

  The image generating apparatus 100 is mounted on a self-propelled excavator with a movable member such as a bucket, an arm, a boom, and a turning mechanism together with a camera and a stereo camera, and displays the surrounding image to the operator while displaying the surrounding image of the excavator. An operation support system that supports movement and operation of these movable members is configured. However, the image generating apparatus 100 is a work machine that does not have a turning mechanism such as a forklift or asphalt finisher, or a work machine that has a movable member such as an industrial machine or a fixed crane, but does not self-propelled. You may comprise the operation assistance system mounted with a camera and a stereo camera, and assisting operation of these work machines.

  In the above-described embodiment, the distance image combining unit 14 generates an output distance image and then combines the generated output distance image with the output image. However, the present invention is not limited to this. For example, the distance image synthesizing unit 14 synthesizes the input image and the input distance image to generate a combined input image, generates a post-combination processing target image on which information related to the distance image is superimposed and displays the combined image. An output image may be generated. In addition, the distance image synthesis unit 14 may generate a combined output image after combining a processing target image and a processing target distance image to generate a combined processing target image, for example.

  In the above-described embodiment, the image generation apparatus 100 includes the camera 2 and the stereo camera 6 separately. Then, the image generation device 100 generates one output image (viewpoint conversion image) based on a plurality of input images from the camera 2 and one output based on the plurality of input distance images from the stereo camera 6. A distance image (viewpoint conversion distance image) is generated. After that, the image generation apparatus 100 combines the output distance image with the output image. However, the present invention is not limited to this configuration. For example, the image generation device 100 generates an output image (viewpoint conversion image) based on three images output by each one imaging mechanism in each of the left-side stereo camera 6L, the rear-side stereo camera 6B, and the right-side stereo camera 6R. In addition, one output distance image (viewpoint conversion distance image) may be generated based on the three input distance images of the three stereo cameras. In this case, the camera 2 can be omitted. In this case, the left-side stereo camera 6L, the rear stereo camera 6B, and the right-side stereo camera 6R are installed at substantially the center of each of the left side surface, the rear surface, and the right side surface of the upper swing body 63 so that the imaging range is not biased. It is desirable that Therefore, a plurality of sets of imaging mechanisms in each stereo camera are juxtaposed at predetermined intervals in the vertical direction. This is because it is easy to install the stereo camera almost at the center of each surface regardless of the size of the juxtaposition interval. Further, when a plurality of sets of image pickup mechanisms are juxtaposed in the horizontal direction, it is difficult to install the stereo camera at substantially the center of each surface depending on the juxtaposition interval.

  In the above-described embodiment, the stereo camera 6 derives the distance between the object captured in each pixel of the basic image and the stereo camera 6 based on the parallax between images output by each of the two sets of imaging mechanisms. Then, the stereo camera 6 generates an input distance image as the distance information of the two-dimensional array by substituting the derived distance into the value of each pixel of the basic image, and the generated input distance image is transmitted to the control unit 1. Output. Then, the control unit 1 generates one output distance image using a plurality of input distance images from the plurality of stereo cameras 6. However, the present invention is not limited to this configuration. For example, the image generation apparatus 100 generates a first output image (first viewpoint conversion image) based on three images output from the first imaging mechanism in each of the three stereo cameras, and the three stereo cameras. A second output image (second viewpoint conversion image) is generated based on the three images output by the second imaging mechanism in each of the above. In addition, the image generation apparatus 100 determines the distance between an object that appears in each pixel of the first viewpoint conversion image and a predetermined reference point based on the parallax between the first viewpoint conversion image and the second viewpoint conversion image. Thus, the output distance image may be generated without using the input distance image.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 6 .... Stereo camera 6L ... Left side stereo camera 6R ... Right side stereo camera 6B ... Rear stereo camera 10 ... Coordinate matching means 11 ... Image generation means 12 ... Pixel extraction means 13 ··· Distance correction means 14 ··· Distance image synthesis means 15 ··· Driving support means 40 ··· Input image / spatial model correspondence map 41 ··· Spatial model · processing target image correspondence map 42 ··· processing target image Output image correspondence map 60: excavator 61 ... lower traveling body 62 ... turning mechanism 63 ... upper turning body 64 ... cab 100 ... image generating device

Claims (11)

A traveling body,
A swivel mounted on the traveling body so as to be turnable;
A plurality of imaging mechanisms attached to the revolving structure;
Display means for displaying an overhead image for displaying the periphery of the excavator, which is generated based on an image on the side of the revolving body and an image behind the revolving body;
Control means for determining the presence or absence of an object having a predetermined height around the swivel body based on outputs of the plurality of imaging mechanisms;
The bird's-eye view image is displayed by combining a side image of the swivel body and an image behind the swivel body so that the respective images are simultaneously displayed obliquely behind the swivel body,
The control means determines that the object is present by the presence of the object at a position where imaging ranges of the plurality of imaging mechanisms overlap,
The overlapping position includes an oblique rear side of the swivel body where a side image of the swivel body and a rear image of the swivel body are combined,
When it is determined that an object is present, the control means outputs an alarm, displays a warning on the overhead image, lights or flashes a warning light, decelerates the shovel, or Limit the operating speed of the excavator, or stop the operation of the excavator,
Excavator . The plurality of imaging mechanisms constitute a stereo camera,
The stereo camera is attached to the side of the revolving body and captures a side of the revolving body from an obliquely upper side, and the stereo camera is attached to the rear of the revolving body and the rear of the revolving body is obliquely upward. Including a rear stereo camera that captures images from
The display means displays a specific image that specifies an image corresponding to the object determined to exist by the control means on the overhead image,
The specific image is located at a position closer than a predetermined distance from the revolving structure, and when an object exists in at least one of the imaging range of the side stereo camera and the imaging range of the rear stereo camera. A corresponding image is identified on the overhead image,
The excavator according to claim 1. Depending on the distance between the object and the swivel body, the color or shade of the specific image changes.
The shovel according to claim 2. The specific image has the same shape as the image of the object, or is a frame image formed so as to surround the image of the object.
The shovel according to claim 2 or 3. The display means displays a CG image of the excavator,
The overhead view image is a viewpoint conversion image surrounding the CG image,
The image of the object is displayed closer to the CG image as it is closer to the ground, and is displayed as far from the CG image as it is farther from the ground.
The excavator according to any one of claims 2 to 4. The bird's-eye view image is an image representing a state when the periphery of the excavator is viewed from above, and the specific image is superimposed and displayed.
The excavator according to any one of claims 2 to 5. The control means generates the overhead image based on at least distance information output from the side stereo camera and an input image, and the distance information corresponds to each of a plurality of pixels constituting the overhead image. Including information on the distance between the object around the swivel body and the side stereo camera;
The excavator according to any one of claims 2 to 6. The control means generates the overhead view image based on at least the distance information and the input image output from the rear stereo camera, and the distance information corresponds to each of a plurality of pixels constituting the overhead view image, Including information about the distance between the object around the swivel body and the rear stereo camera,
The excavator according to any one of claims 2 to 7. The control means, without correcting the pixel value of the pixel corresponding to the distance equal to the installation surface distance, which is the distance between the plane on which the excavator is located and the side stereo camera, from the installation surface distance Correcting the pixel value of the pixel corresponding to the small distance to generate the overhead image,
The excavator according to claim 7 or 8. The control means is smaller than the installation surface distance without correcting the pixel value of the corresponding pixel having a distance equal to the installation surface distance that is the distance between the plane where the excavator is located and the rear stereo camera. Correcting the pixel value of the pixel corresponding to the distance to generate the overhead image,
The excavator according to any one of claims 7 to 9. The control means generates the overhead image by correcting a pixel value of a pixel corresponding to a distance corresponding to a difference that is greater than or equal to a predetermined value with respect to the installation surface distance.
The excavator according to claim 9 or 10.