JP6873889B2 - Excavator and excavator surrounding image generator - Google Patents

Description

The present invention relates to a shovel and an excavator surrounding image generator that generate an output image based on a plurality of input images captured by a camera mounted on the upper swing body of the shovel.

Conventionally, there is known a driving support system that converts each of the images captured by a plurality of in-vehicle cameras that capture the surroundings of a vehicle into a bird's-eye view image and presents an output image obtained by connecting the bird's-eye view images to the driver (for example, , Patent Document 1).

JP-A-2007-109166

However, the driving support system described in Patent Document 1 is not supposed to be mounted on a construction machine that performs a turning motion or excavation, and the visibility cannot be improved when the driving support system is provided in the driver's cab of an excavator.

In view of the above points, it is desirable to improve the visibility of the output image generated based on the plurality of input images captured by the camera.

The excavator according to the embodiment of the present invention includes a lower traveling body that performs a traveling operation, an upper rotating body that is rotatably mounted on the lower traveling body, and a boom that is attached to the upper rotating body and included in an attachment. An arm attached to the boom and included in the attachment, and an image pickup device mounted on the left, right, and rear parts of the upper swivel body so as to image three directions of the upper swivel body. The driver's cab mounted on the upper swivel body, the display unit installed in the driver's cab, and the display unit are at least left and right of the upper swivel body regardless of the orientation of the lower traveling body. And the overlapping regions of the left rear and the right rear of the upper swing body whose imaging range overlaps with the rear region of the imaging device so as to project the excavator in a state of being viewed from directly above. An output image portion is generated for the overlapping region by synthesizing each captured image of the device, and an image obtained by the imaging device is used as an output image in a state including the output image portion regardless of the orientation of the lower traveling body. In addition to displaying, the control unit includes a control unit that displays the output image without simultaneously outputting the input images from the image pickup apparatus mounted at the three locations in the display area centered on the output image.

By the above means, the visibility of the output image generated based on the plurality of input images captured by the camera is improved.

It is a block diagram which shows schematic the structural example of the image generation apparatus which concerns on this invention. It is a figure which shows the configuration example of the excavator which mounts an image generator. It is a figure which shows an example of the spatial model which the input image is projected. It is a figure which shows an example of the relationship between a spatial model and an image plane to be processed. It is a figure for demonstrating the correspondence between the coordinates on the input image plane, and the coordinates on a space model. It is a figure for demonstrating the correspondence between coordinates by the coordinate correspondence means. It is a figure for demonstrating the action of a group of parallel lines. It is a figure for demonstrating the action of the auxiliary line group. It is a flowchart which shows the flow of the processing target image generation processing and output image generation processing. This is a display example of the output image (No. 1). This is a display example of the output image (No. 2). It is a figure for demonstrating the same color illusion. It is a figure for demonstrating an example of the method of generating the light-dark pattern which brings about the same color illusion. It is a figure which shows the state which arranged the lattice pattern of FIG. 13 on the output image plane. It is a flowchart which shows the flow of the lattice pattern forming process. It is a comparison diagram which shows the difference between the output image shown in FIG. 11 and the output image to which the checkered pattern which brings about the same color illusion is applied to the output image of FIG. This is a display example of the output image (No. 3). It is a figure for demonstrating the effect of preventing disappearance of an object in the overlap area of the imaging range of each of two cameras. It is a comparison diagram which shows the difference between the output image shown in FIG. 17 and the output image to which the checkered pattern which brings about the same color illusion is applied to the output image of FIG.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generation device 100 according to the present invention.

The image generation device 100 is, for example, a device that generates an output image based on an input image captured by a camera 2 mounted on a construction machine and presents the output image to the driver. It is composed of an input unit 3, a storage unit 4, and a display unit 5.

FIG. 2 is a diagram showing a configuration example of an excavator 60 on which the image generator 100 is mounted. The excavator 60 has an upper swivel body 63 placed on a crawler type lower traveling body 61 via a swivel mechanism 62. It is mounted freely around the swivel shaft PV.

Further, the upper swing body 63 is provided with a cab (driver's cab) 64 on the front left side thereof, an excavation attachment E on the front center portion thereof, and a camera 2 (right side camera 2R, rear camera 2B) on the right side surface and the rear surface thereof. ) Is provided. It is assumed that the display unit 5 is installed at a position in the cab 64 that is easily visible to the driver.

Next, each component of the image generation device 100 will be described.

The control unit 1 is a computer equipped with a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. A program corresponding to each of the attachment means 10 and the output image generation means 11 is stored in a ROM or NVRAM, and the CPU is made to execute a process corresponding to each means while using the RAM as a temporary storage area.

The camera 2 is a device for acquiring an input image that reflects the periphery of the excavator 60, and is attached to, for example, the right side surface and the rear surface of the upper swivel body 63 so that an area that becomes a blind spot of the driver in the cab 64 can be imaged. (See FIG. 2), a right-side camera 2R and a rear camera 2B equipped with an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side surface and the rear surface (for example, the front surface and the left side surface) of the upper swing body 63, and a wide-angle lens or a fisheye lens is attached so as to be able to image a wide range. You may be.

Further, the camera 2 acquires an input image according to the control signal from the control unit 1, and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the camera 2 sends the corrected input image corrected to the apparent distortion and tilt caused by using those lenses to the control unit 1. Although it is output, the input image in which the apparent distortion and tilt are not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 will correct the apparent distortion and tilt.

The input unit 3 is a device for allowing the operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

The display unit 5 is a device for displaying image information, for example, a liquid crystal display or a projector installed in a cab 64 (see FIG. 2) of a construction machine, and various types output by the control unit 1. Display the image.

Further, the image generation device 100 generates a processing target image based on the input image, and performs image conversion processing on the processing target image so that the positional relationship with the surrounding obstacles and the sense of distance can be intuitively grasped. After generating the output image to be output, the output image may be presented to the driver.

The "processed image" is an image that is generated based on the input image and is the target of image conversion processing (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.), and is, for example, the ground surface. When an input image obtained by a camera that captures an image from above and includes a horizontal image (for example, an empty part) due to its wide angle of view is used in the image conversion process, the horizontal image is The input image is projected onto a predetermined spatial model so that it is not displayed unnaturally (for example, the empty part is not treated as if it is on the ground surface), and then the projected image projected on the spatial model is transformed into another two. This is an image suitable for image conversion processing, which is obtained by reprojecting onto a dimensional plane. The image to be processed may be used as it is as an output image without performing image conversion processing.

The "spatial model" is at least a plane or curved surface other than the processing target image plane which is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane or an angle with the processing target image plane. It is a projection target of an input image composed of one or more planes or curved surfaces including a plane or curved surface to be formed.

The image generation device 100 may generate an output image by performing an image conversion process on the projected image projected on the spatial model without generating the image to be processed. Further, the projected image may be used as it is as an output image without performing image conversion processing.

FIG. 3 is a diagram showing an example of the space model MD on which the input image is projected, and FIG. 3 (A) shows the relationship between the excavator 60 and the space model MD when the excavator 60 is viewed from the side. Shown, FIG. 3B shows the relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from above.

As shown in FIG. 3, the spatial model MD has a semi-cylindrical shape, and has a plane region R1 inside the bottom surface thereof and a curved surface region R2 inside the side surface thereof.

Further, FIG. 4 is a diagram showing an example of the relationship between the spatial model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the spatial model MD. In FIG. 4, for the sake of clarification, the space model MD is shown in a cylindrical shape instead of the semi-cylindrical shape as shown in FIG. 3, but the space model MD is either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the following figures. Further, as described above, the image plane R3 to be processed may be a circular region including the plane region R1 of the space model MD, or may be an annular region not including the plane region R1 of the space model MD.

Next, various means included in the control unit 1 will be described.

The coordinate mapping means 10 is a means for associating the coordinates on the input image plane on which the input image captured by the camera 2 is located, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3. For example, various parameters related to the camera 2 such as the optical center of the camera 2, the focal distance, the CCD size, the optical axis direction vector, the camera horizontal direction vector, and the projection method, which are preset or input via the input unit 3. And, based on the mutual positional relationship between the input image plane, the spatial model MD, and the processing target image plane R3, the coordinates on the input image plane, the coordinates on the spatial model MD, and the processing target image are determined in advance. The coordinates on the plane R3 are associated with each other, and their correspondence is stored in the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 of the storage unit 4.

When the coordinate mapping means 10 does not generate the processing target image, the coordinate mapping between the coordinates on the spatial model MD and the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

The output image generation means 11 is a means for generating an output image. For example, by performing scale conversion, affine conversion, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image can be obtained. An input image / spatial model in which the coordinates on the position output image plane are associated with each other, the correspondence is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the coordinate matching means 10 stores the values. The value of each pixel in the output image (for example, brightness value, hue value, saturation value, etc.) and the value of each pixel in the input image while referring to the correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image by associating with.

Further, the output image generation means 11 is preset, or is input via the input unit 3, the optical center, focal distance, CCD size, optical axis direction vector, camera horizontal direction vector, and projection method of the virtual camera. Based on various parameters such as, the coordinates on the processing target image plane R3 and the coordinates on the output image plane where the output image is located are associated with each other, and the correspondence relationship is matched with the processing target image / output image correspondence map 42 of the storage unit 4. The value of each pixel in the output image (for example, the brightness value) while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored in , The hue value, the saturation value, etc.) and the value of each pixel in the input image are associated with each other to generate an output image.

The output image generation means 11 may generate the output image by changing the scale of the image to be processed without using the concept of the virtual camera.

Further, when the output image generation means 11 does not generate the image to be processed, the output image generation means 11 associates the coordinates on the spatial model MD with the coordinates on the output image plane according to the image conversion process performed, and inputs the input image / spatial model. While referring to the correspondence map 40, the output image is generated by associating the value of each pixel in the output image (for example, the brightness value, the hue value, the saturation value, etc.) with the value of each pixel in the input image. In this case, the output image generation means 11 omits associating the coordinates on the processing target image plane R3 with the coordinates on the output image plane and storing the correspondence in the processing target image / output image correspondence map 42. To do.

Next, an example of specific processing by the coordinate mapping means 10 and the output image generation means 11 will be described.

The coordinate mapping means 10 can map the coordinates on the input image plane and the coordinates on the spatial model by using, for example, Hamilton's quaternion.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the spatial model, and the input image plane of the camera 2 is a UVW Cartesian coordinate with the optical center C of the camera 2 as the origin. It is represented as a plane in the system and the spatial model is represented as a solid plane in the XYZ Cartesian coordinate system.

First, the coordinate mapping means 10 optically coordinates the origin of the XYZ coordinate system in order to convert the coordinates on the spatial model (coordinates on the XYZ coordinate system) into the coordinates on the input image plane (coordinates on the UVW coordinate system). After moving in parallel to the center C (origin of the UVW coordinate system), the X-axis is the U-axis, the Y-axis is the V-axis, and the Z-axis is the -W-axis (the sign "-" is in the opposite direction. This is due to the fact that the UVW coordinate system has the front of the camera in the + W direction and the XYZ coordinate system has the -Z direction in the vertical direction.) The XYZ coordinate system is rotated so as to match each other.

When a plurality of cameras 2 exist, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate mapping means 10 parallels the XYZ coordinate systems to each of the plurality of UVW coordinate systems. It will be moved and rotated.

In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis, and the X axis is U. Since it is realized by rotating so as to coincide with the axis, the coordinate mapping means 10 can combine these two rotations into one rotation operation by describing this conversion in Hamilton's quaternary number. it can.

By the way, the rotation for matching a certain vector A with another vector B corresponds to the process of rotating by the angle formed by the vector A and the vector B about the normal of the surface where the vector A and the vector B extend. Assuming that the angle is θ, the angle θ is calculated from the inner product of the vector A and the vector B.

で表されることとなる。

Will be represented by.

Further, the unit vector N of the normal of the surface extending between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

で表されることとなる。

Will be represented by.

The quaternion is when i, j, and k are imaginary units, respectively.

を満たす超複素数であり、本実施例において、四元数Ｑは、実成分をｔ、純虚成分をａ、ｂ、ｃとして、

In this embodiment, the quaternion Q is a hypercomplex number satisfying the above conditions, where t is the real component and a, b, and c are the pure imaginary components.

で表されるものとし、四元数Ｑの共役四元数は、

The conjugate quaternion of the quaternion Q is

で表されるものとする。

It shall be represented by.

The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, and c while setting the real component t to 0 (zero), and t, a, b. It is also possible to express a rotation motion about an arbitrary vector by each component of, c.

Further, the quaternion Q can be expressed as one rotation operation by integrating a plurality of consecutive rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. The point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as the axis can be expressed as follows.

ここで、本実施例において、Ｚ軸を−Ｗ軸に一致させる回転を表す四元数をＱｚとすると、ＸＹＺ座標系におけるＸ軸上の点Ｘは、点Ｘ'に移動させられるので、点Ｘ'は、

Here, in this embodiment, assuming that the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X', so that the point X'is

で表されることとなる。

Will be represented by.

Further, in this embodiment, assuming that the quaternion representing the rotation that makes the line connecting the point X'on the X axis and the origin coincide with the U axis is Qx, "the Z axis matches the -W axis, and further. , The quaternion R representing "rotation that makes the X-axis coincide with the U-axis"

で表されることとなる。

Will be represented by.

From the above, the coordinates P'when the arbitrary coordinates P on the spatial model (XYZ coordinate system) are expressed by the coordinates on the input image plane (UVW coordinate system) are

で表されることとなり、四元数Ｒがカメラ２のそれぞれで不変であることから、座標対応付け手段１０は、以後、この演算を実行するだけで空間モデル（ＸＹＺ座標系）上の座標を入力画像平面（ＵＶＷ座標系）上の座標に変換することができる。

Since the quaternion R is invariant in each of the cameras 2, the coordinate mapping means 10 can obtain the coordinates on the spatial model (XYZ coordinate system) simply by executing this calculation thereafter. It can be converted to coordinates on the input image plane (UVW coordinate system).

After converting the coordinates on the spatial model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate mapping means 10 uses the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. The incident angle α formed by the line segment CP'connecting the arbitrary coordinates P on the model with the coordinates P'represented by the UVW coordinate system and the optical axis G of the camera 2 is calculated.

Further, the coordinate mapping means 10 has the intersection E and the coordinates P'of the plane H and the optical axis G in the plane H parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and including the coordinates P'. The deviation angle φ formed by the line segment EP'connecting the two and the U'axis in the plane H, and the length of the line segment EP'are calculated.

Since the image height h is usually a function of the incident angle α and the focal length f in the optical system of the camera, the coordinate mapping means 10 normally projects (h = ftanα) and stereographic projection (h = fsinα). , Stereographic projection (h = 2ftan (α / 2)), stereographic projection (h = 2fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

After that, the coordinate mapping means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the deviation angle φ, and uses a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, the coordinates P (P') on the spatial model MD can be associated with the coordinates on the input image plane R4.

Assuming that the pixel size per pixel in the U-axis direction of the input image plane R4 is a _U and the pixel size per pixel in the V-axis direction of the input image plane R4 is a _V , the coordinates P on the spatial model MD The coordinates (u, v) on the input image plane R4 corresponding to (P') are

で表されることとなる。

Will be represented by.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on one or more input image planes R4 existing for each camera, and coordinates on the spatial model MD and the camera identifier. , And the coordinates on the input image plane R4 are associated and stored in the input image / spatial model correspondence map 40.

Further, since the coordinate mapping means 10 calculates the coordinate conversion using quaternions, it has an advantage that gimbal lock is not generated unlike the case where the coordinate conversion is calculated using Euler angles. .. However, the coordinate mapping means 10 is not limited to the one that calculates the coordinate conversion using quaternions, and may calculate the coordinate conversion using Euler angles.

When it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate mapping means 10 inputs the coordinates P (P') on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

Next, among the coordinates on the spatial model MD, the process of reprojecting the coordinates on the curved surface region R2 (coordinates having a component in the Z-axis direction) onto the processing target image plane R3 on the XY plane will be described.

FIG. 6 is a diagram for explaining the mapping between coordinates by the coordinate mapping means 10, and FIG. 6A is an input image plane R4 of the camera 2 that employs normal projection (h = ftanα) as an example. It is a figure which shows the correspondence relation between the above coordinate and the coordinate on the space model MD, and the coordinate associating means 10 is on the coordinate on the input image plane R4 of a camera 2 and the space model MD corresponding to the coordinate. Both coordinates are associated with each other so that each of the lines connecting the coordinates of the camera 2 passes through the optical center C of the camera 2.

In the example of FIG. 6A, the coordinate mapping means 10 associates the coordinates K1 on the input image plane R4 of the camera 2 with the coordinates L1 on the plane region R1 of the spatial model MD, and the input image plane R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the spatial model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

When the camera 2 employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the coordinate mapping means 10 uses each projection method. Correspondingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the spatial model MD.

Specifically, the coordinate mapping means 10 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equal-angle projection (h = 2fsin (α / 2)). 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the spatial model MD. In this case, the line segments K1-L1 and the line segments K2-L2 do not pass through the optical center C of the camera 2.

FIG. 6B is a diagram showing the correspondence between the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3, and the coordinate mapping means 10 is located on the XZ plane. A parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the spatial model MD and the processing target image corresponding to the coordinates are introduced. Both coordinates are associated with each other so that the coordinates on the plane R3 both lie on one of the parallel line groups PL.

In the example of FIG. 6B, the coordinate mapping means 10 sets both coordinates on the assumption that the coordinates L2 on the curved surface region R2 of the spatial model MD and the coordinates M2 on the processing target image plane R3 lie on a common parallel line. Make it correspond.

The coordinate mapping means 10 can map the coordinates on the plane region R1 of the spatial model MD to the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface region R2. However, in the example of FIG. 6B, since the plane region R1 and the processing target image plane R3 are common planes, the coordinates L1 on the plane region R1 of the spatial model MD and the processing target image plane R3 are on. It has the same coordinate value as the coordinate M1 of.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. It is stored in the spatial model / processing target image correspondence map 41.

FIG. 6C is a diagram showing the correspondence between the coordinates on the image plane R3 to be processed and the coordinates on the output image plane R5 of the virtual camera 2V that employs normal projection (h = ftanα) as an example. In the output image generation means 11, each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates passes through the optical center CV of the virtual camera 2V. To associate both coordinates.

In the example of FIG. 6C, the output image generation means 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane region R1 of the spatial model MD). , The coordinates N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

When the virtual camera 2V employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the output image generation means 11 uses each projection method. Correspondingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the output image generation means 11 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equidistant angle projection (h = 2fsin (α /)). 2)), equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segments M1-N1 and the line segments M2-N2 do not pass through the optical center CV of the virtual camera 2V.

In this way, the output image generation means 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. In the output image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored in the processing target image / output image correspondence map 42 in association with each other. An output image is generated by associating the value of each pixel with the value of each pixel in the input image.

Note that FIG. 6D is a diagram in which FIGS. 6A to 6C are combined, and the camera 2, the virtual camera 2V, the plane area R1 and the curved surface area R2 of the space model MD, and the processing target. The mutual positional relationship of the image plane R3 is shown.

Next, with reference to FIG. 7, the operation of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described.

FIG. 7A is a diagram when an angle β is formed between the parallel line group PL located on the XZ plane and the image plane R3 to be processed, and FIG. 7B is a diagram on the XZ plane. It is a figure in the case where the angle β1 (β1> β) is formed between the position parallel line group PL and the processing target image plane R3. Further, the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 7 (A) and 7 (B) correspond to the coordinates Ma to Md on the processing target image plane R3, respectively. It is assumed that the respective intervals of the coordinates La to Ld in FIG. 7 (A) are equal to the respective intervals of the coordinates La to Ld in FIG. 7 (B). Although the parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, it actually exists so as to extend radially from all points on the Z axis toward the image plane R3 to be processed. It shall be. The Z axis in this case is referred to as a "reprojection axis".

As shown in FIGS. 7A and 7B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the angle between the parallel line group PL and the processing target image plane R3. It decreases linearly as it increases (it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md). On the other hand, in the example of FIG. 7, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the processing target image plane R3, so that the distance between the coordinate groups does not change. ..

As for the change in the spacing of these coordinate groups, only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD among the image portions on the output image plane R5 (see FIG. 6) is linearly enlarged or expanded. It means that it will be reduced.

Next, with reference to FIG. 8, an alternative example of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described.

FIG. 8A is a diagram in which all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the image plane R3 to be processed, and FIG. 8B is an auxiliary diagram. It is a figure in the case where all of the line group AL extends from the start point T2 (T2> T1) on the Z axis toward the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 8 (A) and 8 (B) corresponds to each of the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8 (A), the coordinates Mc and Md are not shown because they are outside the region of the image plane R3 to be processed.), And the intervals between the coordinates La to Ld in FIG. 8 (A) are shown in FIG. It is assumed that it is equal to the respective intervals of the coordinates La to Ld in 8 (B). The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the image plane R3 to be processed. It shall be. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (high) between the start point and the origin O of the auxiliary line group AL. (S) decreases non-linearly as it increases (the greater the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md, the greater the decrease in each interval). On the other hand, in the example of FIG. 8, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the processing target image plane R3, so that the distance between the coordinate groups does not change. ..

The change in the spacing of these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6) as in the case of the parallel line group PL. It means that only the image portion is non-linearly enlarged or reduced.

In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the spatial model MD, and the spatial model MD. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface region R2 can be enlarged or reduced linearly or non-linearly, the road surface image in the vicinity of the excavator 60 can be enlarged or reduced. Quickly and flexibly make objects located around the excavator 60 (objects in the image when the excavator 60 is viewed horizontally) without affecting (virtual image when the excavator 60 is viewed from directly above). It can be enlarged or reduced to improve the visibility of the blind spot area of the excavator 60.

Next, referring to FIG. 9, a process in which the image generation device 100 generates a processing target image (hereinafter, referred to as “processing target image generation processing”), and an output image using the generated processing target image. The process of generating (hereinafter, referred to as "output image generation process") will be described. Note that FIG. 9 is a flowchart showing the flow of the processing target image generation processing (steps S1 to S3) and the output image generation processing (steps S4 to S6). Further, it is assumed that the arrangement of the camera 2 (input image plane R4), the spatial model (plane area R1 and curved surface area R2), and the image plane R3 to be processed is determined in advance.

First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the spatial model MD by the coordinate associating means 10 (step S1).

Specifically, the coordinate mapping means 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. One calculates a point that intersects the curved surface area R2 of the space model MD, and the coordinates on the curved surface area R2 corresponding to the calculated points are one on the curved surface area R2 corresponding to the one coordinate on the processing target image plane R3. It is derived as coordinates, and the correspondence relationship is stored in the spatial model / processed target image correspondence map 41. The angle formed between the parallel line group PL and the image plane R3 to be processed may be a value stored in advance in the storage unit 4 or the like, and the operator dynamically moves through the input unit 3. It may be a value to be entered.

Further, when one coordinate on the processing target image plane R3 matches one coordinate on the plane region R1 of the space model MD, the coordinate associating means 10 converts the one coordinate on the plane region R1 into the processing target image. It is derived as one coordinate corresponding to that one coordinate on the plane R3, and the correspondence relationship is stored in the spatial model / processing target image correspondence map 41.

After that, the control unit 1 associates one coordinate on the spatial model MD derived by the above process with the coordinate on the input image plane R4 by the coordinate associating means 10 (step S2).

Specifically, the coordinate mapping means 10 acquires the coordinates of the optical center C of the camera 2 that normally employs projection (h = ftanα), and is a line segment extending from one coordinate on the spatial model MD, which is the optical center. The point where the line segment passing through C intersects with the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated points are set on the input image plane R4 corresponding to the one coordinate on the spatial model MD. It is derived as one coordinate, and the correspondence is stored in the input image / spatial model correspondence map 40.

After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the spatial model MD and the coordinates on the input image plane R4 (step S3), and still all the coordinates. If it is determined that the above is not associated (NO in step S3), the processes of steps S1 and S2 are repeated.

On the other hand, when it is determined that all the coordinates are associated with each other (YES in step S3), the control unit 1 starts the output image generation process after ending the process target image generation process, and the output image generation means 11 As a result, the coordinates on the image plane R3 to be processed and the coordinates on the output image plane R5 are associated with each other (step S4).

Specifically, the output image generation means 11 generates an output image by performing scale conversion, affine transformation, or distortion conversion on the image to be processed, and is determined by the contents of the scale conversion, affine transformation, or distortion conversion performed. The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

Alternatively, when the output image generation means 11 generates an output image using the virtual camera 2V, the output image generation means 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the projection method adopted. , The correspondence may be stored in the processing target image / output image correspondence map 42.

Alternatively, when the output image generation means 11 generates an output image using a virtual camera 2V that employs normal projection (h = ftanα), the output image generation means 11 acquires the coordinates of the optical center CV of the virtual camera 2V and then obtains the coordinates. A line segment extending from one coordinate on the output image plane R5, the point at which the line segment passing through the optical center CV intersects the processing target image plane R3 is calculated, and the processing target image plane R3 corresponding to the calculated point is calculated. The coordinates may be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence may be stored in the processing target image / output image correspondence map 42.

After that, the control unit 1 uses the output image generation means 11 to refer to the input image / spatial model correspondence map 40, the spatial model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42, and the input image plane. The correspondence between the coordinates on R4 and the coordinates on the spatial model MD, the correspondence between the coordinates on the spatial model MD and the coordinates on the processing target image plane R3, and the correspondence between the coordinates on the processing target image plane R3 and the output image plane R5. By tracing the correspondence with the above coordinates, the values (for example, brightness value, hue value, saturation value, etc.) possessed by the coordinates on the input image plane R4 corresponding to each coordinate on the output image plane R5 are acquired. Then, the acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the output image generation means 11 uses the plurality of coordinates on the plurality of input image planes R4, respectively. Even if a statistical value based on the value of (for example, an average value, a maximum value, a minimum value, an intermediate value, etc.) is derived and the statistical value is adopted as the value of the one coordinate on the output image plane R5. Good.

After that, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6), and still corresponds to all the coordinate values. If it is determined that the attachment is not attached (NO in step S6), the processes of steps S4 and S5 are repeated.

On the other hand, when it is determined that the values of all the coordinates are associated with each other (YES in step S6), the control unit 1 generates an output image and ends this series of processing.

When the image generation device 100 does not generate the processing target image, the processing target image generation processing is omitted, and the "coordinates on the processing target image plane" in step S4 in the output image generation processing are "coordinates on the processing target image plane" on the spatial model. It shall be read as "coordinates".

With the above configuration, the image generation device 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the construction machine and the surrounding obstacles.

Further, the image generation device 100 performs coordinate mapping from the processing target image plane R3 to the input image plane R4 via the spatial model MD, thereby inputting each coordinate on the processing target image plane R3 into the input image plane. Compared with the case where one or a plurality of coordinates on R4 can be reliably associated with each other and the coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD (in this case). Can make sure that each coordinate on the input image plane R4 corresponds to one or more coordinates on the processing target image plane R3, but a part of the coordinates on the processing target image plane R3 is the input image plane R4. It may not be associated with any of the above coordinates. In that case, it is necessary to perform interpolation processing or the like on a part of the coordinates on the processing target image plane R3) to obtain a higher quality processing target image. Can be generated quickly.

Further, the image generation device 100 changes the angle formed between the parallel line group PL and the image plane R3 to be processed when only the image corresponding to the curved surface region R2 of the spatial model MD is enlarged or reduced. By rewriting only the portion related to the curved surface region R2 in the spatial model / processing target image correspondence map 41, the desired enlargement or reduction can be realized without rewriting the contents of the input image / spatial model correspondence map 40. ..

Further, when changing the appearance of the output image, the image generation device 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters related to scale conversion, affine conversion, or distortion conversion. It is possible to generate a desired output image (scale conversion image, affine conversion image or distortion conversion image) without rewriting the contents of the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41.

Similarly, when the viewpoint of the output image is changed, the image generation device 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42, and the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the spatial model / processing target image correspondence map 41.

FIG. 10 is a display example when the output image generated by using the input images of the two cameras 2 (right side camera 2R and rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. ..

The image generation device 100 generates an image to be processed by projecting the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD and then reprojecting them onto the processing target image plane R3. Then, an output image is generated by performing image conversion processing (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated image to be processed, and the vicinity of the excavator 60 is viewed from the sky. An image looking down (an image in the plane region R1) and an image looking at the periphery in the horizontal direction from the excavator 60 (an image in the image plane R3 to be processed) are displayed at the same time.

When the image generation device 100 does not generate the image to be processed, the output image is generated by performing an image conversion process (for example, a viewpoint conversion process) on the image projected on the spatial model MD. It shall be.

Further, the output image is trimmed into a circle so that the image when the excavator 60 performs the turning operation can be displayed without discomfort, and the center CTR of the circle is on the cylindrical central axis of the spatial model MD and the excavator 60 is swiveled. It is generated so as to be on the axis PV, and is displayed so as to rotate about the center CTR according to the turning operation of the excavator 60. In this case, the central axis of the cylinder of the spatial model MD may or may not coincide with the reprojection axis.

The radius of the spatial model MD is, for example, 5 meters, and the angle formed by the parallel line group PL with the image plane R3 to be processed or the starting point height of the auxiliary line group AL is the rotation of the excavator 60. When an object (for example, a worker) is located at a position separated from the center by the maximum reach of the excavation attachment E (for example, 12 meters), the object is sufficiently large (for example, 7) on the display unit 5. It is greater than or equal to a millimeter.) Can be set to be displayed.

Further, as the output image, the CG image of the excavator 60 may be arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. This is to make it easier to understand the positional relationship between the excavator 60 and the object appearing in the output image. As the output image, a frame image including various information such as the orientation may be arranged around the frame image.

Next, with reference to FIGS. 11 to 16, a process for preventing the difference in brightness (luminance) between the input images by the image generation device 100 will be described.

FIG. 11 shows the input images of the three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) mounted on the excavator 60, and the output images generated by using the input images. It is a figure which shows.

The image generation device 100 generates an image to be processed by projecting the input images of each of the three cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD and then reprojecting them onto the processing target image plane R3. Then, an output image is generated by performing image conversion processing (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated image to be processed, and the vicinity of the excavator 60 is viewed from the sky. An image looking down (an image in the plane region R1) and an image looking down from the shovel 60 in the horizontal direction (an image in the image plane R3 to be processed) are displayed at the same time.

In FIG. 11, the input image of the left side camera 2L and the input image of the rear camera 2B each include overlapping parts obtained by capturing the same place around the excavator 60 from different angles, and the overlapping parts are illuminations related to each camera. Images are taken with different brightness due to differences in the environment and the like, and the brightness is different. The same applies to the overlapping portion between the input image of the right side camera 2R and the input image of the rear camera 2B.

Therefore, in the output image generated based on these three input images, when the coordinates on the output image plane can be associated with the coordinates on a plurality of input image planes, the coordinates on the output image plane are incident. Assuming that it is associated with the coordinates on the input image plane with respect to the camera with the smallest angle, the area on the output image based on the input image of the left camera 2L and the rear camera 2B are captured even though the same location is captured. The boundary between the area on the output image based on the input image of (see the area R6 surrounded by the one-point chain line in the figure), and the area on the output image based on the input image of the right camera 2R and the rear camera 2B. The brightness changes abruptly at the boundary between the area on the output image based on the input image (see the area R7 surrounded by the one-point chain line in the figure), which is unnatural for the driver who sees the output image. It will make you feel that.

Therefore, the image generation device 100 utilizes the effect of the checker shadow illusion (same color illusion) to make the difference in brightness between the input images inconspicuous.

FIG. 12 is a diagram for explaining the same color illusion (Edward H. Adelson, "Checker shadow illusion", 1995, Internet <URL: http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html. >), FIG. 12 (A) shows a checker pattern (lattice pattern) used to bring about the same color illusion, and FIG. 12 (B) shows points P1 to P3 on the checker pattern (lattice pattern). A graph showing the transition of brightness at points P4 to P6 is shown.

As shown in FIG. 12 (A), it is sandwiched between two unit patterns (a unit pattern including the point P1 and a unit pattern including the point P3) having a brightness higher than the brightness of the unit pattern including the point P2. The brightness of the unit pattern including the point P2 is sandwiched between two unit patterns (a unit pattern including the point P4 and a unit pattern including the point P6) having a brightness lower than the brightness of the unit pattern including the point P5. It appears to be lower than the brightness of the unit pattern including that point P5.

However, this difference in brightness is an observation that recognizes that each unit pattern forms a lattice pattern and that the unit pattern group including points P4 to P6 is in the shadow of the object OBJ. This is due to the optical illusion of a person, and in reality, as shown in FIG. 12B, the brightness of the unit pattern including the point P2 is equal to the brightness of the unit pattern including the point P5.

In this way, the observer has the same brightness in the unit pattern including each of the points P1, P3, and P5, and the same brightness in the unit pattern including each of the points P2, P4, and P6. You will have the illusion of having something.

FIG. 13 is a diagram for explaining an example of a method of generating a light-dark pattern that causes the same color illusion, and is a diagram of two wave sources (for example, a right-side camera 2R and a rear-side camera 2B. The right-side camera 2R is a diagram. It is shown on the left side of the figure, but this is for clarification of the figure, and here, the diagonally upper right side of the figure corresponds to the rear of the excavator 60) to the peak (solid circle) and the valley. It shows two waves that spread and interfere with each other while forming (dashed circle).

Further, in FIG. 13, the two fan-shaped regions 2Ra and 2Ba indicate the imaging range of the right side camera 2R and the imaging range of the rear camera 2B, respectively, and the abdominal line ANL represented by the thick solid line has two waves. The node line NL represented by a thick broken line indicates a line connecting points that strengthen each other, and indicates a line connecting points where two waves weaken each other. The abdominal line ANL and the node line NL will appear alternately as shown in FIG.

Further, in FIG. 13, a line drawn by one valley of the wave spreading from the right-side camera 2R as a wave source (broken line circle), a line drawn by one mountain generated next to the one valley (solid line circle), and the rear Each of the diamond-shaped areas defined by the line drawn by one valley of the wave spreading from the camera 2B (broken line circle) and the line drawn by one mountain generated next to that one valley (solid line circle) , The unit pattern region LT shall be formed.

It is assumed that a plurality of unit pattern regions LT shown in FIG. 13 are drawn on the output image plane, and the rear camera 2B passes through the unit pattern region LT1 through which the abdominal line ANL of the unit pattern regions LT passes (filled in gray). The input image of the right-side camera 2R is associated with the unit pattern area LT2 through which the node line NL of the unit pattern area LT passes (filled in white). Two input images having different brightness can form a lattice pattern that gives the same color illusion.

The input image of the right-side camera 2R is associated with the unit pattern area LT1 through which the abdominal line ANL of the unit pattern area LT passes (filled in gray), and the node line NL of the unit pattern area LT. Similarly, when the input image of the rear camera 2B is associated with the unit pattern region LT2 through which the (filled in white) passes, the two input images having different average brightness form a lattice pattern that brings about the same color illusion. Will be possible.

Further, although the lattice pattern of FIG. 13 is formed by using two waves having the same wavelength and phase, it may be formed by using two waves having one or both different wavelengths and phases. This is to enable the size and shape of the unit pattern regions LT1 and LT2 to be flexibly adjusted.

FIG. 14 shows a state in which the lattice pattern shown in FIG. 13 is arranged on the output image plane (plane region R1), and is a right-side camera located on the right rear side (lower right direction in the figure) of the CG image 60G of the excavator 60. The lattice pattern arranged in the overlapping area between the imaging range of 2R and the imaging range of the rear camera 2B, and the imaging range and rear of the left camera 2L located on the left rear side (lower left direction in the figure) of the CG image 60G of the excavator 60. A grid pattern arranged in an area overlapping the image pickup range of the camera 2B is shown.

In FIG. 14, the grid pattern arranged in the overlapping region between the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B is, for example, the input image of the right-side camera 2R in the unit pattern region LT1 (filled in gray). Is associated with, and the input image of the rear camera 2B is associated with the unit pattern area LT2 (filled in white).

Further, the grid pattern arranged in the overlapping region between the imaging range of the left camera 2L and the imaging range of the rear camera 2B corresponds to, for example, the input image of the left camera 2L corresponding to the unit pattern region LT3 (filled in gray). It is assumed that the input image of the rear camera 2B is associated with the unit pattern area LT4 (filled in white).

In FIG. 15, the image generator 100 corresponds to the coordinates on the input image plane of either of the two cameras with each coordinate on the output image plane corresponding to the overlapping region of the imaging range of each of the two cameras. It is a flowchart which shows the flow of the process of forming a lattice pattern which brings about the same color illusion (hereinafter, referred to as "the lattice pattern formation process").

For example, the control unit 1 of the image generation device 100 uses the coordinate mapping means 10 to perform one coordinate on the plane region R1 of the spatial model MD and on a plurality of input image planes in step S2 of the processing target image generation processing of FIG. When associating one coordinate on the plane region R1 of the spatial model MD with one coordinate in one of the two input image planes corresponding to each of the two cameras when the coordinates can be associated with each other. , It is assumed that this lattice pattern forming process is executed.

First, the control unit 1 acquires one coordinate on the plane region R1 of the spatial model MD corresponding to the region where the imaging ranges of the two cameras (for example, the right side camera 2R and the rear camera 2B) overlap. (Step S11).

Next, the control unit 1 acquires the coordinates of the optical center of each of the two cameras (step S12).

Next, the control unit 1 selects a camera associated with one coordinate on the plane region R1 in the spatial model MD acquired in step S11 (step S13).

Specifically, the control unit 1 sets the coordinates of the optical center of the right-side camera 2R as (X _cam1 , Y _cam1 ) and the coordinates of the optical center of the rear camera 2B as (X _cam2 , Y _cam2 ), and associates them. Assuming that one coordinate on the plane region R1 of the target spatial model MD is (X _target , Y _target ),

で表される条件式が真である場合、その平面領域Ｒ１上の一座標に対応付けるカメラとして右側方カメラ２Ｒを選択し、上述の条件式が偽である場合、その平面領域Ｒ１上の一座標に対応付けるカメラとして後方カメラ２Ｂを選択する。

If the conditional expression represented by is true, the right-side camera 2R is selected as the camera associated with one coordinate on the plane area R1, and if the above conditional expression is false, one coordinate on the plane area R1 is selected. Select the rear camera 2B as the camera associated with.

The control unit 1 selects the rear camera 2B as the camera associated with one coordinate on the plane region R1 when the above conditional expression is true, and when the above conditional expression is false, the plane. The right side camera 2R may be selected as the camera associated with one coordinate on the area R1.

The above-mentioned conditional expression is _{a determination formula for determining whether the coordinates (X target} , Y _target ) on the plane region R1 are included in the unit pattern region LT1 shown in FIG. 14 or included in the unit pattern region LT2. Corresponds to.

Further, in the present embodiment, the control unit 1 has one coordinate (two-dimensional coordinate) and two on the plane region R1 on the assumption that the coordinates of the optical center are two-dimensional coordinates and the wave generated from the wave source is a plane wave. The camera is selected based on the two-dimensional distance between the coordinates of the optical center of each camera (two-dimensional coordinates projected on the plane including the plane region R1), but the coordinates of the optical center (height information). (Including) Three-dimensional coordinates, and while the wave generated from the wave source is a spherical wave, between one coordinate (three-dimensional coordinates) on the plane region R1 and the coordinates (three-dimensional coordinates) of the optical centers of the two cameras. The camera may be selected based on the three-dimensional distance of.

Further, in the present embodiment, the control unit 1 is on the plane region R1 of the spatial model MD corresponding to the region where the imaging ranges of the two cameras (for example, the right side camera 2R and the rear camera 2B) overlap. Although the camera associated with one coordinate is selected, the camera associated with one coordinate on the image plane R3 to be processed, which corresponds to the area where the imaging ranges of the two cameras overlap, may be selected.

In this case, the control unit 1 has one coordinate (two-dimensional coordinates) on the processing target image plane R3 and the coordinates of the optical centers of the two cameras (two-dimensional coordinates projected on the plane including the processing target image plane R3). ) May be selected based on the two-dimensional distance between the two cameras, one coordinate (three-dimensional coordinates) on the image plane R3 to be processed and the coordinates of the optical centers of the two cameras (three-dimensional coordinates). ) May be selected based on the three-dimensional distance between them. The image plane R3 to be processed may include the plane region R1.

After that, the control unit 1 associates one coordinate on the input image plane of the selected camera with one coordinate on the plane region R1 of the spatial model MD by the coordinate mapping means 10 (step S14), and the spatial model MD The above coordinates, the camera identifier, and the coordinates on the input image plane are associated and stored in the input image / spatial model correspondence map 40.

After that, the control unit 1 sets all the coordinates on the plane region R1 of the spatial model MD corresponding to the region where the imaging ranges of the two cameras overlap to the coordinates on the input image plane of one of the two cameras. It is determined whether or not the coordinates are associated (step S15), and if it is determined that all the coordinates are not yet associated (NO in step S15), the processes of steps S11 to S14 are repeated.

On the other hand, when it is determined that all the coordinates are associated with each other (YES in step S15), the control unit 1 ends this lattice pattern forming process.

In the above description, the control unit 1 sets each coordinate on the plane region R1 of the spatial model MD or each coordinate on the processing target image plane R3 corresponding to the region where the imaging ranges of the two cameras overlap. The coordinates on the input image plane of one of the two cameras are associated with each other, and each coordinate on the curved surface region R2 of the spatial model MD corresponding to the region where the imaging ranges of the two cameras overlap is set to two. It may be associated with the coordinates on the input image plane of one of the cameras.

In this way, the control unit 1 can easily associate the coordinates on the spatial model MD with the coordinates on the input image plane of one of the two cameras by using the conditional expression as described above, which is appropriate. A checkerboard pattern can be generated.

FIG. 16 is a comparison diagram showing the difference between the output image shown in FIG. 11 and the output image to which the grid pattern that brings about the same color illusion is applied to the output image of FIG. 11, and FIG. 16 (A) is a comparison diagram of FIG. The output image shown is shown, and FIG. 16B shows an output image to which a checkerboard pattern that results in the same color illusion is applied.

The area R6 surrounded by the one-point chain line in FIG. 16A including the boundary between the area on the output image based on the input image of the left camera 2L and the area on the output image based on the input image of the rear camera 2B is Although a remarkable difference in brightness is presented, the area on the output image based on the input image of the left camera 2L and the area on the output image based on the input image of the rear camera 2B coexist in a grid pattern. The difference in brightness of the region R8 surrounded by the one-point chain line in FIG. 16B is inconspicuous, and it is difficult for the driver who sees the output image including the region R8 to feel unnatural.

Similarly, it is surrounded by a single point chain line in FIG. 16 (A) including a boundary between a region on the output image based on the input image of the right-side camera 2R and a region on the output image based on the input image of the rear camera 2B. The region R7 presents a remarkable difference in brightness, but the region on the output image based on the input image of the right-side camera 2R and the region on the output image based on the input image of the rear camera 2B form a grid pattern. The area R9 surrounded by the single-point chain line in FIG. 16B, which is mixed with each other, has an inconspicuous difference in brightness, which makes it difficult for the driver who sees the output image including the area R9 to feel unnatural. ing.

Next, referring to FIGS. 17 to 19, when the image generation device 100 generates an output image portion corresponding to the overlapping region of the imaging range of each of the two cameras, the object in the output image portion disappears. The process of preventing this from happening will be described.

FIG. 17 shows the input images of the three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) mounted on the excavator 60, and the output images generated by using the input images. It is a figure which shows.

The image generation device 100 projects the coordinates on the input image planes of the three cameras 2 onto the coordinates on the plane area R1 and the curved surface area R2 of the spatial model MD, and then reprojects them onto the image plane R3 to be processed. The output image is generated by generating an image to be processed and performing image conversion processing (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image, and excavator. An image looking down on the vicinity of 60 from the sky (an image in the plane region R1) and an image looking at the periphery in the horizontal direction from the shovel 60 (an image in the image plane R3 to be processed) are displayed.

In FIG. 17, the input image of the right-side camera 2R and the input image of the rear camera 2B are each within the overlapping region of the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B (of the right-side camera 2R). The area R10 surrounded by the two-point chain line in the input image and the area R11 surrounded by the two-point chain line in the input image of the rear camera 2B are captured.

However, the input image of the right-side camera 2R and the output image generated based on the input image of the rear camera 2B have coordinates on the input image plane with respect to the camera having the smallest incident angle on the output image plane. As shown in FIG. 17, the person in the overlapping area disappears (see the area R12 surrounded by the one-point chain line in the output image).

Therefore, the image generation device 100 utilizes the checkerboard pattern formed to bring about the same color illusion to generate the output image portion corresponding to the overlapping region of the imaging range of each of the two cameras. It shall prevent the disappearance of the object in.

FIG. 18 is a diagram for explaining the effect of preventing the disappearance of an object in the overlapping region of the imaging range of each of the two cameras, and FIG. 18 (A) is a checkerboard pattern for bringing about the same color illusion shown in FIG. FIG. 18 (B) shows a partially enlarged view of FIG. 18 (A).

In FIG. 18A, the projected image PRJ1 surrounded by a broken line is such that the image of the object OBJ1 in the input image of the rear camera 2B changes the rear camera 2B and the object OBJ1 by viewpoint conversion for generating a road surface image. It represents an image extended in the extension direction of the connecting line (an image displayed when a road surface image in the output image portion of the input image of the rear camera 2B is generated).

Further, the projected image PRJ2 surrounded by a broken line is a line in which the image of the object OBJ1 in the input image of the right-side camera 2R connects the right-side camera 2R and the object OBJ1 by viewpoint conversion for generating a road surface image. It represents an image extended in the extension direction (an image displayed when a road surface image in the output image portion is generated using the input image of the right-side camera 2R).

The projected image PRJ1 and the projected image PRJ2 are not displayed as they are on the final output image, but are displayed in a partially cutout state as shown in FIG. 18B. Will be done.

Further, in FIG. 18A, of the broken line region representing the projected image PRJ1, the portion filled with gray represents the portion overlapping with the unit pattern region LT1 (see FIG. 13) through which the abdominal line ANL passes. It is shown that the coordinates on the input image plane of the rear camera 2B (coordinates in the region forming the image of the object OBJ1) are associated with each coordinate on the output image corresponding to the overlapping portion.

On the other hand, in the broken line region representing the projected image PRJ1, the portion filled with white represents the overlapping portion with the unit pattern region LT2 (see FIG. 13) through which the node line NL passes, and the output corresponding to the overlapping portion. It is shown that each coordinate on the image is associated with the coordinates on the input image plane of the right camera 2R (coordinates in the region that does not form the image of the object OBJ1).

In the broken line area representing the projected image PRJ1, the coordinates on the input image plane of the right camera 2R (in the area where the image of the object OBJ1 is not formed) are set to the coordinates on the output image corresponding to the part filled with gray. (Coordinates) are associated with each coordinate on the output image corresponding to the part filled in white in the broken line region representing the projected image PRJ1, and the coordinates on the input image plane of the rear camera 2B (the image of the object OBJ1). (Coordinates in the region forming the above) may be associated with each other.

In this case, the projected image PRJ1 is one of the circles orthogonal to the extension direction (the circles drawn by the peaks and valleys of the waves whose wave source is the position of the rear camera 2B) as shown in FIG. 18B. It is cut out by the unit pattern region LT2 that includes the portion as its boundary line, but since the adjacent unit pattern regions LT1 are in contact with each other at the apex, it is difficult to be divided into fine slices, and each of the unit pattern regions LT2 is Since it includes a part of the circle orthogonal to the extension direction, its outline is also stored in a clear state that is easily recognized.

Further, the projected image PRJ1 tends to be elongated in the direction away from the camera as the height of the object OBJ1 is higher, and is enlarged more as the distance from the camera is increased. However, as the distance from the camera increases, the projected image is magnified to the same degree as the projected image PRJ1, so that the cutout state is maintained almost uniformly.

Further, in FIG. 18A, of the broken line region representing the projected image PRJ2, the portion filled with black represents the overlapping portion with the unit pattern region LT2 (see FIG. 13) through which the node line NL passes. It is shown that the coordinates on the input image plane of the right-side camera 2R (coordinates in the region forming the image of the object OBJ1) are associated with each coordinate on the output image corresponding to the overlapping portion.

On the other hand, in the broken line region representing the projected image PRJ2, the portion filled with white represents the overlapping portion with the unit pattern region LT1 (see FIG. 13) through which the abdominal line ANL passes, and the output corresponding to the overlapping portion. It is shown that each coordinate on the image is associated with the coordinates on the input image plane of the rear camera 2B (coordinates in the region that does not form the image of the object OBJ1).

In the broken line region representing the projected image PRJ2, the coordinates on the input image plane of the rear camera 2B (coordinates in the region that does not form the image of the object OBJ1) are added to the coordinates on the output image corresponding to the portion filled in black. ) Is associated with each coordinate on the output image corresponding to the part filled in white in the broken line region representing the projected image PRJ2, and the coordinates on the input image plane of the right camera 2R (an image of the object OBJ1 is formed). Coordinates in the area to be used) may be associated with each other.

In this case, the projected image PRJ2, like the projected image PRJ1, has a circle (a wave source having the position of the right-side camera 2R as a wave source) perpendicular to the extending direction as shown in FIG. 18 (B). It is cut out by the unit pattern region LT1 that includes a part of the drawn circle as its boundary line, but it is difficult to be divided into fine slices because the adjacent unit pattern regions LT2 are in contact with each other at the vertices. Since each of the unit pattern regions LT1 includes a part of a circle orthogonal to the extending direction thereof, the contour thereof is also stored in a clear state so that it can be easily recognized.

Further, like the projected image PRJ1, the projected image PRJ2 tends to be elongated in the direction away from the camera as the height of the object OBJ1 is higher, and is enlarged larger as the distance from the camera is increased. As the pattern regions LT1 and LT2 are both farther away from the camera, they are magnified more to the same extent as the projected image PRJ2, so that the cutout state is maintained substantially uniformly.

FIG. 19 is a comparison diagram showing the difference between the output image shown in FIG. 17 and the output image to which the grid pattern that brings about the same color illusion is applied to the output image of FIG. The output image shown is shown, and FIG. 19B shows an output image to which a checkerboard pattern that results in the same color illusion is applied.

The area R13 surrounded by the one-point chain line in FIG. 19A including the boundary between the area on the output image based on the input image of the right-side camera 2R and the area on the output image based on the input image of the rear camera 2B is , The state where the person disappears is presented, but the area on the output image based on the input image of the right side camera 2R and the area on the output image based on the input image of the rear camera 2B are mixed in a grid pattern. The area R14 surrounded by the one-point chain line in FIG. 19B is presented in a state in which the person can be easily recognized without disappearing, and the driver who sees the output image including the area R14 is presented with the person. We are making sure that we can recognize the existence.

With the above configuration, the image generation device 100 uses a checkerboard pattern to bring about the same color illusion to generate an output image portion corresponding to an overlapping region of the imaging range of each of the two cameras. It is possible to prevent the object in the part from disappearing, and a part of it compared to the case where the two input images are joined so that the respective partial areas of the two input images are alternately arranged in a comb-tooth shape. The projected image of the object to which is cut out can be displayed in a state that is easier for the driver to recognize.

Although the preferred examples of the present invention have been described in detail above, the present invention is not limited to the above-mentioned examples, and various modifications and substitutions are made to the above-mentioned examples without departing from the scope of the present invention. Can be added.

For example, in the above embodiment, the image generation device 100 adopts a cylindrical space model MD as a space model, but a space model having another columnar shape such as a polygonal prism may be adopted, and the bottom surface and the space model may be adopted. A space model composed of two side surfaces may be adopted, or a space model having only side surfaces may be adopted.

Further, the image generation device 100 is mounted on a self-propelled construction machine equipped with movable members such as a bucket, an arm, a boom, and a swivel mechanism together with a camera, and moves and moves the construction machine while presenting a surrounding image to the driver. It is incorporated in an operation support system that supports the operation of these movable members, but it is mounted together with a camera on other objects to be operated that have movable members but do not run on their own, such as industrial machines or fixed cranes. It may be incorporated into an operation support system that supports the operation of another operated object.

1 ... Control unit 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input unit 4 ... Storage unit 5 ... Display unit 10 ...・ Coordinate mapping means 11 ・ ・ ・ Output image generation means 40 ・ ・ ・ Input image / spatial model compatible map 41 ・ ・ ・ Spatial model / processed target image compatible map 42 ・ ・ ・ Processing target image / output image compatible map 60 ・・ ・ Excavator 61 ・ ・ ・ Lower traveling body 62 ・ ・ ・ Swivel mechanism 63 ・ ・ ・ Upper swivel body 64 ・ ・ ・ Cab

Claims (12)

The lower traveling body that performs the running operation and
An upper swivel body that is freely mounted on the lower traveling body and
The boom attached to the upper swing body and included in the attachment,
With the arm attached to the boom and included in the attachment,
An imaging device mounted at three locations on the left, right, and rear of the upper swivel body so as to image the three directions of the upper swivel body.
The driver's cab mounted on the upper swing body and
The display unit installed in the driver's cab and
The left of the upper swivel body whose display overlaps at least the left, right, and rear regions of the upper swivel body with the imaging range of the imaging device, regardless of the orientation of the lower traveling body. An output image portion is generated for the overlapping region by synthesizing the captured images of the adjacent imaging devices so that the overlapping region of the rear and the right rear is projected in a state where the excavator is viewed from directly above. The image obtained by the imaging device is displayed as an output image in a state including the output image portion regardless of the orientation of the lower traveling body, and the imaging mounted at the three locations in the display area centered on the output image. It has a control unit that displays the output image without simultaneously outputting the input image from the device.
Excavator. The image captured by the image pickup device is captured so as to include a part of the vehicle body.
The excavator according to claim 1. The output image displays various information around the image captured by the imaging device.
The excavator according to claim 1 or 2. The control unit generates an image of the excavator looking down from the sky, excluding at least a portion that projects a region in front of the excavator.
The excavator according to any one of claims 1 to 3. The control unit generates an image including a scene when looking backward from the excavator based on an image captured by an image pickup device mounted on the rear portion of the upper swivel body.
The excavator according to any one of claims 1 to 4. The image captured by the image pickup apparatus mounted on the rear portion of the upper swivel body includes a part of the image of the upper swivel body.
A part of the image of the upper swivel body is displayed in a concave shape on the display unit.
The excavator according to any one of claims 1 to 5. The input image contains an empty part ,
The excavator according to any one of claims 1 to 6. A lower traveling body that performs a traveling operation, an upper rotating body that is rotatably mounted on the lower traveling body, a boom that is attached to the upper rotating body and included in the attachment, and a boom that is attached to the boom and included in the attachment. An imaging device mounted on the left, right, and rear parts of the upper swing body so as to image three directions of the upper swing body, and a driver's cab mounted on the upper swing body. And a display unit installed in the driver's cab, and a peripheral image generation device for the excavator mounted on the excavator.
The left of the upper swivel body whose display overlaps at least the left, right, and rear regions of the upper swivel body with the imaging range of the imaging device, regardless of the orientation of the lower traveling body. An output image portion is generated for the overlapping region by synthesizing the captured images of the adjacent imaging devices so that the overlapping region of the rear and the right rear is projected in a state where the excavator is viewed from directly above. The image obtained by the imaging device is displayed as an output image in a state including the output image portion regardless of the orientation of the lower traveling body, and the output image is mounted on the three locations in the display area centered on the output image. It has a control unit that displays the output image without simultaneously outputting the input image from the image pickup device.
Excavator surrounding image generator. The image captured by the image pickup device is captured so as to include a part of the vehicle body.
The excavator surrounding image generator according to claim 8. In the output image, the front of the CG image of the excavator is displayed so as to coincide with the upper part of the screen of the display unit.
The excavator surrounding image generator according to claim 8 or 9. The image captured by the image pickup apparatus mounted on the rear portion of the upper swivel body includes a part of the image of the upper swivel body.
A part of the image of the upper swivel body is displayed in a concave shape on the display unit.
The excavator surrounding image generator according to any one of claims 8 to 10. The output image displays various information around the image captured by the imaging device.
The excavator surrounding image generator according to any one of claims 8 to 11.

Images