JP2020002773A - Shovel and shovel peripheral image generation device - Google Patents

Abstract

To improve visibility of an output image generated based on a plurality of input images photographed by a camera.SOLUTION: A shovel 60 according to the example of the invention includes a lower travelling body 61 that performs a travelling operation, an upper rotating body 63 that is turnably mounted on the lower travelling body 61, a boom mounted on the upper rotating body 63, an arm mounted on the boom, cameras 2 mounted on three portions of a left part, a right part and a rear part of the upper rotating body 63 so as to image three directions of the upper rotating body 63, a driver cab 64 mounted on the upper rotating body 63, a display part 5 installed in the driver cab 64, and a control part 1 which projects at least left, right and rear regions of the shovel irrespective of the direction of the lower travelling body 61 onto the display part 5, generates an image in which the shovel is looked down from the above as an output image, and indicates surrounded regions R6 and R7 to the output image.SELECTED DRAWING: Figure 11

Description

  The present invention relates to a shovel that generates an output image based on a plurality of input images captured by a camera mounted on an upper swing body of the shovel, and a surrounding image generation device of the shovel.

  2. Description of the Related Art Conventionally, there is known a driving support system that converts images captured by a plurality of in-vehicle cameras that capture the periphery of a vehicle into bird's-eye view images, and presents an output image obtained by joining the bird's-eye view images to a driver (for example, a driving support system). And Patent Document 1.).

JP 2007-109166 A

  However, the driving support system described in Patent Literature 1 does not assume that the driving support system is mounted on a construction machine that performs a turning operation or excavation, and cannot improve visibility when provided in a cab of a shovel.

  In view of the above, it is desirable to improve the visibility of an output image generated based on a plurality of input images captured by a camera.

  A shovel according to an embodiment of the present invention is a lower traveling body that performs a traveling operation, an upper rotating body that is swingably mounted on the lower traveling body, a boom that is attached to the upper rotating body and included in an attachment, Attached to the boom, the arm included in the attachment, the left, right, and rear of the upper revolving unit so as to image three directions of the upper revolving unit, an imaging device, A driver's cab mounted on the upper revolving unit, a display unit installed in the driver's cab, and the display unit includes at least the left, right, and rear of the shovel regardless of the orientation of the lower traveling unit. A control unit that projects an area, generates an image of the shovel looking down from the sky as an output image, and indicates the area surrounded by the output image.

  With the above-described means, the visibility of an output image generated based on a plurality of input images captured by the camera is improved.

FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation device according to the present invention. FIG. 2 is a diagram illustrating a configuration example of a shovel on which the image generation device is mounted. FIG. 3 is a diagram illustrating an example of a space model onto which an input image is projected. FIG. 6 is a diagram illustrating an example of a relationship between a space model and a processing target image plane. FIG. 4 is a diagram for explaining correspondence between coordinates on an input image plane and coordinates on a space model. FIG. 3 is a diagram for explaining association between coordinates by a coordinate associating unit. It is a figure for explaining an operation of a parallel line group. It is a figure for explaining an operation of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a display example (the 2) of an output image. FIG. 3 is a diagram for explaining the same color illusion. FIG. 7 is a diagram for explaining an example of a method of generating a light-dark pattern that produces the same color illusion. FIG. 14 is a diagram showing a state where the lattice pattern of FIG. 13 is arranged on an output image plane. It is a flowchart which shows the flow of a lattice pattern formation process. FIG. 12 is a comparison diagram illustrating a difference between the output image illustrated in FIG. 11 and an output image to which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 11. It is a display example (the 3) of an output image. FIG. 9 is a diagram for describing an effect of preventing an object from disappearing in an overlapping area of an imaging range of each of two cameras. FIG. 18 is a comparison diagram illustrating a difference between the output image illustrated in FIG. 17 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 17.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation device 100 according to the present invention.

  The image generation device 100 is a device that generates an output image based on an input image captured by a camera 2 mounted on a construction machine and presents the output image to a driver, for example, and includes a control unit 1, a camera 2, It comprises an input unit 3, a storage unit 4, and a display unit 5.

  FIG. 2 is a diagram illustrating a configuration example of a shovel 60 on which the image generation device 100 is mounted. The shovel 60 includes an upper revolving unit 63 on a crawler-type lower traveling unit 61 via a revolving mechanism 62. It is mounted so as to be pivotable about a pivot axis PV.

  The upper revolving superstructure 63 has a cab (operator cab) 64 at a front left portion thereof, a digging attachment E at a front central portion thereof, and a camera 2 (a right camera 2R and a rear camera 2B) at right and rear surfaces thereof. ). It is assumed that the display unit 5 is provided at a position in the cab 64 where the driver can easily view the display.

  Next, each component of the image generation device 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and a NVRAM (Non-Volatile Random Access Memory). The program corresponding to each of the attaching means 10 and the output image generating means 11 is stored in a ROM or NVRAM, and the CPU executes processing corresponding to each means while using the RAM as a temporary storage area.

  The camera 2 is a device for acquiring an input image that reflects the periphery of the shovel 60, and is attached to, for example, the right side surface and the rear surface of the upper revolving unit 63 so as to capture an image of a driver's blind spot in the cab 64. (See FIG. 2), a right camera 2R and a rear camera 2B provided with an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). Note that the camera 2 may be attached to a position other than the right side and the rear side (for example, the front side and the left side) of the upper revolving unit 63, and a wide-angle lens or a fish-eye lens is attached so as to capture a wide range. May be.

  Further, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. Note that, when the camera 2 obtains an input image using a fisheye lens or a wide-angle lens, the camera 2 sends a corrected input image in which apparent distortion or tilt caused by using those lenses has been corrected to the control unit 1. Although the output is performed, the input image whose apparent distortion and tilt have not been corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows the operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, and the like.

  The storage unit 4 is a device for storing various information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display or a projector installed in a cab 64 (see FIG. 2) of a construction machine. Display an image.

  Further, the image generating apparatus 100 generates an image to be processed based on the input image, and performs an image conversion process on the image to be processed so as to intuitively grasp a positional relationship with a peripheral obstacle and a sense of distance. After the output image to be generated is generated, the output image may be presented to the driver.

  The “processing target image” is an image that is generated based on an input image and is an image to be subjected to image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like). When an input image including a horizontal image (for example, a sky portion) is used in the image conversion process due to the wide angle of view of the input image by the camera that captures the image from above, the horizontal image is The input image is projected onto a predetermined space model so that the image is not displayed unnaturally (for example, the sky portion is not treated as being on the ground), and then the projected image projected onto the space model is This is an image suitable for image conversion processing, which is obtained by reprojecting onto a dimensional plane. The processing target image may be used as it is as an output image without performing image conversion processing.

  The “space model” is a plane or a curved surface other than the processing target image plane which is the plane where the processing target image is located (for example, a plane parallel to the processing target image plane, or an angle between the processing target image plane and the processing target image plane). The projection target of the input image includes one or more planes or curved surfaces including a plane or a curved surface to be formed.

  Note that the image generation device 100 may generate an output image by performing an image conversion process on a projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image without performing image conversion processing.

  FIG. 3 is a diagram illustrating an example of a space model MD onto which an input image is projected. FIG. 3A illustrates a relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from the side. FIG. 3B shows a relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and has a plane region R1 inside its bottom surface and a curved region R2 inside its side surface.

  FIG. 4 is a diagram illustrating an example of the relationship between the space model MD and the processing target image plane. The processing target image plane R3 is, for example, a plane including the plane region R1 of the space model MD. FIG. 4 shows the space model MD in a cylindrical shape instead of the semi-cylindrical shape as shown in FIG. 3 for clarity. May be used. The same applies to the following drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the space model MD, or may be an annular area not including the plane area R1 of the space model MD.

  Next, various means of the control unit 1 will be described.

  The coordinate associating means 10 is means for associating the coordinates on the input image plane where the input image captured by the camera 2 is located, the coordinates on the space model MD, and the coordinates on the processing target image plane R3, For example, various parameters related to the camera 2 such as an optical center, a focal length, a CCD size, an optical axis direction vector, a camera horizontal direction vector, and a projection method which are set in advance or input through the input unit 3. And the coordinates on the input image plane, the coordinates on the spatial model MD, and the processing target image based on the predetermined mutual positional relationship between the input image plane, the space model MD, and the processing target image plane R3. The coordinates on the plane R3 are associated with each other, and their correspondence is stored in the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41 of the storage unit 4. .

  When the processing target image is not generated, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on the processing target image plane R3, and determines the correspondence between the spatial model and the processing target. The storage in the image correspondence map 41 is omitted.

  The output image generation unit 11 is a unit for generating an output image. For example, by performing scale conversion, affine transformation, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image are converted. The input image / spatial model in which the coordinates are associated with the coordinates on the output image plane in which they are located, the correspondence is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the values are stored in the coordinate association means 10 Referring to the correspondence map 40 and the space model / processing target image correspondence map 41, the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image To generate an output image.

  Further, the output image generating means 11 is provided with an optical center, a focal length, a CCD size, an optical axis direction vector, a camera horizontal direction vector, a camera horizontal direction, and a projection method, which are set in advance or input through the input unit 3. The coordinates on the processing target image plane R3 and the coordinates on the output image plane where the output image is located are associated with each other based on various parameters such as the processing target image / output image correspondence map 42 in the storage unit 4. While referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 in which the coordinate associating means 10 stores the values (e.g., luminance values). , Hue value, saturation value, etc.) and the value of each pixel in the input image, to generate an output image.

  Note that the output image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  When the processing target image is not generated, the output image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane according to the applied image conversion processing, and outputs the input image / space model. While referring to the correspondence map 40, an output image is generated by associating the value of each pixel in the output image (for example, a luminance value, a hue value, a saturation value, etc.) with the value of each pixel in the input image. In this case, the output image generation unit 11 omits the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane, and the storage of the correspondence in the processing target image / output image correspondence map 42. I do.

  Next, an example of a specific process performed by the coordinate association unit 10 and the output image generation unit 11 will be described.

  The coordinate associating means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, Hamiltonian quaternions.

  FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 has UVW orthogonal coordinates with the optical center C of the camera 2 as the origin. The spatial model is represented as one plane in the system, and the spatial model is represented as a three-dimensional plane in the XYZ rectangular coordinate system.

  First, the coordinate associating means 10 optically sets the origin of the XYZ coordinate system to convert coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system). After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" indicates that the directions are opposite). This means that the XYZ coordinate system is rotated so as to match the UVW coordinate system with the + W direction in front of the camera and the XYZ coordinate system with the -Z direction vertically below.

  When a plurality of cameras 2 exist, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate associating unit 10 executes the XYZ coordinate system in parallel with each of the plurality of UVW coordinate systems. Will be moved and rotated.

  In the above-described conversion, the XYZ coordinate system is moved in parallel so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the -W axis. Since this is realized by rotating the axis so that it coincides with the axis, the coordinate associating means 10 can combine these two rotations into one rotation operation by describing this transformation as a Hamiltonian quaternion. it can.

  By the way, the rotation for making a certain vector A coincide with another vector B corresponds to a process of rotating the vector A and the vector B by an angle formed by the vector A and the vector B about an axis of a normal to a plane formed by the vector A and the vector B. If the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It will be represented by

  Further, a unit vector N of a normal line of a plane formed by the vector A and the vector B is obtained from an outer product of the vector A and the vector B.

It will be represented by

  Note that the quaternion is obtained when i, j, and k are respectively imaginary units.

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

And the conjugate quaternion of the quaternion Q is

It is assumed that

  The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, and c while setting the real component t to 0 (zero). , C, can be used to represent a rotating motion about an arbitrary vector.

  Further, the quaternion Q can be expressed as one rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. The point D (ex, ey, ez) when rotated by the angle θ with C (l, m, n) as the axis can be expressed as follows.

Here, in the present embodiment, assuming that a quaternion representing rotation for matching the Z axis with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X 'is

It will be represented by

  Further, in the present embodiment, if a quaternion representing a rotation that causes a line connecting the point X ′ on the X axis and the origin to coincide with the U axis is Qx, “the Z axis coincides with the −W axis, and , The quaternion R representing "the rotation that makes the X axis coincide with the U axis"

It will be represented by

  As described above, the coordinate P ′ when the arbitrary coordinate P on the space model (XYZ coordinate system) is represented by the coordinate on the input image plane (UVW coordinate system) is

Since the quaternion R is invariable in each of the cameras 2, the coordinate associating means 10 thereafter calculates the coordinates on the space model (XYZ coordinate system) only by executing this operation. It can be converted to coordinates on the input image plane (UVW coordinate system).

  After converting the coordinates on the space model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate associating means 10 determines the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model with a coordinate P ′ expressed in a UVW coordinate system and the optical axis G of the camera 2 is calculated.

  In addition, the coordinate associating means 10 determines the intersection point E between the plane H and the optical axis G and the coordinate P 'on a plane H parallel to the input image plane R4 (for example, a CCD surface) of the camera 2 and including the coordinates P'. Is calculated from the line segment EP ′ connecting the line and the U ′ axis on the plane H, and the length of the line segment EP ′.

  In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f, so that the coordinate associating means 10 performs normal projection (h = ftanα) and orthographic projection (h = fsinα). , Stereoscopic projection (h = 2ftan (α / 2)), equi-solid angle projection (h = 2fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

  Thereafter, the coordinate associating means 10 decomposes the calculated image height h into a U component and a V component on the UV coordinate system using the argument φ, and obtains a numerical value corresponding to a pixel size per pixel of the input image plane R4. By the division, the coordinates P (P ′) on the space model MD can be associated with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

It will be represented by

  In this way, the coordinate associating means 10 associates the coordinates on the space model MD with the coordinates on one or more input image planes R4 existing for each camera, And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Also, since the coordinate associating means 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angles, there is an advantage that no gimbal lock is generated. . However, the coordinate associating unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may calculate the coordinate conversion using the Euler angles.

  If it is possible to associate with coordinates on a plurality of input image planes R4, the coordinate associating means 10 converts the coordinates P (P ') on the space model MD into an input for the camera having the smallest incident angle α. The coordinates may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

  Next, among the coordinates on the space model MD, the process of reprojecting the coordinates on the curved surface region R2 (coordinates having components in the Z-axis direction) onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between the coordinates by the coordinate associating means 10. FIG. 6A shows an example of the input image plane R4 of the camera 2 adopting the normal projection (h = ftanα) as an example. FIG. 9 is a diagram showing a correspondence relationship between the upper coordinates and the coordinates on the space model MD, wherein the coordinate associating means 10 is configured to display the coordinates on the input image plane R4 of the camera 2 and the coordinates on the space model MD corresponding to the coordinates. The two coordinates are associated with each other so that each line segment connecting the coordinates of the two passes through the optical center C of the camera 2.

  In the example of FIG. 6A, the coordinate associating unit 10 associates the coordinates K1 on the input image plane R4 of the camera 2 with the coordinates L1 on the plane area R1 of the space model MD, and outputs the coordinates of the input image plane R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  When the camera 2 adopts a projection method other than the normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate associating means 10 uses the respective projection schemes. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate associating means 10 performs a predetermined function (for example, orthographic projection (h = fsinα), stereoscopic projection (h = 2ftan (α / 2)), and equi-solid angle projection (h = 2fsin (α / 2)), the coordinates on the input image plane and the coordinates on the space model MD are associated based on the equidistant projection (h = fα). In this case, the line segments K1-L1 and K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 6B is a diagram showing the correspondence between the coordinates on the curved surface area R2 of the space model MD and the coordinates on the processing target image plane R3. The coordinate associating means 10 is located on the XZ plane. A parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the space model MD and the processing target image corresponding to the coordinates are introduced. The coordinates on the plane R3 are made to correspond to one of the parallel line groups PL so that the two coordinates are associated with each other.

  In the example of FIG. 6B, the coordinate associating unit 10 determines that the coordinates L2 on the curved surface area R2 of the space model MD and the coordinates M2 on the processing target image plane R3 ride on a common parallel line, and Make it correspond.

  Note that the coordinate associating means 10 can associate the coordinates on the plane region R1 of the space model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface region R2. However, in the example of FIG. 6B, since the plane area R1 and the processing target image plane R3 are a common plane, the coordinates L1 on the plane area R1 of the space model MD and the processing target image plane R3 Has the same coordinate value as the coordinate M1.

  In this way, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the space model MD with the coordinates on the processing target image plane R3. And stores it in the space model / processing target image correspondence map 41.

  FIG. 6C is a diagram showing the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting the normal projection (h = ftanα) as an example. The output image generation unit 11 determines that each line segment connecting the coordinates of the virtual camera 2V on the output image plane R5 and the coordinates corresponding to the coordinates on the processing target image plane R3 passes through the optical center CV of the virtual camera 2V. So that the two coordinates are associated with each other.

  In the example of FIG. 6C, the output image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (the plane region R1 of the space model MD). The coordinates N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  When the virtual camera 2V employs a projection method other than the normal projection (for example, orthographic projection, stereoscopic projection, equi-solid angle projection, equidistant projection, etc.), the output image generation means 11 outputs the respective projection scheme. , The coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the output image generation unit 11 outputs a predetermined function (for example, orthographic projection (h = fsinα), stereoscopic projection (h = 2ftan (α / 2)), and equi-solid angle projection (h = 2fsin (α / 2)), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated based on the equidistant projection (h = fα). In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the output image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and sets the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. Is stored in the processing target image / output image correspondence map 42 in association with the input image / spatial model correspondence map 40 and the space model / processing target image correspondence map 41 stored by the coordinate associating means 10. An output image is generated by associating the value of each pixel with the value of each pixel in the input image.

  FIG. 6D is a view obtained by combining FIGS. 6A to 6C, and shows the camera 2, the virtual camera 2V, the plane region R1 and the curved surface region R2 of the space model MD, and the processing target. The mutual positional relationship of the image plane R3 is shown.

  Next, the operation of the parallel line group PL introduced by the coordinate associating means 10 for associating the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 7A is a diagram in the case where an angle β is formed between the parallel line group PL located on the XZ plane and the processing target image plane R3, and FIG. FIG. 9 is a diagram illustrating a case where an angle β1 (β1> β) is formed between a parallel line group PL and a processing target image plane R3. 7A and 7B, the coordinates La to Ld on the curved surface region R2 of the space model MD correspond to the coordinates Ma to Md on the processing target image plane R3, respectively. Each interval between the coordinates La to Ld in FIG. 7A is assumed to be equal to each interval between the coordinates La to Ld in FIG. 7B. Although the parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, it actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. It shall be. Note that the Z axis in this case is referred to as a “reprojection axis”.

  As shown in FIGS. 7A and 7B, the distance between the coordinates Ma to Md on the processing target image plane R3 is determined by the angle between the parallel line group PL and the processing target image plane R3. It decreases linearly as it increases (it uniformly decreases irrespective of the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md). On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, so that the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is that only the image portion corresponding to the image projected on the curved surface region R2 of the space model MD among the image portions on the output image plane R5 (see FIG. 6) is linearly enlarged or enlarged. Means shrinking.

  Next, with reference to FIG. 8, an alternative example of the parallel line group PL introduced by the coordinate associating means 10 for associating the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described.

  FIG. 8A is a diagram when all of the auxiliary line groups AL located on the XZ plane extend from the starting point T1 on the Z axis toward the processing target image plane R3, and FIG. FIG. 11 is a diagram illustrating a case where all the line groups AL extend from a start point T2 (T2> T1) on the Z axis toward a processing target image plane R3. 8A and 8B, the coordinates La to Ld on the curved surface region R2 of the space model MD correspond to the coordinates Ma to Md on the processing target image plane R3, respectively. In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are outside the area of the processing target image plane R3.), And the intervals between the coordinates La to Ld in FIG. It is assumed that the interval is equal to each of the coordinates La to Ld in FIG. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation. However, actually, the auxiliary line group AL extends radially from an arbitrary point on the Z axis toward the processing target image plane R3. It shall be. Note that, as in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is determined by the distance between the starting point of the auxiliary line group AL and the origin O (high ) Increases non-linearly (the larger the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md, the greater the decrease in the distance between them). On the other hand, in the example of FIG. 8, the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3, so that the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the space model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. This means that only the image portion is non-linearly enlarged or reduced.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD without affecting the space model MD. Image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface region R2 can be linearly or non-linearly enlarged or reduced, so that a road surface image near the shovel 60 can be obtained. (Virtual image when the shovel 60 is viewed from directly above) without affecting the objects located around the shovel 60 (objects in the image when the surroundings are viewed horizontally from the shovel 60) quickly and flexibly. The excavator 60 can be enlarged or reduced, and the visibility of the blind spot area of the shovel 60 can be improved.

  Next, referring to FIG. 9, a process in which the image generation device 100 generates a processing target image (hereinafter, referred to as “processing target image generation process”), and an output image is generated using the generated processing target image. The process of generating (hereinafter, referred to as “output image generating process”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation processing (steps S1 to S3) and the output image generation processing (steps S4 to S6). It is assumed that the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved area R2), and the processing target image plane R3 are determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD by the coordinate association unit 10 (step S1).

  Specifically, the coordinate associating unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3, and obtains the angle of the parallel line group PL extending from one coordinate on the processing target image plane R3. A point at which one intersects the curved surface region R2 of the space model MD is calculated, and coordinates on the curved surface region R2 corresponding to the calculated points are calculated on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived, and the correspondence is stored in the space model / processing target image correspondence map 41. Note that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be an input value.

  When one coordinate on the processing target image plane R3 coincides with one coordinate on the plane area R1 of the space model MD, the coordinate associating unit 10 compares the one coordinate on the plane area R1 with the processing target image plane. The coordinates are derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing target image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate associating means 10 to associate one coordinate on the space model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate associating means 10 acquires the coordinates of the optical center C of the camera 2 employing the normal projection (h = ftanα), and is a line segment extending from one coordinate on the space model MD. A point at which a line segment passing through C intersects the input image plane R4 is calculated, and coordinates on the input image plane R4 corresponding to the calculated point are calculated on the input image plane R4 corresponding to the one coordinate on the space model MD. And the correspondence is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 have been associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3), and all the coordinates have been determined. Is determined not to correspond (NO in step S3), the processing of step S1 and step S2 is repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 terminates the processing target image generation processing and starts the output image generation processing. Thus, the coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

  Specifically, the output image generation unit 11 generates an output image by performing scale conversion, affine transformation, or distortion conversion on the processing target image, and is determined by the content of the performed scale conversion, affine transformation, or distortion conversion. The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating an output image using the virtual camera 2V, the output image generating means 11 calculates coordinates on the output image plane R5 from coordinates on the processing target image plane R3 according to the adopted projection method. The correspondence may be stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating an output image using the virtual camera 2V that employs normal projection (h = ftanα), the output image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V, A point at which a line segment extending from one coordinate on the output image plane R5 and passing through the optical center CV intersects with the processing target image plane R3 is calculated, and a point on the processing target image plane R3 corresponding to the calculated point is calculated. The coordinates may be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence may be stored in the processing target image / output image correspondence map 42.

  After that, the control unit 1 causes the output image generation means 11 to refer to the input image / spatial model correspondence map 40, the spatial model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42 while referring to the input image plane. Correspondence between coordinates on R4 and coordinates on space model MD, correspondence between coordinates on space model MD and coordinates on processing target image plane R3, and coordinates on processing target image plane R3 and output image plane R5 Following the correspondence with the above coordinates, the values (for example, luminance value, hue value, saturation value, etc.) of the coordinates on the input image plane R4 corresponding to each coordinate on the output image plane R5 are obtained. Then, the obtained value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the output image generating means 11 outputs the plurality of coordinates on the plurality of input image planes R4. (For example, an average value, a maximum value, a minimum value, an intermediate value, and the like) are derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. Good.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 have been associated with the coordinate values on the input image plane R4 (step S6), and determines whether all the coordinate values have been associated. If it is determined that no attachment is made (NO in step S6), the processes in steps S4 and S5 are repeated.

  On the other hand, when it is determined that all the coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends this series of processing.

  If the image to be processed is not to be generated, the image generating apparatus 100 omits the processing to generate an image to be processed, and replaces the “coordinates on the plane of the image to be processed” in step S4 in the output image generation processing with the “ "Coordinates".

  With the above configuration, the image generation device 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the construction machine and the surrounding obstacle.

  Further, the image generating apparatus 100 executes the correspondence of coordinates from the processing target image plane R3 back to the input image plane R4 via the space model MD, so that each coordinate on the processing target image plane R3 is converted to the input image plane R3. One or a plurality of coordinates on R4 can be surely corresponded. Compared with the case where the correspondence of coordinates is executed in the order from the input image plane R4 to the processing target image plane R3 via the space model MD (in this case, Can ensure that each coordinate on the input image plane R4 corresponds to one or more coordinates on the processing target image plane R3, but some of the coordinates on the processing target image plane R3 In some cases, the coordinates may not be associated with any of the above coordinates, in which case it is necessary to perform interpolation processing or the like on some of the coordinates on the processing target image plane R3.) It is possible to rapidly generate.

  In addition, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generation device 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. By simply rewriting only the portion related to the curved surface region R2 in the space model / processing target image correspondence map 41, desired enlargement or reduction can be realized without rewriting the contents of the input image / space model correspondence map 40. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 only needs to change the values of various parameters related to scale conversion, affine conversion, or distortion conversion and rewrite the processing target image / output image correspondence map 42. Thus, a desired output image (scale converted image, affine transformed image, or distorted transformed image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of the various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing target image correspondence map 41.

  FIG. 10 is a display example when an output image generated using the input images of the two cameras 2 (the right camera 2R and the rear camera 2B) mounted on the shovel 60 is displayed on the display unit 5. .

  The image generation apparatus 100 generates the processing target image by projecting the respective input images of the two cameras 2 on the plane region R1 and the curved surface region R2 of the space model MD, and re-projecting the input image on the processing target image plane R3. Then, an output image is generated by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image, and the vicinity of the shovel 60 is viewed from above. The image looking down (the image in the plane region R1) and the image of the periphery viewed horizontally from the shovel 60 (the image in the image plane R3 to be processed) are simultaneously displayed.

  When the image generation device 100 does not generate the processing target image, the output image is generated by performing an image conversion process (for example, a viewpoint conversion process) on the image projected on the space model MD. Shall be.

  In addition, the output image is trimmed into a circle so that the image when the shovel 60 performs the turning operation can be displayed without a sense of incongruity. It is generated so as to be on the axis PV, and is displayed so as to rotate around the center CTR thereof in accordance with the turning operation of the shovel 60. In this case, the center axis of the cylinder of the space model MD may or may not match the reprojection axis.

  Note that the radius of the space model MD is, for example, 5 meters, and the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is determined by turning the shovel 60. When an object (for example, a worker) is present at a position separated from the center by the maximum reach distance of the excavation attachment E (for example, 12 meters), the object is sufficiently large on the display unit 5 (for example, 7). Millimeters or more.) It can be set to be displayed.

  Further, the output image may be such that the CG image of the shovel 60 is arranged such that the front of the shovel 60 coincides with the upper part of the screen of the display unit 5 and the center of rotation coincides with the center CTR. This is to make it easier to understand the positional relationship between the shovel 60 and the object appearing in the output image. The output image may include a frame image including various types of information such as the azimuth around the output image.

  Next, with reference to FIGS. 11 to 16, a description will be given of a process in which the image generating device 100 does not make the difference in brightness (luminance) between input images stand out.

  FIG. 11 illustrates input images of three cameras 2 (left camera 2L, right camera 2R, and rear camera 2B) mounted on the shovel 60, and output images generated using the input images. FIG.

  The image generation device 100 generates the processing target image by projecting the input images of the three cameras 2 on the plane region R1 and the curved surface region R2 of the space model MD and re-projecting the input image on the processing target image plane R3. Then, an output image is generated by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image, and the vicinity of the shovel 60 is viewed from above. The image looking down (the image in the plane region R1) and the image of the periphery viewed horizontally from the shovel 60 (the image in the image plane R3 to be processed) are simultaneously displayed.

  In FIG. 11, the input image of the left camera 2L and the input image of the rear camera 2B each include overlapping portions of the same location around the shovel 60 taken from different angles, and the overlapping portions are respectively illuminations of the cameras. Images are taken at different brightnesses due to differences in environment and the like, and the brightness is different. The same applies to the overlapping portion between the input image of the right camera 2R and the input image of the rear camera 2B.

  Therefore, if an output image generated based on these three input images can be associated with coordinates on a plurality of input image planes, the coordinates on the output image plane will be incident. Assuming that the camera is associated with the coordinates of the camera having the smallest angle on the input image plane, the area on the output image based on the input image of the left camera 2L and the rear camera 2B despite the fact that the same place is imaged. Between the region on the output image based on the input image of the right camera 2R (see the region R6 surrounded by the dashed line in the figure) and the region on the output image based on the input image of the right camera 2R and the rear camera 2B. At the boundary between the input image and the region on the output image (see the region R7 surrounded by the dashed line in the figure), the luminance of the driver suddenly changes. So that the would feel unnatural.

  Therefore, the image generating apparatus 100 uses the effect of the checker shadow illusion (same color illusion) to make the difference in luminance between input images less noticeable.

  FIG. 12 is a diagram for explaining the same color illusion (Edward H. Adelson, "Checker shadow illusion", 1995, Internet <URL: http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html) >), FIG. 12A shows a checker pattern (lattice pattern) used to provide the same color illusion, and FIG. 12B shows points P1 to P3 on the checker pattern (lattice pattern). 5 is a graph showing a change in luminance at points P4 to P6.

  As shown in FIG. 12A, two unit patterns having a luminance higher than the luminance of the unit pattern including the point P2 (the unit pattern including the point P1 and the unit pattern including the point P3) are sandwiched. The luminance of the unit pattern including the point P2 is sandwiched between two unit patterns (a unit pattern including the point P4 and a unit pattern including the point P6) having a lower luminance than the luminance of the unit pattern including the point P5. It appears that the luminance is lower than the luminance of the unit pattern including the point P5.

  However, this difference in luminance is an observation that recognizes that each unit pattern forms a lattice pattern and that a unit pattern group including points P4 to P6 is in the shadow of the object OBJ. In fact, as shown in FIG. 12B, the luminance of the unit pattern including the point P2 is equal to the luminance of the unit pattern including the point P5.

  Thus, the observer states that the unit pattern including each of the points P1, P3, and P5 has the same luminance, and the unit pattern including each of the points P2, P4, and P6 has the same luminance. You will have the illusion of having something.

  FIG. 13 is a diagram for describing an example of a method of generating a light-dark pattern that causes the same color illusion, and includes two wave sources (for example, a right camera 2R and a rear camera 2B. The right camera 2R is a diagram). This is for the sake of clarity of the figure, in which the diagonally upper right of the figure corresponds to the rear of the shovel 60) to the peak (solid circle) and the valley. 2 shows two waves that form a (dashed circle) and spread and interfere with each other.

  In FIG. 13, two fan-shaped regions 2Ra and 2Ba indicate an imaging range of the right camera 2R and an imaging range of the rear camera 2B, respectively, and the antinode ANL represented by a thick solid line indicates that two waves A line connecting points that strengthen each other is shown, and a nodal line NL represented by a thick broken line indicates a line connecting points where two waves weaken each other. In addition, the antinode line ANL and the node line NL appear alternately as shown in FIG.

  In FIG. 13, a line drawn by one valley of a wave spreading using the right camera 2R as a wave source (dashed line circle), a line drawn by one peak generated after the one valley (solid line circle), and Each of the rhombic regions defined by a line drawn by one valley of a wave spreading with the camera 2B as a wave source (broken circle) and a line drawn by one crest generated after the one valley (solid circle) is , A unit pattern region LT is formed.

  The plurality of unit pattern areas LT shown in FIG. 13 are drawn on the output image plane, and the rear camera 2B is placed in the unit pattern area LT1 through which the abdominal line ANL of the unit pattern areas LT passes (filled in gray). If the input image of the right camera 2R is associated with the unit pattern area LT2 through which the node line NL passes (filled in white) of the unit pattern area LT, the average Two input images having different luminances can form a lattice pattern that produces the same color illusion.

  The input image of the right camera 2R is associated with the unit pattern area LT1 through which the antinode line ANL of the unit pattern area LT passes (filled in gray), and the node line NL of the unit pattern area LT Similarly, in a case where the input image of the rear camera 2B is associated with the unit pattern area LT2 through which the image passes (filled in white), two input images having different average luminances form a lattice pattern that causes the same color illusion. Can be done.

  Although the lattice pattern in FIG. 13 is formed using two waves having the same wavelength and phase, it may be formed using two waves having one or both different wavelengths and phases. This is because the sizes and shapes of the unit pattern regions LT1 and LT2 can be flexibly adjusted.

  FIG. 14 shows a state in which the lattice pattern shown in FIG. 13 is arranged on the output image plane (plane area R1), and is a right camera located at the right rear (lower right direction in the figure) of the CG image 60G of the shovel 60. A lattice pattern arranged in an overlapping area of the imaging range of the 2R and the imaging range of the rear camera 2B, and the imaging range and the rear of the left camera 2L on the rear left (lower left in the drawing) of the CG image 60G of the shovel 60. 5 shows a grid pattern arranged in an overlapping area with the imaging range of the camera 2B.

  In FIG. 14, the lattice pattern arranged in the overlapping area of the imaging range of the right camera 2R and the imaging range of the rear camera 2B is, for example, an input image of the right camera 2R in a unit pattern area LT1 (filled in gray). And the input image of the rear camera 2B is associated with the unit pattern area LT2 (filled in white).

  In addition, the lattice pattern arranged in the overlapping area of the imaging range of the left camera 2L and the imaging range of the rear camera 2B is, for example, a unit pattern area LT3 (filled in gray) corresponding to the input image of the left camera 2L. It is assumed that the input image of the rear camera 2B is associated with the unit pattern area LT4 (filled in white).

  FIG. 15 shows that the image generation apparatus 100 corresponds the coordinates on the input image plane of one of the two cameras to the coordinates on the output image plane corresponding to the overlapping area of the respective imaging ranges of the two cameras. 9 is a flowchart illustrating a flow of a process of forming a lattice pattern that causes the same color illusion (hereinafter, referred to as a “lattice pattern forming process”).

  For example, the control unit 1 of the image generating apparatus 100 uses the coordinate associating unit 10 to execute one coordinate on the plane region R1 of the space model MD and a plurality of input image planes in step S2 of the processing target image generating process in FIG. When it is possible to associate one coordinate on the plane region R1 of the space model MD with one coordinate on one of two input image planes corresponding to the two cameras, This lattice pattern forming process is executed.

  First, the control unit 1 acquires one coordinate on the plane region R1 of the space model MD corresponding to the region where the imaging ranges of the two cameras (for example, the right camera 2R and the rear camera 2B) overlap. (Step S11).

  Next, the control unit 1 acquires the coordinates of the optical center of each of the two cameras (step S12).

  Next, the control unit 1 selects a camera corresponding to one coordinate on the plane region R1 in the space model MD acquired in step S11 (step S13).

Specifically, the control unit 1 sets the coordinates of the optical center of the right camera 2R to (X _cam1 , Y _cam1 ) and the coordinates of the optical center of the rear camera 2B to (X _cam2 , Y _cam2 ). Assuming that one coordinate on the plane region R1 of the target space model MD is (X _target , Y _target ),

Is true, the right camera 2R is selected as the camera to be associated with one coordinate on the plane area R1, and if the above conditional expression is false, one coordinate on the plane area R1 is selected. The rear camera 2B is selected as the camera to be associated with.

  Note that the control unit 1 selects the rear camera 2B as the camera to be associated with one coordinate on the plane area R1 when the above conditional expression is true, and when the above conditional expression is false, The right camera 2R may be selected as the camera associated with one coordinate on the region R1.

Note that the above conditional expression is a determination expression for determining whether the coordinates (X _target , Y _target ) on the plane region R1 are included in the unit pattern region LT1 shown in FIG. 14 or are included in the unit pattern region LT2. Is equivalent to

  Further, in the present embodiment, the control unit 1 assumes that the coordinates of the optical center are two-dimensional coordinates, and that the wave generated from the wave source is a plane wave, The camera is selected based on the two-dimensional distance between the coordinates of the optical centers of the two cameras (two-dimensional coordinates projected on a plane including the plane region R1), but the coordinates of the optical centers are set to (height information Between the one coordinate (three-dimensional coordinates) on the plane region R1 and the coordinates (three-dimensional coordinates) of the respective optical centers of the two cameras while the wave generated from the wave source is a spherical wave. The camera may be selected based on the three-dimensional distance.

  Further, in the present embodiment, the control unit 1 controls the two cameras (for example, the right camera 2R and the rear camera 2B) on the plane region R1 of the space model MD corresponding to the region where the imaging ranges overlap. Although the camera corresponding to one coordinate is selected, a camera corresponding to one coordinate on the processing target image plane R3 corresponding to an area where the imaging ranges of the two cameras overlap may be selected.

  In this case, the control unit 1 calculates one coordinate (two-dimensional coordinates) on the processing target image plane R3 and coordinates of the optical centers of the two cameras (two-dimensional coordinates projected on a plane including the processing target image plane R3). ) May be selected based on the two-dimensional distance between the two cameras, one coordinate (three-dimensional coordinates) on the processing target image plane R3 and the coordinates (three-dimensional coordinates) of the respective optical centers of the two cameras. ) May be selected based on the three-dimensional distance between the cameras. The processing target image plane R3 may include the plane area R1.

  Thereafter, the control unit 1 associates one coordinate on the input image plane of the selected camera with one coordinate on the plane region R1 of the space model MD by the coordinate associating unit 10 (step S14). The above coordinates, the camera identifier, and the coordinates on the input image plane are stored in the input image / space model correspondence map 40 in association with each other.

  Thereafter, the control unit 1 converts all the coordinates on the plane region R1 of the space model MD, which correspond to the region where the imaging ranges of the two cameras overlap, to the coordinates on the input image plane of one of the two cameras. It is determined whether or not the coordinates have been associated (step S15). If it is determined that all the coordinates have not been associated yet (NO in step S15), the processing of steps S11 to S14 is repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S15), the control unit 1 ends the lattice pattern forming process.

  In the above description, the control unit 1 calculates each coordinate on the plane area R1 of the space model MD or each coordinate on the processing target image plane R3 corresponding to the area where the imaging ranges of the two cameras overlap. One of the cameras is associated with the coordinates on the input image plane, and each of the coordinates on the curved surface region R2 of the space model MD, which corresponds to the region where the imaging ranges of the two cameras overlap, is One of the cameras may be associated with the coordinates on the input image plane.

  As described above, the control unit 1 can easily associate each coordinate on the space model MD with the coordinate on one input image plane of the two cameras by using the above-described conditional expression. A grid pattern can be generated.

  FIG. 16 is a comparison diagram showing a difference between the output image shown in FIG. 11 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 11, and FIG. FIG. 16B shows an output image to which a lattice pattern that causes the same color illusion is applied.

  A region R6 surrounded by a dashed line in FIG. 16A including a boundary between a region on the output image based on the input image of the left camera 2L and a region on the output image based on the input image of the rear camera 2B is , The area on the output image based on the input image of the left camera 2L and the area on the output image based on the input image of the rear camera 2B are mixed in a lattice pattern. In a region R8 surrounded by a dashed line in FIG. 16B, the difference in luminance is not conspicuous, and it is difficult for a driver who looks at an output image including the region R8 to feel unnatural.

  Similarly, the region surrounded by the dashed line in FIG. 16A including the boundary between the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B. Although the region R7 presents a remarkable difference in luminance, the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B have a lattice pattern. In the region R9 surrounded by the one-dot chain line in FIG. 16B, the difference in luminance is not conspicuous, making it difficult for the driver who looks at the output image including the region R9 to feel unnatural. ing.

  Next, referring to FIGS. 17 to 19, when the image generating apparatus 100 generates an output image portion corresponding to the overlapping area of the respective imaging ranges of the two cameras, the object in the output image portion disappears. The following describes a process for preventing such a situation.

  FIG. 17 shows input images of three cameras 2 (left camera 2L, right camera 2R, and rear camera 2B) mounted on the shovel 60, and output images generated using the input images. FIG.

  The image generating apparatus 100 projects the coordinates of the three cameras 2 on the input image plane onto the coordinates on the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects the coordinates on the processing target image plane R3. An image to be processed is generated by performing an image conversion process (for example, scale conversion, affine transformation, distortion conversion, viewpoint conversion process, etc.) on the generated processing target image, and an excavator is generated. An image in which the vicinity of 60 is viewed from above (an image in the plane region R1) and an image in which the periphery is viewed in the horizontal direction from the shovel 60 (an image in the processing target image plane R3) are displayed.

  In FIG. 17, the input image of the right camera 2R and the input image of the rear camera 2B are respectively located within the overlapping area of the imaging range of the right camera 2R and the imaging range of the rear camera 2B. A region R10 surrounded by a two-dot chain line in the input image and a region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B are captured.

  However, the output image generated based on the input image of the right side camera 2R and the input image of the rear camera 2B has the coordinates on the output image plane whose coordinates on the input image plane regarding the camera having the smallest incident angle. As shown in FIG. 17, the person in the overlapping area is lost (see an area R12 surrounded by a dashed line in the output image).

  Therefore, the image generation apparatus 100 uses the lattice pattern formed to provide the same color illusion to generate an output image portion corresponding to the overlapping area of the respective imaging ranges of the two cameras. To prevent the object in the area from disappearing.

  FIG. 18 is a diagram for explaining the effect of preventing the disappearance of an object in the overlapping area of the respective imaging ranges of the two cameras. FIG. 18A shows a lattice pattern for producing the same-color illusion shown in FIG. FIG. 18 (B) is a partially enlarged view of FIG. 18 (A).

  In FIG. 18A, a projection image PRJ1 surrounded by a dashed line is obtained by transforming the image of the object OBJ1 in the input image of the rear camera 2B with the rear camera 2B and the object OBJ1 by changing the viewpoint for generating a road surface image. This is an image that is extended in the extension direction of the connecting line (an image displayed when a road surface image in the output image portion is generated using the input image of the rear camera 2B).

  Further, the projected image PRJ2 surrounded by the broken line is a line of the line connecting the right camera 2R and the object OBJ1 by the viewpoint transformation for generating the road surface image by the viewpoint conversion for generating the road surface image. This is an image that is extended in the extension direction (an image displayed when a road surface image in an output image portion is generated using an input image of the right camera 2R).

  Note that the projected image PRJ1 and the projected image PRJ2 are not displayed as they are on the final output image, but are displayed in a partially truncated state as shown in FIG. Will be done.

  In FIG. 18A, in the broken line region representing the projection image PRJ1, the gray portion is an overlapping portion with the unit pattern region LT1 (see FIG. 13) through which the antinode line ANL passes. This indicates that coordinates on the input image plane of the rear camera 2B (coordinates in the area where the image of the object OBJ1 is formed) are associated with the respective coordinates on the output image corresponding to the overlapping portion.

  On the other hand, in the broken line region representing the projected image PRJ1, the portion filled with white represents an overlapping portion with the unit pattern region LT2 (see FIG. 13) through which the node line NL passes, and an output corresponding to the overlapping portion. This indicates that the coordinates on the input image plane of the right camera 2R (coordinates in an area where the image of the object OBJ1 is not formed) are associated with each coordinate on the image.

  In the dashed-line area representing the projected image PRJ1, the coordinates on the input image plane of the right camera 2R (in the area where the image of the object OBJ1 is not formed) are added to the coordinates on the output image corresponding to the gray-filled portion. Of the dashed line area representing the projected image PRJ1, and the coordinates (the image of the object OBJ1) on the input image plane of the rear camera 2B are assigned to the respective coordinates on the output image corresponding to the portion painted white. May be associated with each other.

  In this case, as shown in FIG. 18B, the projected image PRJ1 is one of circles perpendicular to the direction of extension (a circle drawn by a wave peak and a valley with the position of the rear camera 2B as a wave source). Is cut out by the unit pattern area LT2 including the portion as a boundary line, but since the adjacent unit pattern areas LT1 are in contact with the vertices, it is difficult to be divided in a fine shape, and each of the unit pattern areas LT2 is Since a part of the circle orthogonal to the extension direction is included, the outline is also stored in a clear state that is easily recognized.

  Also, the projected image PRJ1 tends to be extended longer in the direction away from the camera as the height of the object OBJ1 is higher, and to be larger as the object OBJ1 is moved further away from the camera. However, the unit pattern areas LT1 and LT2 Also, as the distance from the camera increases, the projection image PRJ1 expands to a greater extent with the same degree, so that the notch state is also maintained substantially uniformly.

  Also, in FIG. 18A, in the broken line region representing the projected image PRJ2, a portion painted black is an overlapping portion with the unit pattern region LT2 (see FIG. 13) through which the node line NL passes, This indicates that the coordinates on the input image plane of the right camera 2R (the coordinates in the area where the image of the object OBJ1 is formed) are associated with the respective coordinates on the output image corresponding to the overlapping portion.

  On the other hand, in the broken line region representing the projected image PRJ2, a portion filled with white represents an overlapping portion with the unit pattern region LT1 (see FIG. 13) through which the antinode line ANL passes, and an output corresponding to the overlapping portion. This shows that the coordinates on the input image plane of the rear camera 2B (coordinates in an area where an image of the object OBJ1 is not formed) are associated with each coordinate on the image.

  In the dashed-line area representing the projected image PRJ2, the coordinates on the input image plane of the rear camera 2B (the coordinates in the area where the image of the object OBJ1 is not formed) are added to the respective coordinates on the output image corresponding to the portion painted black. ), And the coordinates (formation of the image of the object OBJ1) on the input image plane of the right camera 2R are formed at the coordinates on the output image corresponding to the portion filled with white in the broken line region representing the projection image PRJ2. (Coordinates in a region to be processed) may be associated with each other.

  In this case, similarly to the projected image PRJ1, the projected image PRJ2 has a circle perpendicular to its extending direction (wave peaks and valleys whose wave source is the position of the right camera 2R as shown in FIG. 18B). This is cut out by the unit pattern region LT1 including a part of the drawing circle as a boundary line, but since the adjacent unit pattern regions LT2 have their vertices in contact with each other, it is difficult to cut them into a fine pattern. Since each of the unit pattern areas LT1 includes a part of a circle orthogonal to the direction in which the unit pattern area LT1 extends, the outline thereof is also stored in a clear state that is easy to recognize.

  Further, similarly to the projection image PRJ1, the projection image PRJ2 has a tendency to be elongated in a direction away from the camera as the height of the object OBJ1 is increased, and to be enlarged as the distance from the camera increases. Since both the pattern areas LT1 and LT2 are further enlarged to the same degree as the projected image PRJ2 as the distance from the camera is increased, the cutout state is also maintained substantially uniform.

  FIG. 19 is a comparison diagram showing the difference between the output image shown in FIG. 17 and the output image of FIG. 17 to which a lattice pattern that gives the same color illusion is applied, and FIG. 19B shows the output image shown in FIG. 19B, and FIG. 19B shows an output image to which a lattice pattern that causes the same color illusion is applied.

  The region R13 surrounded by the dashed line in FIG. 19A including the boundary between the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B is Although the state in which the person has disappeared is presented, the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B are mixed in a lattice pattern. A region R14 surrounded by a dashed line in FIG. 19B presents the person in an easily recognizable state without losing the person, and provides the driver who has viewed the output image including the region R14 the person with the person. We make sure we can recognize their existence.

  With the above configuration, the image generating apparatus 100 uses the lattice pattern for providing the same color illusion to generate an output image portion corresponding to the overlapping area of the respective imaging ranges of the two cameras. It is possible to prevent the object in the part from disappearing, and compared with the case where the two input images are connected so that the respective partial areas of the two input images are alternately arranged in a comb shape, a part of the Can be displayed in a state in which the driver can easily recognize the projected image of the object that is to be cut off.

  Although the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the above-described embodiment, and various modifications and substitutions may be made to the above-described embodiment without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generating apparatus 100 adopts a cylindrical spatial model MD as a spatial model, but may adopt a spatial model having another columnar shape such as a polygonal column, A space model composed of two side surfaces may be employed, or a space model having only the side surfaces may be employed.

  The image generating apparatus 100 is mounted together with a camera on a self-propelled construction machine including movable members such as a bucket, an arm, a boom, and a turning mechanism, and moves and moves the construction machine while presenting a surrounding image to the driver. Although incorporated into an operation support system that supports the operation of these movable members, they are mounted together with the camera on other operated objects that have movable members but do not run by themselves, such as industrial machines or fixed cranes. It may be incorporated in an operation support system that supports the operation of another operated body.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left camera 2R ... Right camera 2B ... Rear camera 3 ... Input part 4 ... Storage part 5 ... Display part 10 ...・ Coordinate associating means 11 ・ ・ ・ Output image generating means 40 ・ ・ ・ Input image / spatial model corresponding map 41 ・ ・ ・ Spatial model / processing target image corresponding map 42 ・ ・ ・ Processing target image / output image corresponding map 60 ・..Excavator 61: Lower traveling body 62: Revolving mechanism 63: Upper revolving body 64: Cab

Claims (10)

A lower traveling body that performs a traveling operation,
An upper revolving structure rotatably mounted on the lower traveling structure,
A boom attached to the upper swing body and included in the attachment,
An arm attached to the boom and included in the attachment;
An imaging device mounted on three portions of a left portion, a right portion, and a rear portion of the upper revolving unit so as to image three directions of the upper revolving unit,
A cab mounted on the upper revolving superstructure,
A display unit installed in the cab,
On the display unit, at least the left, right, and rear areas of the shovel are projected regardless of the orientation of the lower traveling body, and an image in which the shovel is looked down from above is generated as an output image, and the output image And a control unit that indicates an enclosed area.
Excavator. The enclosed area is shown in a substantially circular shape,
The shovel according to claim 1. A lower traveling body that performs a traveling operation,
An upper revolving structure rotatably mounted on the lower traveling structure,
A boom attached to the upper swing body and included in the attachment,
An arm attached to the boom and included in the attachment;
An imaging device mounted on three portions of a left portion, a right portion, and a rear portion of the upper revolving unit so as to image three directions of the upper revolving unit,
A cab mounted on the upper revolving superstructure,
A display unit installed in the cab,
On the display unit, at least the left, right, and rear areas of the shovel are projected regardless of the orientation of the lower traveling body, and an image in which the shovel is looked down from above is generated as an output image, and in the output image, A control unit that displays an upper part of the object existing around the upper revolving superstructure at a position farther from the shovel than a position where the lower part of the object is displayed,
Excavator. Displaying various information around the output image,
The shovel according to claim 1. The output image displayed on the display unit includes an image of the sky or the horizon,
The shovel according to claim 1. A lower traveling body that performs a traveling operation, an upper revolving body that is rotatably mounted on the lower traveling body, a boom that is attached to the upper revolving body and that is included in an attachment, and that is attached to the boom and included in the attachment. Arm, imaging devices mounted at three locations on the left, right, and rear of the upper revolving unit so as to image three directions of the upper revolving unit, and a cab mounted on the upper revolving unit And a display unit installed in the driver's cab, mounted on a shovel comprising: a shovel surrounding image generation device,
On the display unit, at least the left, right, and rear areas of the shovel are projected regardless of the orientation of the lower traveling body, and an image in which the shovel is looked down from above is generated as an output image, and the output image Having a control unit indicating the enclosed area,
Excavator peripheral image generation device. The enclosed area is shown in a substantially circular shape,
The shovel surrounding image generation device according to claim 6. A lower traveling body that performs a traveling operation, an upper revolving body that is rotatably mounted on the lower traveling body, a boom that is attached to the upper revolving body and that is included in an attachment, and that is attached to the boom and included in the attachment. Arm, imaging devices mounted at three locations on the left, right, and rear of the upper revolving unit so as to image three directions of the upper revolving unit, and a cab mounted on the upper revolving unit And a display unit installed in the driver's cab, mounted on a shovel comprising: a shovel surrounding image generation device,
On the display unit, at least the left, right, and rear areas of the shovel are projected regardless of the orientation of the lower traveling body, and an image in which the shovel is looked down from above is generated as an output image, and in the output image, A control unit that displays an upper part of the object existing around the upper revolving superstructure at a position farther from the shovel than a position where the lower part of the object is displayed,
Excavator peripheral image generation device. Displaying various information around the output image,
A peripheral image generating apparatus for a shovel according to claim 6. The output image displayed on the display unit includes an image of the sky or the horizon,
The shovel surrounding image generation device according to claim 6.