JP2014236232A - Image generation device and operation supporting system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generation device in which a range subjected to loss prevention processing can be adjusted appropriately.SOLUTION: An image generation device 100 includes means 10 for making the coordinate in a spatial model MD correspond to the coordinate in an input image plane, and output image generation means 11 for generating an output image by making the value of a coordinate in an input image plane correspond to the value of a coordinate in an output image plane via a coordinate in the spatial model MD. The spatial model MD has a plane region R1 including a duplicate region capable of making the coordinates in the input image planes of two cameras out of a plurality of cameras correspond. The image generation device 100 sets a plane region R1a of height HT different from the plane region R1, and determines whether or not the coordinate Pa1 is subjected to loss prevention processing, on the basis of the positions of two coordinates Pa1v, Pa2v of the plane region R1a corresponding to the coordinate Pa1 of the plane region R1.

Description

  The present invention relates to an image generation apparatus that generates an output image based on a plurality of input images captured by a plurality of cameras attached to an object to be operated, and an operation support system using the apparatus.

  Conventionally, there is known a driving support system that converts each of images captured by a plurality of in-vehicle cameras that capture the periphery of a vehicle into a bird's-eye view image and presents an output image obtained by connecting the bird's-eye view images to the driver (for example, , See Patent Document 1).

  When this driving support system generates an overlapping area image corresponding to the overlapping area of the imaging ranges of two cameras, the driving support system is a comb-teeth shape that is a boundary line of one complicated shape that divides the overlapping area image into two. , And the two bird's-eye view images are connected so that the partial regions of the two bird's-eye view images captured by the two cameras are alternately arranged in a comb-tooth shape.

  Normally, an object with a height that exists in the overlapping area is projected in two directions in the extension direction of the line connecting each of the two cameras and the object. The two regions are divided into two regions, one region is made to correspond to the bird's eye view image by one camera, and the other region is made to correspond to the bird's eye view image by the other camera. It will disappear.

  In this respect, this driving support system can prevent the height object from disappearing from the overlapping area image by the above-mentioned connection, and make the object easy for the driver to recognize. I can do it.

  In addition, the applicant of the present invention eliminates an object existing in the overlapping region from the output image in order to prevent the driver from feeling unnatural by the image using the comb-like boundary line as described above. An image generation apparatus that generates an output image that prevents the difference in brightness between input images from being noticeable while preventing the occurrence of the image is proposed (for example, see Patent Document 2).

JP 2007-109166 A JP 2012-256295 A

  However, the driving support system described in Patent Document 1 performs a process for preventing the disappearance of the object as described above (hereinafter referred to as “disappearance prevention process”) for the entire overlapping area of the imaging ranges of the two cameras. Make it. For this reason, the greater the overlapping of the imaging ranges of the two cameras, the larger the range in which the comb-like boundary line is set, which may make the driver who sees the output image feel unnatural. In addition, the image generation apparatus described in Patent Document 2 does not mention adjustment of a range in which the loss prevention process is performed.

  In view of the above-described points, an object of the present invention is to provide an image generation apparatus capable of appropriately adjusting a range in which a loss prevention process is performed, and an operation support system using the apparatus.

  In order to achieve the above object, an image generation apparatus according to an embodiment of the present invention is an image generation apparatus that generates an output image based on a plurality of input images captured by a plurality of cameras attached to an operated body. A coordinate association means for associating coordinates in a spatial model arranged around the object to be operated with coordinates in an input image plane on which the input image is located, and via the coordinates in the spatial model, the input Output image generation means for generating the output image by associating the value of the coordinate on the image plane and the value of the coordinate on the output image plane on which the output image is located, and the spatial model includes the plurality of cameras. A first plane area including a first overlapping output portion capable of associating coordinates on the input image plane of each of the two cameras, and the image generation device includes: A second plane area having a predetermined height different from the one plane area and including a second overlapping output portion capable of associating coordinates on the input image planes of the two cameras, and The two overlapping output portions are a first camera corresponding portion that associates coordinates in one input image plane of the two cameras and a second camera corresponding portion that associates coordinates in the other input image plane of the two cameras. And the image generation device outputs the portion corresponding to each of the first camera corresponding portion and the second camera corresponding portion of the first overlapping region to be a non-applicable range of the loss prevention processing. Generate an image.

  An operation support system according to an embodiment of the present invention is an operation support system that supports movement or operation of an object to be operated, for moving or operating the above-described image generation device and the object to be operated. And a display unit that is installed in the operation room and displays an output image generated by the image generation apparatus.

  With the above-described means, the present invention can provide an image generation apparatus capable of appropriately adjusting the range in which the loss prevention process is performed, and an operation support system using the apparatus.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on this invention. It is a figure which shows the structural example of the shovel mounted with an image generation apparatus. It is a figure which shows an example of the space model on which an input image is projected. It is a figure which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a display example (the 2) of an output image. It is a figure for demonstrating the same color illusion. It is a figure for demonstrating an example of the method of producing | generating the light-and-dark pattern which brings about the same color illusion. It is a figure which shows the state which has arrange | positioned the lattice pattern of FIG. 13 on an output image plane. It is a flowchart which shows the flow of a lattice pattern formation process. FIG. 12 is a comparison diagram showing a difference between the output image shown in FIG. 11 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 11. It is a display example (the 3) of an output image. It is a figure for demonstrating the loss | disappearance prevention effect of the object in the overlapping area | region of each imaging range of two cameras. FIG. 18 is a contrast diagram showing a difference between the output image shown in FIG. 17 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 17. FIG. 5 is a first diagram illustrating an output image that has been subjected to a loss prevention process; It is a figure which shows the relationship between the object which exists behind a shovel, and the projection image of the object in the road surface image part of an output image. It is a schematic diagram which shows the relationship between the duplication output part in a 1st plane area | region, and the duplication output part in a 2nd plane area | region. It is a flowchart which shows the flow of a 2nd disappearance prevention process application range adjustment process. It is a flowchart which shows the flow of a loss prevention process necessity determination process. It is explanatory drawing of the setting method of a boundary line. FIG. 10 is a second diagram illustrating an output image that has been subjected to a loss prevention process. It is a flowchart which shows the flow of a 3rd loss prevention process application range adjustment process.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing a configuration example 100 of an image generation apparatus according to the present invention.

  The image generation apparatus 100 is an apparatus that generates an output image based on an input image captured by a camera 2 mounted on a construction machine and presents the output image to a driver, for example, a control unit 1, a camera 2, An input unit 3, a storage unit 4, and a display unit 5 are included.

  FIG. 2 is a diagram illustrating a configuration example of the excavator 60 on which the image generating apparatus 100 is mounted. The excavator 60 is configured such that the upper swing body 63 is placed on the crawler-type lower traveling body 61 via the swing mechanism 62. It is mounted so as to be pivotable around the pivot axis PV.

  The upper swing body 63 includes a cab (operator's cab) 64 on the front left side, a drilling attachment E on the front center, and the camera 2 (right camera 2R, rear camera 2B) on the right and rear surfaces. ). In addition, the display part 5 shall be installed in the position in the cab 64 where the driver | operator is easy to visually recognize.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), etc. A program corresponding to each of the attaching means 10 and the output image generating means 11 is stored in the ROM or NVRAM, and the CPU executes processing corresponding to each means while using the RAM as a temporary storage area.

  The camera 2 is a device for acquiring an input image that reflects the periphery of the excavator 60. For example, the camera 2 is attached to the right side surface and the rear surface of the upper swing body 63 so as to be able to capture an area that is a blind spot of the driver in the cab 64. (See FIG. 2), a right side camera 2R and a rear camera 2B provided with an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens or a fisheye lens is attached so as to capture a wide range. It may be.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Although it is output, an input image in which the apparent distortion or tilt is not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display, a projector, or the like installed in a cab 64 (see FIG. 2) of a construction machine. Display an image.

  Further, the image generating apparatus 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding obstacles and a sense of distance can be intuitively grasped. After generating the output image to be performed, the output image may be presented to the driver.

  The “processing target image” is an image that is generated based on an input image and that is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.). When an input image including a horizontal image (for example, an empty portion) is used in an image conversion process with an image captured from above by a camera that captures the image from above, the horizontal image is The input image is projected onto a predetermined spatial model so that it is not unnaturally displayed (for example, the sky part is not treated as being on the ground surface), and then the projected image projected onto the spatial model is changed to another two. It is an image suitable for image conversion processing, which is obtained by reprojecting onto a dimensional plane. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “spatial model” is a plane or curved surface other than the processing target image plane that is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane or an angle with the processing target image plane). A plane or a curved surface to be formed), and a projection target of an input image composed of one or a plurality of planes or curved surfaces.

  Note that the image generation apparatus 100 may generate an output image by performing image conversion processing on the projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image as it is without being subjected to image conversion processing.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected. FIG. 3A illustrates a relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. FIG. 3B shows the relationship between the excavator 60 and the space model MD when the excavator 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and includes a planar region R1 inside the bottom surface and a curved region R2 inside the side surface.

  FIG. 4 is a diagram illustrating an example of a relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is a plane including the plane area R1 of the space model MD, for example. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for the sake of clarity, the space model MD may be either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the subsequent drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the spatial model MD, or may be an annular area not including the plane area R1 of the spatial model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate association means 10 is a means for associating coordinates on the input image plane where the input image captured by the camera 2 is located, coordinates on the space model MD, and coordinates on the processing target image plane R3. For example, various parameters relating to the camera 2 such as optical center, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc., which are set in advance or input via the input unit 3 And the coordinates on the input image plane, the coordinates on the space model MD, and the processing target image based on the predetermined positional relationship among the input image plane, the spatial model MD, and the processing target image plane R3. The coordinates on the plane R3 are associated with each other, and the corresponding relationship is stored in the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 of the storage unit 4. .

  When the processing target image is not generated, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

  The output image generation unit 11 is a unit for generating an output image. For example, by performing scale conversion, affine conversion, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image are changed. The input image / space model in which the coordinates on the output image plane that is positioned are associated, the correspondence is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the value is stored in the coordinate association means 10. With reference to the correspondence map 40 and the spatial model / processing object image correspondence map 41, the value of each pixel in the output image (for example, the luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image. To generate an output image.

  Further, the output image generation means 11 is preset or input via the input unit 3, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method. The coordinates on the processing target image plane R3 and the coordinates on the output image plane where the output image is located are associated with each other based on various parameters such as the processing target image / output image correspondence map 42 in the storage unit 4. And the values of each pixel in the output image (for example, the luminance value) while referring to the input image / spatial model correspondence map 40 and the spatial model / processing object image correspondence map 41 stored by the coordinate matching means 10. , Hue value, saturation value, etc.) and the value of each pixel in the input image are associated with each other to generate an output image.

  Note that the output image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the processing target image is not generated, the output image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane in accordance with the applied image conversion process, and the input image / space model. With reference to the correspondence map 40, the output image is generated by associating the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) with the value of each pixel in the input image. In this case, the output image generation unit 11 omits the association between the coordinates on the processing target image plane R3 and the coordinates on the output image plane, and the storage of the correspondence relationship in the processing target image / output image correspondence map 42. To do.

  Next, an example of specific processing by the coordinate association unit 10 and the output image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 has UVW orthogonal coordinates with the optical center C of the camera 2 as the origin. The space model is represented as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate association unit 10 converts the coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is optically converted. After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" indicates that the direction is reversed) This means that the XYZ coordinate system is rotated so that the UVW coordinate system coincides with the + W direction in front of the camera and the XYZ coordinate system in the −Z direction vertically below.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate association unit 10 uses an XYZ coordinate system in parallel with each of the plurality of UVW coordinate systems. It will be moved and rotated.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. Since it is realized by rotating to coincide with the axis, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this transformation in Hamilton's quaternion. it can.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the surface extending between the vector A and the vector B as an axis. If the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It will be expressed as

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

The conjugate quaternion of the quaternion Q is

It shall be represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. A point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

It will be expressed as

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , The quaternion R representing "rotation to match the X axis with the U axis"

It will be expressed as

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

Since the quaternion R is invariable in each of the cameras 2, the coordinate association unit 10 thereafter performs the above calculation to obtain the coordinates on the space model (XYZ coordinate system). It can be converted into coordinates on the input image plane (UVW coordinate system).

  After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 determines the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model with a coordinate P ′ represented in the UVW coordinate system and the optical axis G of the camera 2 is calculated.

  In addition, the coordinate association unit 10 includes an intersection E between the plane H and the optical axis G, and a coordinate P ′ in a plane H that is parallel to the input image plane R4 (for example, CCD plane) of the camera 2 and includes the coordinate P ′. Are calculated, and the deviation angle φ formed by the line segment EP ′ connecting the two and the U ′ axis in the plane H, and the length of the line segment EP ′.

  In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate matching means 10 performs normal projection (h = ftanα) and orthographic projection (h = fsinα). Image height by selecting an appropriate projection method such as stereo projection (h = 2 ftan (α / 2)), equisolid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, the coordinates P (P ′) on the space model MD can be associated with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

It will be expressed as

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  Note that, when it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate associating means 10 inputs the coordinates P (P ′) on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with coordinates on the image plane R4, or may be associated with coordinates on the input image plane R4 selected by the operator.

  Next, a process of reprojecting coordinates on the curved surface area R2 (coordinates having a component in the Z-axis direction) among the coordinates on the spatial model MD onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between coordinates by the coordinate association means 10, and FIG. 6A shows an input image plane R4 of the camera 2 that employs a normal projection (h = ftanα) as an example. FIG. 6 is a diagram showing a correspondence relationship between the coordinates on the upper surface and the coordinates on the space model MD, and the coordinate associating means 10 displays the coordinates on the input image plane R4 of the camera 2 and the space model MD corresponding to the coordinates. Both the coordinates are made to correspond to each other so that each of the line segments connecting the coordinates passes through the optical center C of the camera 2.

  In the example of FIG. 6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane area R1 of the space model MD, and inputs the image 2 R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 6B is a diagram showing a correspondence relationship between coordinates on the curved surface region R2 of the space model MD and coordinates on the processing target image plane R3, and the coordinate association unit 10 is positioned on the XZ plane. The parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the spatial model MD and the processing target image corresponding to the coordinates are introduced. The coordinates on the plane R3 are associated with each other such that the coordinates are on one of the parallel line groups PL.

  In the example of FIG. 6B, the coordinate matching means 10 assumes that the coordinate L2 on the curved surface region R2 of the space model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line, and both coordinates are obtained. Make it correspond.

  The coordinate association means 10 can associate the coordinates on the plane area R1 of the spatial model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface area R2. However, in the example of FIG. 6B, since the plane area R1 and the processing target image plane R3 are a common plane, the coordinates L1 on the plane area R1 of the spatial model MD and the processing target image plane R3 The coordinate M1 has the same coordinate value.

  In this way, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. And stored in the space model / processing object image correspondence map 41.

  FIG. 6C is a diagram illustrating a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V that adopts the normal projection (h = ftanα) as an example. The output image generation means 11 passes through the optical center CV of the virtual camera 2V each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates. In this way, the two coordinates are associated with each other.

  In the example of FIG. 6C, the output image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane area R1 of the space model MD). The coordinate N2 on the output image plane R5 of the virtual camera 2V is associated with the coordinate M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  Note that when the virtual camera 2V employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the output image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the output image generation means 11 is a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the output image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. Are stored in the processing target image / output image correspondence map 42 and the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored by the coordinate matching means 10 are referred to in the output image. An output image is generated by associating the value of each pixel with the value of each pixel in the input image.

  FIG. 6D is a combination of FIGS. 6A to 6C. The camera 2, the virtual camera 2V, the plane area R1 and the curved area R2 of the space model MD, and the processing target. The mutual positional relationship of image plane R3 is shown.

  Next, the operation of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 7A is a diagram in the case where the angle β is formed between the parallel line group PL positioned on the XZ plane and the processing target image plane R3, and FIG. 7B is the diagram on the XZ plane. It is a figure in case angle (beta) 1 ((beta) 1> (beta)) is formed between the parallel line group PL and the process target image plane R3. Also, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 7A and 7B corresponds to the coordinates Ma to Md on the processing target image plane R3, respectively. Assume that the intervals between the coordinates La to Ld in FIG. 7A are equal to the intervals between the coordinates La to Ld in FIG. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. It shall be. In this case, the Z axis is referred to as a “reprojection axis”.

  As shown in FIGS. 7A and 7B, the distance between the coordinates Ma to Md on the processing target image plane R3 is such that the angle between the parallel line group PL and the processing target image plane R3 is the same. As it increases, it decreases linearly (it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md). On the other hand, since the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is such that only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD is linearly enlarged or out of the image portion on the output image plane R5 (see FIG. 6). It means to be reduced.

  Next, an alternative example of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 8A is a diagram in the case where all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3, and FIG. It is a figure in case all the line groups AL extend toward the process target image plane R3 from the starting point T2 (T2> T1) on the Z axis. 8A and 8B, the coordinates La to Ld on the curved surface region R2 of the spatial model MD correspond to the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are outside the region of the processing target image plane R3.) The intervals between the coordinates La to Ld in FIG. It is assumed that the interval between the coordinates La to Ld in 8 (B) is equal. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the processing target image plane R3. It shall be. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (high) between the starting point of the auxiliary line group AL and the origin O. As the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md is larger, the width of reduction of each interval is larger. On the other hand, since the coordinate group on the plane region R1 of the spatial model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image portion is enlarged or reduced nonlinearly.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and does not affect the space model MD. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 can be linearly or nonlinearly enlarged or reduced, the road surface image in the vicinity of the excavator 60 Without affecting the excavator 60 (virtual image when viewed from directly above), an object located in the periphery of the excavator 60 (an object in the image when viewed in the horizontal direction from the excavator 60) can be quickly and flexibly The visibility of the blind spot area of the excavator 60 can be improved.

  Next, referring to FIG. 9, the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. Processing to be generated (hereinafter referred to as “output image generation processing”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Further, it is assumed that the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved surface area R2), and the processing target image plane R3 is determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD using the coordinate association unit 10 (step S1).

  Specifically, the coordinate association unit 10 obtains an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. One point that intersects the curved surface region R2 of the space model MD is calculated, and a coordinate on the curved surface region R2 corresponding to the calculated point is set to one on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived as coordinates, and the corresponding relationship is stored in the space model / processing object image correspondence map 41. It should be noted that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be a value to be entered.

  In addition, when one coordinate on the processing target image plane R3 matches one coordinate on the plane area R1 of the space model MD, the coordinate association unit 10 uses the one coordinate on the plane area R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing object image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate association unit 10 to associate one coordinate on the spatial model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate association unit 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα), is a line segment extending from one coordinate on the space model MD, and the optical center A point where the line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are set on the input image plane R4 corresponding to the one coordinate on the space model MD. And the corresponding relationship is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3), and all the coordinates are still set. Are determined to be not associated (NO in step S3), the processes in steps S1 and S2 are repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation process, starts the output image generation process, and outputs the output image generation unit 11. Thus, the coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

  Specifically, the output image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image, and is determined by the content of the applied scale conversion, affine conversion, or distortion conversion. The correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating the output image using the virtual camera 2V, the output image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method. The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating an output image using the virtual camera 2V that employs normal projection (h = ftanα), the output image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V, A line segment extending from one coordinate on the output image plane R5 and calculating a point where a line segment passing through the optical center CV intersects the processing target image plane R3, and on the processing target image plane R3 corresponding to the calculated point. The coordinates may be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  After that, the control unit 1 uses the output image generation unit 11 to refer to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. Correspondence between coordinates on R4 and coordinates on space model MD, correspondence between coordinates on space model MD and coordinates on processing target image plane R3, and coordinates on processing target image plane R3 and output image plane R5 The correspondence with the upper coordinates is traced, and values (for example, luminance values, hue values, saturation values, etc.) possessed by the coordinates on the input image plane R4 corresponding to the respective coordinates on the output image plane R5 are acquired. Then, the acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the output image generation unit 11 respectively outputs the plurality of coordinates on the plurality of input image planes R4. A statistical value (for example, an average value, a maximum value, a minimum value, an intermediate value, or the like) based on the value of L is derived, and the statistical value is adopted as the value of the one coordinate on the output image plane R5. Good.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6), and all the coordinate values are still associated. If it is determined that it is not attached (NO in step S6), the processes in steps S4 and S5 are repeated.

  On the other hand, if the control unit 1 determines that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

  Note that, when the processing target image is not generated, the image generation apparatus 100 omits the processing target image generation processing, and sets “coordinates on the processing target image plane” in step S4 in the output image generation processing to “on the spatial model”. It shall be read as "coordinates".

  With the above configuration, the image generation apparatus 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the construction machine and the surrounding obstacle.

  Further, the image generation apparatus 100 associates coordinates on the processing target image plane R3 from the processing target image plane R3 to the input image plane R4 through the spatial model MD, thereby obtaining the coordinates on the processing target image plane R3. One or more coordinates on R4 can be reliably associated, and compared with a case where coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD (in this case) Can reliably correspond each coordinate on the input image plane R4 to one or a plurality of coordinates on the processing target image plane R3. However, a part of the coordinates on the processing target image plane R3 is part of the input image plane R4. In some cases, the coordinates may not be associated with any of the above coordinates, and in such a case, it is necessary to perform interpolation processing or the like on a part of the coordinates on the processing target image plane R3). It is possible to rapidly generate.

  Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generating apparatus 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Thus, it is possible to realize a desired enlargement or reduction without rewriting the contents of the input image / space model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the space model / processing object image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters relating to scale conversion, affine transformation, or distortion transformation. The desired output image (scale-converted image, affine-transformed image, or distortion-converted image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing object image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

  The image generation apparatus 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects them onto the processing target image plane R3 to generate a processing target image. Then, an image conversion process (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion process, etc.) is performed on the generated processing target image to generate an output image, and the vicinity of the excavator 60 can be seen from above. An image looking down (image in the plane region R1) and an image of the periphery viewed from the excavator 60 in the horizontal direction (image in the processing target image plane R3) are displayed at the same time.

  When the image generation apparatus 100 does not generate the processing target image, the output image is generated by performing image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD. Shall.

  The output image is trimmed in a circle so that the image when the excavator 60 performs the turning motion can be displayed without a sense of incongruity, and the center CTR of the circle is on the cylindrical central axis of the space model MD and the excavator 60 is turned. It is generated so as to be on the axis PV, and is displayed so as to rotate about its center CTR in accordance with the turning operation of the excavator 60. In this case, the cylindrical central axis of the space model MD may or may not coincide with the reprojection axis.

  The radius of the space model MD is, for example, 5 meters, and the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is the turning of the shovel 60. When an object (for example, a worker) exists at a position away from the center by a maximum reachable distance (for example, 12 meters) of the excavation attachment E, the object is sufficiently large (for example, 7) It can be set to be displayed.

  Further, the output image may be a CG image of the excavator 60 arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. This is to make the positional relationship between the shovel 60 and the object appearing in the output image easier to understand. Note that a frame image including various kinds of information such as an orientation may be arranged around the output image.

  Next, with reference to FIGS. 11 to 16, processing for preventing the image generation apparatus 100 from highlighting the difference in brightness (luminance) between input images will be described.

  FIG. 11 shows input images of the three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) mounted on the excavator 60, and output images generated using these input images. FIG.

  The image generation apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the space model MD and then reprojects them onto the process target image plane R3 to generate a process target image. Then, an image conversion process (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion process, etc.) is performed on the generated processing target image to generate an output image, and the vicinity of the excavator 60 can be seen from above. An image looking down (image in the plane region R1) and an image of the periphery viewed from the excavator 60 in the horizontal direction (image in the processing target image plane R3) are displayed at the same time.

  In FIG. 11, the input image of the left camera 2L and the input image of the rear camera 2B each include overlapping portions obtained by imaging the same place around the excavator 60 from different angles, and these overlapping portions are respectively illuminations for the respective cameras. Images are taken with different brightness depending on the environment or the like, and the brightness is different. The same applies to the overlapping portion between the input image of the right camera 2R and the input image of the rear camera 2B.

  Therefore, the output image generated based on these three input images has the coordinates on the output image plane incident when the coordinates on the output image plane can be associated with the coordinates on the plurality of input image planes. Assuming that the camera is associated with the coordinates on the input image plane regarding the camera with the smallest corner, the area on the output image based on the input image of the left camera 2L and the rear camera 2B, even though the same place is captured The boundary between the region on the output image based on the input image (see the region R6 surrounded by the one-dot chain line in the figure) and the region on the output image based on the input image of the right side camera 2R and the rear camera 2B The brightness changes abruptly at the boundary with the region on the output image based on the input image (see the region R7 surrounded by the one-dot chain line in the figure), and the driver who viewed the output image So that the would feel unnatural.

  Therefore, the image generation apparatus 100 uses the effect of the checker shadow illusion (same color illusion) so as not to make a difference in luminance between input images conspicuous.

  FIG. 12 is a diagram for explaining the same color illusion (Edward H. Adelson, “Checker shadow illusion”, 1995, Internet <URL: http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html 12) shows a checker pattern (lattice pattern) used for providing the same color illusion, and FIG. 12 (B) shows points P1 to P3 on the checker pattern (lattice pattern). And the graph showing transition of the brightness | luminance in the point P4-the point P6 is shown.

  As shown in FIG. 12A, two unit patterns (a unit pattern including the point P1 and a unit pattern including the point P3) having a luminance higher than that of the unit pattern including the point P2 are sandwiched. The luminance of the unit pattern including the point P2 is sandwiched between two unit patterns (a unit pattern including the point P4 and a unit pattern including the point P6) having a luminance lower than that of the unit pattern including the point P5. It appears that the brightness is lower than the brightness of the unit pattern including the point P5.

  However, this difference in luminance is an observation that recognizes that each unit pattern forms a lattice pattern and that the unit pattern group including the points P4 to P6 is in the shadow of the object OBJ. In fact, as shown in FIG. 12B, the luminance of the unit pattern including the point P2 is equal to the luminance of the unit pattern including the point P5.

  Thus, the observer has the same brightness in the unit pattern including each of the points P1, P3, and P5, and the unit pattern including each of the points P2, P4, and P6 has the same brightness. You will have an illusion of what you have.

  FIG. 13 is a diagram for explaining an example of a method for generating a light / dark pattern that causes the same color illusion, and includes two wave sources (for example, a right camera 2R and a rear camera 2B. Note that the right camera 2R is illustrated in FIG. This is for the sake of clarity of the drawing, and here, the upper right side of the drawing corresponds to the rear of the excavator 60) to the mountain (solid circle) and the valley Two waves that spread and interfere with each other while forming a (dashed circle) are shown.

  In FIG. 13, two fan-shaped areas 2Ra and 2Ba indicate the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B, respectively. The abdominal line ANL represented by a thick solid line has two waves. A line connecting points that strengthen each other is shown, and a node line NL indicated by a thick broken line indicates a line connecting points where two waves weaken each other. The abdominal line ANL and the nodal line NL appear alternately as shown in FIG.

  Further, in FIG. 13, a line drawn by one trough of a wave that spreads using the right-side camera 2R as a wave source (broken line circle), a line drawn by one mountain generated after the one trough (solid circle), and the rear Each of the rhombus-shaped regions defined by a line (broken line circle) drawn by one trough of a wave spreading using the camera 2B as a wave source and a line (solid line circle) drawn by one mountain generated next to the one valley are The unit pattern region LT is formed.

  It is assumed that a plurality of unit pattern areas LT shown in FIG. 13 are drawn on the output image plane, and the rear camera 2B is passed to the unit pattern area LT1 through which the abdominal line ANL of the unit pattern areas LT passes (filled in gray). If the input image of the right side camera 2R is associated with the unit pattern region LT2 through which the nodal line NL of the unit pattern region LT passes (filled in white), the average image Two input images having different luminances can form a lattice pattern that causes the same color illusion.

  Note that the input image of the right-side camera 2R is associated with the unit pattern area LT1 through which the abdominal line ANL in the unit pattern area LT passes (filled in gray), and the nodal line NL in the unit pattern area LT. Similarly, when the input image of the rear camera 2B is associated with the unit pattern region LT2 through which the image passes (filled in white), two input images having different average luminances form a lattice pattern that causes the same color illusion. Will be able to.

  13 is formed by using two waves having the same wavelength and phase, it may be formed by using two waves having different wavelengths and / or phases. This is because the size and shape of the unit pattern areas LT1 and LT2 can be adjusted flexibly.

  FIG. 14 shows a state in which the lattice pattern shown in FIG. 13 is arranged on the output image plane (planar region R1), and the right-side camera 2R is located on the right rear side (lower right direction in the figure) of the CG image of the excavator 60. The imaging range of the left-side camera 2L and the rear camera 2B located in the lattice pattern arranged in the overlapping area of the imaging range of the rear camera 2B and the imaging range of the rear camera 2B and the left rear of the CG image of the shovel 60 (lower left direction in the figure) The lattice pattern arrange | positioned in the duplication area | region with an imaging range is shown.

  In FIG. 14, the grid pattern arranged in the overlapping area of the imaging range of the right camera 2R and the imaging range of the rear camera 2B is, for example, an input image of the right rear camera 2R in the unit pattern area LT1 (filled in gray). And the input image of the rear camera 2B is associated with the unit pattern area LT2 (filled in white).

  The grid pattern arranged in the overlapping area of the imaging range of the left camera 2L and the imaging range of the rear camera 2B corresponds to, for example, the input image of the left rear camera 2L corresponding to the unit pattern area LT3 (filled in gray). It is assumed that the input image of the rear camera 2B is associated with the unit pattern region LT4 that is attached (filled in white).

  In FIG. 15, the image generation apparatus 100 corresponds to coordinates on the output image plane corresponding to the overlapping area of the imaging ranges of the two cameras with coordinates on the input image plane of either of the two cameras. 2 is a flowchart showing a flow of processing for forming a lattice pattern that causes the same color illusion (hereinafter referred to as “lattice pattern formation processing”).

  The control unit 1 of the image generation apparatus 100, for example, uses the coordinate matching means 10 on one coordinate and a plurality of input image planes on the plane region R1 of the spatial model MD in step S2 of the processing target image generation process in FIG. When it is possible to associate one coordinate with the other coordinate, one coordinate on the plane region R1 of the spatial model MD is associated with one coordinate on one of the two input image planes corresponding to each of the two cameras. Suppose that this lattice pattern forming process is executed.

  First, the control unit 1 acquires one coordinate on the plane region R1 of the spatial model MD corresponding to the region where the imaging ranges of two cameras (for example, the right camera 2R and the rear camera 2B) overlap. (Step S11).

  Next, the control part 1 acquires the coordinate of the optical center in each of two cameras (step S12).

  Next, the control part 1 selects the camera matched with one coordinate on plane area | region R1 in the space model MD acquired at step S11 (step S13).

Specifically, the control unit 1 sets the coordinates of the optical center of the right-side camera 2R as (X _cam1 , Y _cam1 ) and the coordinates of the optical center of the rear camera 2B as (X _cam2 , Y _cam2 ). When one coordinate on the plane region R1 of the target space model MD is (X _target , Y _target ),

Is true, the right-side camera 2R is selected as a camera to be associated with one coordinate on the plane area R1, and one coordinate on the plane area R1 is selected when the above-described conditional expression is false. The rear camera 2R is selected as the camera to be associated with.

  The control unit 1 selects the rear camera 2B as a camera to be associated with one coordinate on the plane region R1 when the above-described conditional expression is true, and when the above-described conditional expression is false, the control unit 1 The right side camera 2R may be selected as a camera associated with one coordinate on the region R1.

Note that the conditional expression described above is a determination formula for determining whether the coordinates (X _target , Y _target ) on the plane region R1 are included in the unit pattern region LT1 shown in FIG. 14 or included in the unit pattern region LT2. It corresponds to.

  Further, in this embodiment, the control unit 1 assumes that the coordinates of the optical center are two-dimensional coordinates, and that the wave generated from the wave source is a plane wave, one coordinate (two-dimensional coordinates) and two on the plane region R1. The cameras are selected based on the two-dimensional distance between the coordinates of the optical centers of the two cameras (two-dimensional coordinates projected on the plane including the plane region R1). Between the one coordinate (three-dimensional coordinate) on the plane region R1 and the coordinates of the respective optical centers (three-dimensional coordinates) on the plane region R1, with the wave generated from the wave source as a spherical wave. The camera may be selected based on the three-dimensional distance.

  Further, in the present embodiment, the control unit 1 is on the plane area R1 of the spatial model MD corresponding to an area where the imaging ranges of two cameras (for example, the right side camera 2R and the rear camera 2B) overlap. Although the camera associated with one coordinate is selected, the camera associated with one coordinate on the processing target image plane R3 corresponding to the region where the imaging ranges of the two cameras overlap may be selected.

  In this case, the control unit 1 uses one coordinate (two-dimensional coordinate) on the processing target image plane R3 and the coordinates of the optical centers of the two cameras (two-dimensional coordinates projected on the plane including the processing target image plane R3). The camera may be selected on the basis of the two-dimensional distance between them, and one coordinate (three-dimensional coordinate) on the processing target image plane R3 and the coordinates of the optical centers of the two cameras (three-dimensional coordinates). The camera may be selected based on the three-dimensional distance between The processing target image plane R3 may include a plane area R1.

  After that, the control unit 1 associates one coordinate on the input image plane of the selected camera with one coordinate on the plane region R1 of the space model MD by the coordinate association unit 10 (step S14), and the space model MD. The upper coordinates, the camera identifier, and the coordinates on the input image plane are associated and stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 sets all the coordinates on the plane area R1 of the spatial model MD corresponding to the area where the imaging ranges of the two cameras overlap to the coordinates on one input image plane of the two cameras. It is determined whether or not they are associated with each other (step S15). If it is determined that all the coordinates are not yet associated (NO in step S15), the processes in steps S11 to S14 are repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S15), the control unit 1 ends the lattice pattern forming process.

  In the above description, the control unit 1 calculates the coordinates on the plane area R1 of the spatial model MD or the coordinates on the processing target image plane R3 corresponding to the area where the imaging ranges of the two cameras overlap. The coordinates on one input image plane of the two cameras are associated with each other. Further, each coordinate on the curved surface region R2 of the spatial model MD corresponding to the region where the imaging ranges of the two cameras overlap is represented by two. You may make it match | combine with the coordinate on the input image plane of one camera.

  As described above, the control unit 1 can easily associate each coordinate on the space model MD with the coordinate on one input image plane of the two cameras using the conditional expression as described above. A lattice pattern can be generated.

  FIG. 16 is a contrast diagram showing a difference between the output image shown in FIG. 11 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 11, and FIG. The output image shown is shown, and FIG. 16B shows the output image to which the lattice pattern that causes the same color illusion is applied.

  A region R6 surrounded by an alternate long and short dash line in FIG. 16A includes a boundary between a region on the output image based on the input image of the left camera 2L and a region on the output image based on the input image of the rear camera 2B. Although a significant difference in brightness is presented, a region on the output image based on the input image of the left camera 2L and a region on the output image based on the input image of the rear camera 2B are mixed in a lattice pattern. In the region R8 surrounded by the alternate long and short dash line in FIG. 16B, the difference in luminance is inconspicuous, and it is difficult for the driver to feel unnaturalness when viewing the output image including the region R8.

  Similarly, it is surrounded by an alternate long and short dash line in FIG. 16A including a boundary between an area on the output image based on the input image of the right camera 2R and an area on the output image based on the input image of the rear camera 2B. The region R7 presents a significant difference in brightness, but the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B have a grid pattern. In the region R9 surrounded by the alternate long and short dash line in FIG. 16B, the difference in luminance is inconspicuous, and it is difficult for the driver who sees the output image including the region R9 to feel unnaturalness. ing.

  Next, with reference to FIGS. 17 to 19, when the image generation apparatus 100 generates an output image portion corresponding to the overlapping area of the imaging ranges of the two cameras, the object in the output image portion disappears. Deletion prevention processing for preventing this will be described.

  FIG. 17 shows input images of three cameras 2 (left camera 2L, right camera 2R, and rear camera 2B) mounted on the excavator 60, and output images generated using these input images. FIG.

  The image generating apparatus 100 projects the coordinates on the input image plane of each of the three cameras 2 to the coordinates on the plane area R1 and the curved area R2 of the space model MD, and then reprojects them on the processing target image plane R3. The image to be processed is generated, and the output image is generated by subjecting the generated image to be processed to image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.), and the excavator An image in which the vicinity of 60 is looked down from above (image in the plane region R1) and an image in which the periphery is viewed from the shovel 60 in the horizontal direction (image in the processing target image plane R3) are displayed.

  In FIG. 17, the input image of the right-side camera 2R and the input image of the rear camera 2B are respectively in the overlapping area of the imaging range of the right-side camera 2R and the imaging range of the rear camera 2B (the right-side camera 2R A region R10 surrounded by a two-dot chain line in the input image and a region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B are captured.

  However, the output image generated based on the input image of the right-side camera 2R and the input image of the rear camera 2B is the coordinate on the input image plane with respect to the camera having the smallest incident angle on the output image plane. As shown in FIG. 17, the person in the overlapping area disappears (see area R12 surrounded by a one-dot chain line in the output image).

  Therefore, when the image generation apparatus 100 generates an output image portion corresponding to an overlapping region of the imaging ranges of the two cameras using the lattice pattern formed to bring the same color illusion, the output image portion To prevent the disappearance of the object in the area.

  FIG. 18 is a diagram for explaining the effect of preventing disappearance of an object in the overlapping area of the imaging ranges of the two cameras. FIG. 18A is a lattice pattern for providing the same color illusion shown in FIG. FIG. 18 (B) shows a partially enlarged view of FIG. 18 (A).

  In FIG. 18A, a projection image PRJ1 surrounded by a broken line indicates that the image of the object OBJ1 in the input image of the rear camera 2B is converted between the rear camera 2B and the object OBJ1 by viewpoint conversion for generating a road surface image. This is an image that is expanded in the extending direction of the connecting line (an image that is displayed when a road surface image in the output image portion is generated using the input image of the rear camera 2B).

  Further, the projection image PRJ2 surrounded by a broken line is a line connecting the right-side camera 2R and the object OBJ1 by the viewpoint conversion for generating the road surface image of the object OBJ1 in the input image of the right rear camera 2R. This is an image expanded in the extension direction (an image displayed when a road surface image in the output image portion is generated using an input image of the right-side camera 2R).

  Note that the projection image PRJ1 and the projection image PRJ2 are not displayed as they are on the final output image, and are displayed in a partially cut-out state as shown in FIG. Will be.

  Further, in FIG. 18A, a portion filled with gray in a broken line region representing the projection image PRJ1 represents an overlapping portion with a unit pattern region LT1 (see FIG. 13) through which the abdominal line ANL passes, Each coordinate on the output image corresponding to the overlapping portion is associated with a coordinate on the input image plane of the rear camera 2B (coordinate in a region forming an image of the object OBJ1).

  On the other hand, in the broken line area representing the projection image PRJ1, the part filled in white represents the overlapping part with the unit pattern area LT2 (see FIG. 13) through which the nodal line NL passes, and the output corresponding to the overlapping part. Each coordinate on the image is associated with a coordinate on the input image plane of the right-side camera 2R (coordinate in a region where an image of the object OBJ1 is not formed).

  Of the broken line area representing the projection image PRJ1, the coordinates on the input image plane of the right-side camera 2R (in the area where the image of the object OBJ1 is not formed) In the broken line area representing the projection image PRJ1, the coordinates on the input image plane of the rear camera 2B (the image of the object OBJ1) May be associated with each other).

  In this case, as shown in FIG. 18B, the projection image PRJ1 is one of the circles perpendicular to the extension direction (the circles drawn by the peaks and valleys of the waves with the position of the rear camera 2B as the wave source). Is cut out by a unit pattern region LT2 including the portion as a boundary line, but since adjacent unit pattern regions LT1 are in contact with each other, they are difficult to be divided into pieces, and each of the unit pattern regions LT2 Since a part of the circle orthogonal to the extending direction is included, the contour is also stored in a clear state that is easily recognized.

  Further, the projection image PRJ1 has a tendency to be elongated in a direction away from the camera as the height of the object OBJ1 is higher, and to be enlarged as the distance from the camera is larger. However, the unit pattern regions LT1 and LT2 are However, the further away from the camera, the larger the image is enlarged in the same degree as the projection image PRJ1, so that the cutout state is also maintained substantially uniform.

  Further, in FIG. 18A, in the broken line area representing the projection image PRJ2, the part filled in black represents the overlapping part with the unit pattern area LT2 (see FIG. 13) through which the nodal line NL passes, Each coordinate on the output image corresponding to the overlapped portion is associated with a coordinate on the input image plane of the right-side camera 2R (coordinate in a region forming the image of the object OBJ1).

  On the other hand, in the broken line area representing the projection image PRJ2, the part filled in white represents the overlapping part with the unit pattern area LT1 (see FIG. 13) through which the abdominal line ANL passes, and the output corresponding to the overlapping part. Each coordinate on the image indicates that a coordinate on the input image plane of the rear camera 2B (a coordinate in a region where an image of the object OBJ1 is not formed) is associated.

  Of the broken line area representing the projection image PRJ2, the coordinates on the input image plane of the rear camera 2B (coordinates in the area where the image of the object OBJ1 is not formed) are displayed on the coordinates on the output image corresponding to the portion filled in black. ) And the coordinates on the input image plane of the right-side camera 2R (the image of the object OBJ1 is formed at each coordinate on the output image corresponding to the portion painted in white in the broken line area representing the projection image PRJ2. Coordinates in the area to be performed) may be associated with each other.

  In this case, the projection image PRJ2 is similar to the projection image PRJ1, as shown in FIG. 18B, a circle (wave peaks and valleys whose wave source is the position of the right side camera 2R is orthogonal to the extension direction thereof. Is cut out by the unit pattern region LT1 including a part of the circle as a boundary line. However, since the adjacent unit pattern regions LT2 are in contact with each other, it is difficult to be cut into pieces. Since each of the unit pattern regions LT1 includes a part of a circle orthogonal to the extending direction, the contour is also stored in a clear state that is easily recognized.

  Similarly to the projection image PRJ1, the projection image PRJ2 has a tendency to extend longer in the direction away from the camera as the height of the object OBJ1 is higher, and to be enlarged as the distance from the camera increases. As the pattern regions LT1 and LT2 are further away from the camera, the pattern regions LT1 and LT2 are enlarged in the same degree as the projected image PRJ2, so that the cutout state is maintained substantially uniform.

  FIG. 19 is a contrast diagram showing a difference between the output image shown in FIG. 17 and an output image in which a lattice pattern that causes the same color illusion is applied to the output image of FIG. 17, and FIG. The output image shown is shown, and FIG. 19B shows the output image to which the lattice pattern that causes the same color illusion is applied.

  A region R13 surrounded by an alternate long and short dash line in FIG. 19A including the boundary between the region on the output image based on the input image of the right camera 2R and the region on the output image based on the input image of the rear camera 2B is Although the person has disappeared, the area on the output image based on the input image of the right side camera 2R and the area on the output image based on the input image of the rear camera 2B are mixed in a lattice pattern. A region R14 surrounded by an alternate long and short dash line in FIG. 19B presents the person in an easily recognizable state without losing the person, and the driver who viewed the output image including the region R14 has the person It is designed to ensure that it exists.

  With the above configuration, the image generation apparatus 100 uses the lattice pattern for providing the same color illusion to generate an output image portion when generating an output image portion corresponding to the overlapping area of the imaging ranges of the two cameras. The object in the part can be prevented from disappearing, and compared with the case where the two input images are connected so that the partial areas of the two input images are alternately arranged in a comb-teeth shape. The projected image of the object that will be cut out can be displayed in a state in which the driver can easily recognize it.

  Next, referring to FIG. 20 to FIG. 27, a process for adjusting the range to which the disappearance prevention process is applied (hereinafter referred to as “disappearance prevention process application range”) (hereinafter “disappearance prevention process application range adjustment process”). Will be described.

  The upper part of FIG. 20 is a diagram illustrating an output image that has been subjected to the disappearance prevention process. Regions R15L and R15R surrounded by broken lines are output image portions corresponding to overlapping regions (hereinafter referred to as “overlapping output portions”). ). Specifically, the overlapping output portion R15L corresponds to a region where the imaging ranges of the left side camera 2L and the rear camera 2B overlap, and the overlapping output portion R15R overlaps the imaging ranges of the right side camera 2R and the rear camera 2B. Corresponds to the region.

  As illustrated in the upper part of FIG. 20, the image generation apparatus 100 prevents the objects in the overlapping output portion from disappearing when generating an output image by the disappearance prevention process using the lattice pattern.

  However, since the image generation apparatus 100 performs the disappearance prevention process on the entire overlapped output portion, if the overlapped output portion is large, the visibility of the output image may be reduced. For example, as illustrated in the upper part of FIG. 20, the image generation apparatus 100 causes the overlapping output portions R15L and R15R as the disappearance prevention processing application range to greatly extend to the output image portion R16 corresponding to the imaging range of the rear camera 2B. As a result, the visibility of the output image may be reduced. Note that the overlapping output portion as the disappearance prevention processing application range tends to increase as the overlapping area of the imaging ranges of the two cameras increases, that is, as the angle between the optical axes of the two cameras decreases.

  Therefore, the image generation apparatus 100 reduces the visibility of the output image by reducing the application range of the loss prevention process as much as possible while preventing the object from disappearing from the overlapping output portion by the method described below. To prevent.

  Here, an example of the disappearance prevention process application range adjustment process (hereinafter referred to as “first disappearance prevention process application range adjustment process”) will be described with reference to the middle part of FIG. 20 and the lower part of FIG. 20.

  The middle part of FIG. 20 and the lower part of FIG. 20 are diagrams showing output images that have been subjected to the disappearance prevention process, and correspond to the upper part of FIG. As shown in the middle part of FIG. 20, regions R15La and R15Ra surrounded by broken lines represent the application range of disappearance prevention processing after adjustment. Specifically, the disappearance prevention process application range R15La corresponds to a range obtained by excluding the region R17L located on the right side of the vertical plane including the left side surface of the upper swing body 63 from the overlapping output portion R15L in the upper part of FIG. Further, the disappearance prevention process application range R15Ra corresponds to a range obtained by excluding the region R17R on the left side of the vertical plane including the right side surface of the upper swing body 63 from the overlapping output portion R15R in the upper part of FIG. The line segment L1L represents a top view of the vertical plane including the left side surface of the upper swing body 63, and the line segment L1R represents a top view of the vertical plane including the right side surface of the upper swing body 63. The lower part of FIG. 20 shows a state in which the line segments L1L and L1R and the regions R17L and R17R are not shown.

  The image generation apparatus 100 excludes the region R17L from the overlapping output portion R15L, associates only the input image of the rear camera 2B with the region R17L, and does not associate the input image of the left camera 2L with the region R17L. Objects existing in the region are not lost from the output image. The input image of the rear camera 2B associated with the region R17L is a portion close to the center of the imaging range of the rear camera 2B, while the input image of the left camera 2L associated with the region R17L is the left camera 2L. This is because the portion is close to the end of the imaging range. The same applies to the overlapping output portion R15R.

  In this manner, the image generation apparatus 100 can reduce the application range of the disappearance prevention process without causing the object existing in the overlapping region to disappear from the output image. As a result, the image generating apparatus 100 can prevent the visibility of the output image from being lowered by performing the disappearance prevention process.

  Here, another example of the disappearance prevention process application range adjustment process (hereinafter referred to as “second disappearance prevention process application range adjustment process”) will be described with reference to FIG. 21. FIG. 21 is a diagram illustrating a relationship between an object existing behind the excavator 60 and a projected image of the object in the road surface image portion of the output image. 21 are top views of the excavator 60, and A2 and B2 in FIG. 21 are side views of the excavator 60. The top view of A1 corresponds to the side view of A2, and the top view of B1 corresponds to the side view of B2.

  In the present embodiment, the image generation apparatus 100 determines the disappearance prevention process application range while preventing an object less than a predetermined height from disappearing from the output image by using the property related to the spatial model MD having the planar region R1. To do.

  Specifically, the image generating apparatus 100 uses two planar regions having different heights from the road surface. One of the two plane areas (hereinafter referred to as “first plane area”) is the plane area R1 of the spatial model MD used when generating the output image, and in this embodiment, the height from the road surface. Is a planar area of 0 cm, that is, a planar area having a height equal to the road surface. The other of the two planar regions (hereinafter referred to as “second planar region”) is a planar region R1a having a height different from that of the first planar region, and in this embodiment, the height from the road surface. Is a planar region R1a having a human head height HT (for example, 160 cm).

  A1 and A2 in FIG. 21 show the relationship between the object existing behind the excavator 60 and the projected image projected onto the plane region R1, and B1 and B2 in FIG. The relationship with the projection image projected on area | region R1a is shown.

  In A1 of FIG. 21, the projection images PRJa1 and PRJb1 indicated by the alternate long and short dash lines projected onto the plane region R1 are the projection images of the objects OBJa and OBJb, respectively, and are based on the input image of the rear camera 2B. Projected images PRJa2 and PRJb2 indicated by dotted lines projected onto the planar region R1 are projected images of the objects OBJa and OBJb, respectively, and are based on the input image of the left-side camera 2L.

  The point Pa1 represents the point closest to the rear camera 2B in the projection image PRJa1, and the point Pa2 represents the point closest to the left-side camera 2L in the projection image PRJa2. The point Pb1 represents the point closest to the rear camera 2B in the projection image PRJb1, and the point Pb2 represents the point closest to the left-side camera 2L in the projection image PRJb2. In A1, the point Pa1 and the point Pa2 are the same point (overlapping point), and the point Pb1 and the point Pb2 are the same point (overlapping point). In A2, illustration of the left side camera 2L, the object OBJb, the projection images PRJa2, PRJb1, PRJb2, and the points Pa2, Pb1, Pb2 is omitted for clarity of illustration.

  A line BL in A1 is a line formed by a set of points at which the incident angle with respect to the left camera 2L is equal to the incident angle with respect to the rear camera 2B. That is, an angle (incident angle) formed between a line segment connecting an arbitrary point on the line BL and the optical center of the left-side camera 2L with the optical axis of the left-side camera 2L is an arbitrary point and the rear camera. A line segment connecting the optical center of 2B is equal to an angle (incident angle) formed between the optical axis of the rear camera 2B. A broken line area R15L in A1 represents an overlapping output portion on the planar area R1 corresponding to an overlapping area between the imaging range of the left camera 2L and the imaging range of the rear camera 2B.

  Similarly, in B1 of FIG. 21, projection images PRJa1v and PRJb1v indicated by alternate long and short dash lines projected onto the planar region R1a having the height HT are projection images of the objects OBJa and OBJb, respectively, and are based on the input image of the rear camera 2B. . In addition, projection images PRJa2v and PRJb2v indicated by dotted lines projected onto the planar region R1a having the height HT are projection images of the objects OBJa and OBJb, respectively, and are based on the input image of the left-side camera 2L.

  The point Pa1v represents the point closest to the rear camera 2B in the projection image PRJa1v, and the point Pa2v represents the point closest to the left camera 2L in the projection image PRJa2v. The point Pb1v represents the point closest to the rear camera 2B in the projection image PRJb1v, and the point Pb2v represents the point closest to the left-side camera 2L in the projection image PRJb2v. In B1, unlike in the case of A1, the points Pa1v and Pa2v are separate points, and the points Pb1v and Pb2v are separate points. In B2, the left camera 2L, the object OBJb, the projection images PRJa2v, PRJb1v, PRJb2v, and points Pa2v, Pb1v, and Pb2v are omitted for clarity.

  Similarly to the line BL in A1, the line BLv in B1 is a line formed by a set of points where the incident angle with respect to the left camera 2L and the incident angle with respect to the rear camera 2B are equal. A broken line area R15Lv in B1 represents an overlapping output portion on the planar area R1a corresponding to an overlapping area between the imaging range of the left camera 2L and the imaging range of the rear camera 2B.

  As shown in A1 and A2 of FIG. 21, in the planar region R1, the projection image PRJa1 of the object OBJa is the size of the portion at the height of the road surface (the size of the image at the point Pa1, for example, the foot of a person Is a size commensurate with the distance from the excavator 60. That is, the image generating apparatus 100 has a relationship between the size of the CG image of the shovel 60 in the output image and the size of the portion of the projection image PRJa1 near the point Pa1 between the actual size of the shovel 60 and the actual size of the object OBJa. Try to be equal to the relationship. Then, the image generating apparatus 100 determines that the apparent distance between the CG image of the excavator 60 in the output image and the portion in the vicinity of the point Pa1 of the projection image PRJa1 (distance grasped from each size) is the actual excavator 60. The distance from the actual object OBJa is expressed. Further, the projection image PRJa1 has a tendency to be gradually displayed larger as the distance from the shovel 60 increases along a straight line connecting the point Pa1 and the optical center of the rear camera 2B. Similarly, the projection image PRJa2 of the object OBJa has a size of a portion at the height of the road surface (the size of the image at the point Pa2, for example, the size of the image of a person's feet) from the shovel 60. The size will be suitable for the distance. Then, the projected image PRJa2 has a tendency to be gradually displayed with increasing distance from the shovel 60 along a straight line connecting the point Pa2 and the optical center of the rear camera 2B. Therefore, the projected images PRJa1 and PRJa2 have a tendency to expand while expanding their sizes in two different directions from the points Pa1 and Pa2 that are the same point toward the far side. The same applies to the projection images PRJb1 and PRJb2 of the object OBJb.

  On the other hand, as shown in B1 and B2 of FIG. 21, in the planar region R1a, the projection image PRJa1v of the object OBJa is the size of the portion at the height HT (for example, the size of the image of the human head). Becomes a size corresponding to the distance from the excavator 60. That is, the image generation apparatus 100 has a relationship between the size of the CG image of the shovel 60 in the output image and the size of the portion corresponding to the height HT of the projection image PRJa1v, and the size of the actual shovel 60 and the size of the actual object OBJa. To be equal to the relationship. Then, the image generation apparatus 100 determines that the apparent distance between the CG image of the excavator 60 in the output image and the portion at the height HT of the projection image PRJa1 (the distance grasped from each size) is the actual excavator 60. The distance from the actual object OBJa is expressed. Similarly to the projection image PRJa1, the projection image PRJa1v has a tendency to be gradually enlarged as it moves away from the shovel 60 along a straight line connecting the point Pa1v and the optical center of the rear camera 2B. Similarly, in the projection image PRJa2v of the object OBJa, the size of the portion at the height HT (for example, the size of the image of the head of a person) has a size commensurate with the distance from the shovel 60. And projection image PRJa2 has the tendency which is gradually displayed large as it leaves | separates from the shovel 60 along the straight line which connects point Pa2v and the optical center of the rear camera 2B similarly to projection image PRJa2. Therefore, the projection images PRJa1v and PRJa2v extend from the points Pa1v and Pa2v, which are separate points, while expanding their sizes in two different directions, and are portions of the object OBJa at the height HT (for example, the head There is a tendency to overlap. The same applies to the projection images PRJb1v and PRJb2v of the object OBJb.

  Further, as can be seen from A2 and B2 in FIG. 21, the projection images PRJa1, PRJa2, PRJb1, and PRJb2 are similar to the projection images PRJa1v, PRJa2v, PRJb1v, and PRJb2v, respectively. Specifically, the projection image PRJa1 and the projection image PRJa1v are similar to each other with the position of the rear camera 2B as a similar center, and the projection image PRJb1 and the projection image PRJb1v are similar to each other with the position of the rear camera 2B as a similar center. It is. When the height of the rear camera 2B is H1, the ratio of the lengths of the projection image PRJa1 and the projection image PRJa1v and the ratio of the lengths of the projection image PRJb1 and the projection image PRJb1v are all H1: (H1-HT). It is. The projection image PRJa2 and the projection image PRJa2v are similar to each other with the position of the left camera 2L as the center of similarity, and the projection image PRJb2 and the projection image PRJb2v are similar to each other with the position of the left camera 2L as the center of similarity. is there. When the height of the left camera 2L is H2, the ratio of the lengths of the projection image PRJa2 and the projection image PRJa2v and the ratio of the lengths of the projection image PRJb2 and the projection image PRJb2v are both H2: (H2-HT. ).

  In A1, when the input image of the rear camera 2B is projected on the area on the right side of the line BL and the input image of the left camera 2L is projected on the area on the left side of the line BL, the projection image PRJa1 of the object OBJa is the left camera 2L. The projected image PRJa2 of the object OBJa is overwritten by the input image of the rear camera 2B. As a result, the projection images PRJa1 and PRJa2 disappear with a slight portion around the points Pa1 and Pa2. On the other hand, the projection image PRJb2 of the object OBJb is overwritten by the input image of the rear camera 2B, but the projection image PRJb1 of the object OBJb is not overwritten by the input image of the left camera 2L. That is, the projection image PRJb1 of the object OBJb may be lost even if the input image of the rear camera 2B is projected on the area on the right side of the line BL and the input image of the left camera 2L is projected on the area on the left side of the line BL. Absent. This means that the disappearance prevention process can be omitted while preventing the disappearance of the projected image of the object less than the height HT. That is, if the maximum height HT desired to prevent disappearance is determined, it means that the application range of the disappearance prevention process can be determined. In the present embodiment, “disappearance of the projected image of the object OBJa” refers to the information amount (for example, the object OBJa projected by the projected images PRJa1 and PRJa2) in the output image after the disappearance prevention processing, based on the two projected images PRJa1 and PRJa2. (Region) means a state where the amount of information by one of the two input images (for example, the region of the object OBJa on which one of the two input images is projected) is smaller.

  Therefore, the image generating apparatus 100 determines whether or not to perform the disappearance prevention process on one coordinate on the plane area R1 based on the positional relationship between the two coordinates on the plane area R1a corresponding to one coordinate on the plane area R1. To decide.

  Specifically, the image generating apparatus 100 derives two points Pa1v and Pa2v on the plane area R1a corresponding to one overlapping point Pa1 (Pa2) on the plane area R1. More specifically, the image generating apparatus 100 has two points on the planar region R1a based on points where the projected images PRJa1 and PRJa2 in the planar region R1 are similar to the projected images PRJa1v and PRJa2v in the planar region R1a, respectively. Pa1v and Pa2v are derived. Then, the image generation apparatus 100 determines whether the two points Pa1v and Pa2v are on the right side or the left side of the line BLv. Then, the image generating apparatus 100 has one overlapping point Pa1 (Pa2) on the plane region R1 when one of the two points Pa1v and Pa2v exists on the right side of the line BLv and the other exists on the left side of the line BLv. ) To prevent loss. In other words, the image generation apparatus 100 determines that the planar area when the two points Pa1v and Pa2v on the planar area R1a corresponding to one overlapping point Pa1 (Pa2) on the planar area R1 straddle both sides of the line BLv. A loss prevention process is performed on one overlapping point Pa1 (Pa2) on R1.

  In B1 of FIG. 21, since the point Pa1v exists on the right side of the line BLv and the point Pa2v exists on the left side of the line BLv, the image generating apparatus 100 disappears with respect to the overlapping point Pa1 (Pa2) on the plane region R1. Apply prevention treatment. Specifically, the image generation device 100 executes the process in step S13 of FIG. 15 and selects a camera that is associated with the overlapping point Pa1 (Pa2) on the planar region R1.

  Similarly, the image generation device 100 derives two points Pb1v and Pb2v on the plane region R1a corresponding to one overlapping point Pb1 (Pb2) on the plane region R1. Then, the image generating apparatus 100 determines whether the two points Pb1v and Pb2v are on the right side or the left side of the line BLv. Then, the image generation apparatus 100 omits the disappearance prevention process for one point Pb1 (Pb2) on the plane region R1 when both of the two points Pb1v and Pb2v exist on the same side of the line BLv.

  In B1 of FIG. 21, since the point Pb1v exists on the right side of the line BLv and the point Pb2v also exists on the right side of the line BLv, the image generating apparatus 100 disappears with respect to one overlapping point Pb1 (Pb2) on the plane region R1. The prevention process is omitted. Specifically, the image generation device 100 selects the rear camera 2B having a smaller incident angle as a camera associated with one overlapping point Pb1 (Pb2) on the planar region R1.

  Next, with reference to FIG. 22, the relationship between the overlapping output portion in the first plane region and the overlapping output portion in the second plane region will be described. FIG. 22 is a schematic diagram showing the relationship between the overlapping output portion R15L in the first planar region R1 and the overlapping output portion R15Lv in the second planar region R1a.

  As illustrated in FIG. 22, the overlapping output portion R15L includes a portion R15L-1 (right-downward hatched portion) and a portion R15L-2 (left-downward hatched portion) that are determined not to apply the erasure prevention process, and an erasure. A portion R15L-3 (dot hatched portion) determined to apply the prevention process is included. In addition, the overlapping output portion R15Lv includes a left portion R15Lv-1 (a right-down oblique hatching portion) of the line BLv and a right portion R15Lv-2 (a left-down oblique hatching portion) of the line BLv. Further, the part R15L-1 and the part R15Lv-1 are similar to each other with the position of the rear camera 2B as the center of similarity, and the part R15L-2 and the part R15Lv-2 have the position of the left camera 2L as the center of similarity. They are similar to each other. Note that the overlapping output portion R15Lv does not have a portion corresponding to the portion R15L-3 of the overlapping output portion R15L.

  In this embodiment, one coordinate PP1 in the portion R15L-1 of the plane region R1 corresponds to two coordinates PP11 and PP12 in the portion R15Lv-1 of the plane region R1a. Specifically, the coordinates PP1 and the coordinates PP11 are similar to each other with the position of the left camera 2L as the center of similarity, and the coordinates PP1 and PP12 are similar to each other with the position of the rear camera 2B as the center of similarity. In position. The two coordinates PP11 and PP12 are on the left side of the line BLv. For this reason, the disappearance prevention process is not applied to the portion R15L-1 of the planar region R1. One coordinate PP1 in the portion R15L-1 of the plane region R1 is one coordinate (not shown) on the input image plane of the left camera 2L and one coordinate (not shown) on the input image plane of the rear camera 2B. Z)). Further, two coordinates PP11 and PP12 in the portion R15Lv-1 of the plane region R1a are respectively one coordinate PP1 in the portion R15L-1 of the plane region R1 and one coordinate (not shown) in the input image plane of the left side camera 2L. ) And one coordinate (not shown) in the input image plane of the rear camera 2B.

  Similarly, the one coordinate PP2 in the portion R15L-2 of the plane region R1 corresponds to the two coordinates PP21 and PP22 in the portion R15Lv-2 of the plane region R1a. Specifically, the coordinates PP2 and PP21 are similar to each other with the position of the left camera 2L as the similar center, and the coordinates PP2 and PP22 are similar to each other with the position of the rear camera 2B as the similar center. In position. The two coordinates PP21 and PP22 are both on the right side of the line BLv. Therefore, the disappearance prevention process is not applied to the portion R15L-2 of the planar region R1. One coordinate PP2 in the portion R15L-2 of the plane area R1 is one coordinate (not shown) on the input image plane of the left camera 2L and one coordinate (not shown) on the input image plane of the rear camera 2B. Z)). Further, two coordinates PP21 and PP22 in the portion R15Lv-2 of the plane region R1a are respectively one coordinate PP2 in the portion R15L-2 of the plane region R1 and one coordinate (not shown) on the input image plane of the left camera 2L. ) And one coordinate (not shown) in the input image plane of the rear camera 2B.

  On the other hand, one coordinate PP3 in the portion R15L-3 of the planar region R1 corresponds to one coordinate PP31 in the portion R15Lv-1 of the planar region R1a and one coordinate PP32 in the portion R15Lv-2 of the planar region R1a. Specifically, the coordinates PP3 and PP31 are similar to each other with the position of the left camera 2L as the similar center, and the coordinates PP3 and PP32 are similar to each other with the position of the rear camera 2B as the similar center. In position. The coordinate PP31 exists on the left side of the line BLv, and the coordinate PP32 exists on the right side of the line BLv. For this reason, the disappearance prevention process is applied to the portion R15L-3 of the planar region R1. One coordinate PP3 in the portion R15L-3 of the plane region R1 is one coordinate (not shown) in the input image plane of the left camera 2L and one coordinate (not shown) in the input image plane of the rear camera 2B. Z)). Further, two coordinates PP31 and PP32 in the portion R15Lv-1 of the plane region R1a are one coordinate PP3 in the portion R15L-3 of the plane region R1 and one coordinate (not shown) in the input image plane of the left side camera 2L. , And one coordinate (not shown) in the input image plane of the rear camera 2B.

  Next, the flow of the second disappearance prevention process application range adjustment process will be described with reference to FIG. FIG. 23 is a flowchart showing the flow of the second disappearance prevention process application range adjustment process. For example, the image generation apparatus 100 executes the second disappearance prevention process application range adjustment process while sequentially setting all the pixels on the plane region R1 as the target pixel.

  First, the image generating apparatus 100 determines whether or not a pixel corresponding to the target pixel exists on the input image of the first camera (step S21). In the present embodiment, the image generation apparatus 100 refers to the input image / space model correspondence map 40, and a pixel corresponding to one pixel (target pixel) on the planar region R1 exists on the input image of the left-side camera 2L. It is determined whether or not.

  Thereafter, when it is determined that a pixel corresponding to the target pixel exists on the input image of the first camera (YES in step S21), the image generation device 100 determines that the pixel corresponding to the target pixel is on the input image of the second camera. (Step S22). In this embodiment, when it is determined that the pixel corresponding to the target pixel is present on the input image of the left-side camera 2L, the image generation apparatus 100 refers to the input image / space model correspondence map 40 and corresponds to the target pixel. It is determined whether or not the pixel to be present exists on the input image of the rear camera 2B.

  Thereafter, when it is determined that the pixel corresponding to the target pixel exists on the input image plane of the second camera (YES in step S22), the image generation device 100 executes a loss prevention process necessity determination process described later (step S22). S23). In the present embodiment, the image generating apparatus 100 executes a loss prevention process necessity determination process, which will be described later, when it is determined that the pixel corresponding to the target pixel exists on the input images of both the rear camera 2B and the left camera 2L. To do.

  On the other hand, when it is determined that the pixel corresponding to the target pixel does not exist on the input image of the second camera (NO in step S22), the image generation device 100 associates the pixel on the input image of the first camera with the target pixel. (Step S24). In the present embodiment, the image generating apparatus 100, when the pixel corresponding to the target pixel does not exist on the input image of the rear camera 2B and exists only on the input image of the left camera 2L, the input image of the left camera 2L. The upper pixel is associated with the target pixel.

  Also, when the image generating apparatus 100 determines that the pixel corresponding to the target pixel does not exist on the input image of the first camera (NO in step S21), the pixel corresponding to the target pixel is the second camera. It is determined whether or not the image exists on the input image plane (step S25). In this embodiment, even when the image generating apparatus 100 determines that the pixel corresponding to the target pixel does not exist on the input image of the left camera 2L, the pixel corresponding to the target pixel is the input image of the rear camera 2B. Determine if it exists above.

  After that, when it is determined that the pixel corresponding to the target pixel exists on the input image of the second camera (YES in step S25), the image generation apparatus 100 associates the pixel on the input image of the second camera with the target pixel ( Step S26). In the present embodiment, the image generating apparatus 100 determines that the pixel corresponding to the target pixel does not exist on the input image of the left camera 2L but exists only on the input image of the rear camera 2B. Are associated with the target pixel.

  On the other hand, when the image generation device 100 determines that the pixel corresponding to the target pixel does not exist on the input image of the second camera (NO in step S25), the image generation device 100 sets the target pixel as a non-display pixel. Set (step S27). In this embodiment, when the pixel corresponding to the target pixel does not exist on any input image of the rear camera 2B and the left side camera 2L, the image generation apparatus 100 sets the target pixel as a non-display pixel. For example, the image generating apparatus 100 sets the value of the target pixel to a predetermined value (a value corresponding to black).

  Next, the disappearance prevention process necessity determination process in step S23 will be described with reference to FIG. FIG. 24 is a flowchart showing the flow of the determination process necessity determination process.

  First, the image generating apparatus 100 derives the first coordinate and the second coordinate on the second plane area corresponding to the coordinate of the target pixel on the first plane area (step S31). In the present embodiment, the image generating apparatus 100 includes a first coordinate (PP11, PP21, or PP31) and a second coordinate on the plane region R1a corresponding to the coordinate (PP1, PP2, or PP3) of the target pixel on the plane region R1. The coordinates (PP12, PP22, or PP32) are derived (see FIG. 22). Note that the coordinates PP1, PP2, PP3 of the pixel of interest and the first coordinates PP11, PP21, PP31 are at similar positions with the position of the left camera 2L as the center of similarity, and the coordinates PP1, PP2, PP3 of the pixel of interest And the second coordinates PP12, PP22, PP32 are in similar positions with the position of the rear camera 2B as the similar center. The description here relates to the overlapping area of the imaging ranges of the left camera 2L and the rear camera 2B, but the same applies to the overlapping area of the imaging ranges of the right camera 2R and the rear camera 2B.

  Thereafter, the image generating apparatus 100 determines whether or not the first coordinates are on the first camera side of the boundary line (step S32). In the present embodiment, the image generating apparatus 100 determines whether or not the first coordinate is on the left camera 2L side of the boundary line BLv (see FIG. 22).

  Here, an example of a method for setting the boundary line BLv will be described with reference to FIG. FIG. 25 is a top view of the excavator 60, and a line segment 2BG that is a projection image onto the plane region R1a including the optical axis of the rear camera 2B, and a line segment including the optical axis of the left-side camera 2L. A line segment 2LG that is a projection image onto the plane area R1a and a boundary line BLv set on the plane area R1a are shown. The pixel PP0 is a coordinate on the plane region R1a corresponding to the midpoint of the line segment connecting the left camera 2L and the rear camera 2B.

  When the angle δ is formed between the line segment 2BG and the line segment 2LG, the image generating apparatus 100 sets the boundary line BLv so that the pixel PP0 is the starting point and the angle bisector of the angle δ is formed. To do. In this way, the image generating apparatus 100 can set the boundary line BLv in advance based on the camera installation conditions. In addition, although illustration is abbreviate | omitted, the boundary line relevant to the overlap area | region of the imaging range of the right side camera 2R and the rear camera 2B is also set similarly.

  Here, with reference to FIG. 24 again, the description of the loss prevention process necessity determination process is continued.

  If it is determined that the first coordinate is on the first camera side of the boundary line (YES in step S32), the image generating apparatus 100 determines whether the second coordinate is on the second camera side of the boundary line (step S32). S33). In this embodiment, when the image generating apparatus 100 determines that the first coordinate PP31 is on the left side camera 2L side of the boundary line BLv, it determines whether or not the second coordinate PP32 is on the rear camera 2B side of the boundary line BLv. judge.

  Thereafter, when it is determined that the second coordinates are on the second camera side of the boundary line (YES in step S33), the image generation device 100 determines that the disappearance prevention process is necessary, and executes, for example, the process in step S13 of FIG. Then, the camera (pixel on the input image) associated with the target pixel is selected (step S34). In this embodiment, when the image generating apparatus 100 determines that the first coordinate PP31 is on the left camera 2L side of the boundary line BLv and the second coordinate PP32 is on the rear camera 2B side of the boundary line BLv, With respect to the target pixel represented by PP3, it is determined that the erasure prevention process is necessary, and a camera (pixel on the input image) associated with the target pixel is selected.

  On the other hand, if it is determined that the second coordinate is not on the second camera side of the boundary line (NO in step S33), the image generation device 100 determines that the disappearance prevention process is unnecessary, and determines the pixel on the input image of the first camera. Corresponding to the target pixel (step S35). In this embodiment, when the image generating apparatus 100 determines that the first coordinate PP11 is on the left camera 2L side of the boundary line BLv, but the second coordinate PP12 is not on the rear camera 2B side of the boundary line BLv, the coordinate PP1 It is determined that the loss prevention process is unnecessary for the target pixel represented by That is, if the image generating apparatus 100 determines that both the first coordinate PP11 and the second coordinate PP12 are on the left camera 2L side of the boundary line BLv, the image generation apparatus 100 does not need the erasure prevention process for the target pixel represented by the coordinate PP1. And the pixel on the input image of the left side camera 2L is associated with the target pixel.

  If it is determined that the first coordinate is not on the first camera side of the boundary line (NO in step S32), the image generating apparatus 100 determines whether the second coordinate is on the second camera side of the boundary line. Without the loss prevention processing, the coordinates of the input image plane of the second camera are associated with the target pixel (step S36). This is because when the first coordinates are not on the first camera side of the boundary line, it is not possible to geometrically cause the second coordinates to be on the first camera side of the boundary line. In the present embodiment, when the image generating apparatus 100 determines that the first coordinate PP21 is not on the left side camera 2L side of the boundary line BLv, it determines whether or not the second coordinate PP22 is on the rear camera 2B side of the boundary line BLv. For the target pixel represented by the coordinates PP2 without determination, it is determined that the erasure prevention processing is unnecessary, and the coordinates of the input image plane of the rear camera 2B are associated with the target pixel PP2. This is because when the first coordinate PP21 is not on the left side camera 2L side of the boundary line BLv, the second coordinate PP22 cannot geometrically occur on the left side camera 2L side of the boundary line.

  Next, effects of the disappearance prevention process application range adjustment process will be described with reference to FIG. Note that the upper part of FIG. 26 is a diagram showing an output image when the entire output image part corresponding to the overlapping region is subjected to the disappearance prevention process, and corresponds to the upper part of FIG. The middle part of FIG. 26 is a diagram showing an output image when the first disappearance prevention process application range adjustment process is executed, and corresponds to the lower part of FIG. Further, the lower part of FIG. 26 is a diagram illustrating an output image when the second disappearance prevention process application range adjustment process is executed. Note that regions R15Lb and R15Rb surrounded by broken lines in the lower part of FIG. 26 represent the disappearance prevention process application range after the adjustment by the second disappearance prevention process application range adjustment process.

  As can be seen from the comparison of the three output images in FIG. 26, the image generation layer 100 executes the first disappearance prevention process application range adjustment process to prevent the disappearance of the projected image regardless of the height of the object. However, the application range of the loss prevention treatment can be narrowed. Further, the image generation layer 100 performs the second disappearance prevention process application range adjustment process using the planar region R1a having the predetermined height HT, thereby eliminating the disappearance of the projected image of the object having the height less than the predetermined height HT. It is possible to narrow the application range of the disappearance prevention treatment while preventing it.

  Next, with reference to FIG. 27, another example of the disappearance prevention process application range adjustment process (hereinafter referred to as “third disappearance prevention process application range adjustment process”) will be described. FIG. 27 is a flowchart showing the flow of the third disappearance prevention process application range adjustment process. The third disappearance prevention process application range adjustment process is a point in which the value of each pixel in the output image and the value of each pixel in the input image are associated with each other to generate the output image in that the disappearance prevention process application range is determined in advance. This is different from the second disappearance prevention process application range adjustment process for determining whether to apply the disappearance prevention process every time.

  First, the image generating apparatus 100 derives a first overlap output portion of the first plane area and a second overlap output portion of the second plane area (Step S41). In the present embodiment, the image generation apparatus 100 refers to the input image / space model correspondence map 40, and as the first overlapping output portion of the first planar region, the overlapping output portion in the planar region R1 having a height of 0 cm in the spatial model MD. R15L and R15R are derived (see FIG. 20). Further, the image generation apparatus 100 derives an overlapping output portion R15Lv (see B1 in FIG. 21) and R15Rv (not shown) in the planar region R1a having the height HT as the second overlapping output portion of the second planar region. .

  Thereafter, the image generating apparatus 100 divides the second overlapping output portion into a first camera corresponding portion and a second camera corresponding portion (step S42). In the present embodiment, the image generation apparatus 100 divides the overlapping output portion R15Lv of the planar region R1a into a portion R15Lv-1 as a first camera corresponding portion and a portion R15Lv-2 as a second camera corresponding portion (FIG. 22). The coordinates in the portion R15Lv-1 are coordinates that satisfy the condition that the camera with the smallest incident angle becomes the left-side camera 2L. The portion R15Lv-1 is a portion to which the input image of the left-side camera 2L is associated. Is set. The coordinates in the portion R15Lv-2 are coordinates that satisfy the condition that the camera with the smallest incident angle becomes the rear camera 2B, and the portion R15Lv-2 is set as a portion to which the input image of the rear camera 2B is associated. The However, since the process related to the planar area R1a is an intermediate process for adjusting the application range of the disappearance prevention process, it is not necessary to actually associate the input image of the camera 2 with the planar area R1a. Similarly to the overlapping output portion R15Lv, the image generating apparatus 100 divides the overlapping output portion R15Rv of the planar region R1a into a portion R15Rv-1 and a portion R15Rv-2 (both not shown).

  Thereafter, the image generating apparatus 100 derives the disappearance prevention process application range and the disappearance prevention process non-application range in the first overlapping output portion based on the first camera correspondence portion and the second camera correspondence portion (step S43). In the present embodiment, the image generating apparatus 100 derives the portions R15L-1 and R15L-2 in the planar region R1 corresponding to the portions R15Lv-1 and R15Lv-2 in the planar region R1a as the disappearance prevention process non-applicable range ( (See FIG. 22.) Note that the part R15L-1 and the part R15Lv-1 are similar to each other with the position of the camera 2B as the center of similarity, and the part R15L-2 and the part R15Lv-2 are similar to each other with the position of the camera 2L as the center of similarity. is there. Then, the image generation apparatus 100 derives the portion R15L-3 that is not included in any of the portion R15L-1 and the portion R15L-2 among the overlapping output portions R15L as the disappearance prevention process application range. In addition, the image generating apparatus 100, like the portions R15L-1 and R15L-2 in the plane region R1, has portions R15R-1 and R15R-1 in the plane region R1 corresponding to the portions R15Rv-1 and R15Rv-2 in the plane region R1a, respectively. R15R-2 (none of which is shown) is derived as the non-applicable range of the loss prevention process. Then, the image generating apparatus 100 derives a portion R15R-3 (not shown) that is not included in any of the portion R15R-1 and the portion R15R-2 among the overlapping output portions R15R as the disappearance prevention process application range.

  As described above, the image generation apparatus 100 determines the disappearance prevention process application range before starting the generation of the output image. In addition, the image generating apparatus 100 executes, for example, the process in step S13 of FIG. 15 for each coordinate included in the disappearance prevention process application range, and adds a camera associated with each coordinate included in the disappearance prevention process application range. select.

  In the above-described embodiment, the image generation apparatus 100 performs the disappearance prevention process using a lattice pattern, but the disappearance prevention process uses a pattern other than the lattice pattern, such as a comb pattern or a stripe pattern. May be executed. In addition, the image generation apparatus 100 uses an average value of pixel values (for example, luminance values, chromaticity values, etc.) of points (pixels) in two corresponding input images as the pixel value of the output image. May be executed as a loss prevention process.

  In the above-described embodiment, as long as the image generation apparatus 100 is included in the overlap region, the image generation apparatus 100 performs the disappearance prevention process on the image projected on the curved surface region R2. The disappearance prevention process may be omitted for the image projected on the curved surface area R2. An object existing outside the spatial model MD (plane area R1) is not likely to disappear even if the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera having the smallest incident angle. It is.

  In addition, the image generating apparatus 100 determines whether one point on the plane region R1 is present depending on whether two points on the plane region R1a corresponding to one point on the plane region R1 exist across both sides of the line BLv. It is determined whether or not the disappearance prevention process is performed on two overlapping points. However, the present invention is not limited to this configuration. For example, instead of the line BLv formed by a set of points at which the incident angle with respect to the left camera 2L and the incident angle with respect to the rear camera 2B are equal, the image generating apparatus 100 uses the broken line region R15Lv in the plane region R1a as Any line that bisects in the radial direction may be used. In this case, the arbitrary line may be a straight line, a curved line, or a combination thereof.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 employs the cylindrical spatial model MD as the spatial model, but may employ a spatial model having other columnar shapes such as a polygonal column, A spatial model composed of two side surfaces may be adopted, or a spatial model having only side surfaces may be adopted.

  The image generating apparatus 100 is mounted with a camera on a self-propelled construction machine having movable members such as a bucket, an arm, a boom, and a turning mechanism. The image generating apparatus 100 moves and moves the construction machine while presenting a surrounding image to the driver. It is built in an operation support system that supports the operation of these movable members, but it is mounted with other cameras that have movable members such as industrial machines or fixed cranes but do not self-propelled together with the camera. You may incorporate in the operation assistance system which assists operation of another to-be-operated body.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 10 ... -Coordinate association means 11-Output image generation means 40-Input image / spatial model correspondence map 41-Spatial model-processing target image correspondence map 42-Processing target image / output image correspondence map 60- .... Excavator 61 ... Lower traveling body 62 ... Turning mechanism 63 ... Upper turning body 64 ... Cab

Claims (7)

An image generation device that generates an output image based on a plurality of input images captured by a plurality of cameras attached to an object to be operated,
Coordinate associating means for associating coordinates in a space model arranged around the object to be operated with coordinates in an input image plane on which the input image is located;
Output image generation means for generating the output image by associating the coordinate value on the input image plane with the coordinate value on the output image plane on which the output image is located via the coordinates in the spatial model. ,
The spatial model has a first plane area including a first overlapping output portion capable of associating coordinates on input image planes of two cameras of the plurality of cameras,
The image generation device includes a second overlapping output portion having a predetermined height different from the first plane area and including a second overlapping output portion capable of associating coordinates in the input image planes of the two cameras. Set the plane area,
The second overlapping output portion corresponds to a first camera corresponding portion that associates coordinates in one input image plane of the two cameras and a second camera that associates coordinates in the other input image plane of the two cameras. It consists of a corresponding part,
The image generation device generates an output image so that a portion corresponding to each of the first camera corresponding portion and the second camera corresponding portion in the first overlapping region is a non-applicable range of the loss prevention processing;
Image generation device. The image generation device determines whether to perform the disappearance prevention process on one coordinate based on the position of two coordinates in the second plane region corresponding to one coordinate in the first overlapping region.
The image generation apparatus according to claim 1. When the two coordinates in the second plane region exist across both sides of a predetermined line that separates the first camera corresponding part and the second camera corresponding part in the second plane region, Applying a loss prevention process to the one coordinate,
The image generation apparatus according to claim 2. At any point on the predetermined line, the incident angles for each of the two cameras are equal,
The image generation apparatus according to claim 3. The image generation device sets a portion corresponding to each of the first camera corresponding portion and the second camera corresponding portion in the first overlapping region as a non-applicable range of the loss prevention processing in advance.
The image generation apparatus according to claim 1. The loss prevention process includes a process using a lattice pattern, a process using a stripe pattern, or an averaging process.
The image generation apparatus according to any one of claims 1 to 5. An operation support system that supports movement or operation of an object to be operated,
An image generation device according to any one of claims 1 to 6,
A display unit installed in an operation room for moving or operating the object to be operated and displaying an output image generated by the image generation device;
An operation support system comprising: