JP5960007B2 - Perimeter monitoring equipment for work machines - Google Patents

Description

  The present invention relates to a work machine periphery monitoring device that generates an output image based on an input image captured by a camera attached to an object to be operated.

  2. Description of the Related Art Conventionally, there has been known an ambient monitoring device that monitors the surroundings while acquiring the distance and direction of an object existing around the excavator using a laser radar attached to the upper revolving unit of the excavator (for example, Patent Documents). 1).

  The surrounding monitoring apparatus determines that an object is a stationary obstacle when the distance and direction of the object to the excavator do not change over a predetermined time, and determines that the object is a moving obstacle when the distance and direction change. Then, the surroundings monitoring device generates an illustration image of the excavator and the surroundings viewed from above, and displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. In addition, the surrounding monitoring device displays the stationary obstacle and the moving obstacle with different illustrations.

  In this way, the surrounding monitoring device can inform the driver of the state of obstacles around the excavator in an easy-to-understand manner.

JP 2008-163719 A

  However, the output image presented by the surrounding monitoring device described in Patent Document 1 only displays an illustration representing the obstacle at a position on the illustration image corresponding to the actual position of the obstacle. Is not displayed, so it is still insufficient to convey the situation of obstacles around the excavator to the driver in an easy-to-understand manner.

  In view of the above-described points, an object of the present invention is to provide a work machine periphery monitoring device that can easily convey the situation of surrounding objects to an operator.

  To achieve the above object, a work machine periphery monitoring apparatus according to an embodiment of the present invention is a work machine periphery monitoring apparatus that generates an output image based on an input image captured by a camera attached to the work machine. An object detection sensor that detects an object that is attached to the work machine and exists around the work machine, and an object detection sensor that detects the object when the object is detected by the object detection sensor is identified. And highlighting means for highlighting a predetermined area of the output image according to the identification result.

  With the above-described means, the present invention can provide a work machine periphery monitoring device that can easily convey the situation of surrounding objects to the operator.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of the shovel mounted with an image generation apparatus. It is a figure which shows an example of the space model on which an input image is projected. It is a figure which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is an example of a display of an output image. It is a top view of the shovel in which an image generation apparatus is mounted. It is a figure which shows each input image of the three cameras mounted in the shovel, and the output image produced | generated using those input images. It is a figure for demonstrating the image loss prevention process which prevents the loss | disappearance of the object in the overlapping imaging area of each imaging area of two cameras. FIG. 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying image loss prevention processing to the output image of FIG. 12. It is a figure explaining a predetermined area selection process. It is a flowchart which shows the flow of an emphasis display process. It is an example of a display of the output image containing the predetermined area | region highlighted.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

  The image generation apparatus 100 is an example of a work machine periphery monitoring apparatus that monitors the periphery of the work machine. The image generation apparatus 100 includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, and an object detection sensor 6. Composed. Specifically, the image generation device 100 generates an output image based on an input image captured by the camera 2 mounted on the work machine and presents the output image to the operator.

  FIG. 2 is a diagram illustrating a configuration example of an excavator 60 as a work machine on which the image generating apparatus 100 is mounted. The excavator 60 is placed on a crawler-type lower traveling body 61 via a turning mechanism 62. A swing body 63 is mounted so as to be swingable around a swing axis PV.

  The upper swing body 63 includes a cab (operator's cab) 64 on the front left side, a drilling attachment E on the front center, and the camera 2 (right camera 2R, rear camera 2B) on the right and rear surfaces. ) And an object detection sensor 6 (right side object detection sensor 6R, rear object detection sensor 6B). In addition, the display part 5 is installed in the position in the cab 64 where the operator is easy to visually recognize.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), etc. A program corresponding to each of the attaching unit 10, the image generating unit 11, and the highlighting unit 12 is stored in a ROM or NVRAM, and the CPU executes a process corresponding to each unit while using the RAM as a temporary storage area.

  The camera 2 is a device for acquiring an input image that reflects the surroundings of the excavator 60, and is attached to the right side surface and the rear surface of the upper swing body 63 so that, for example, an area that is a blind spot of the operator in the cab 64 can be imaged. (See FIG. 2), a right side camera 2R and a rear camera 2B provided with an image sensor such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens or a fisheye lens is attached so as to capture a wide range. It may be.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Although it is output, an input image in which the apparent distortion or tilt is not corrected may be output to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display, a projector, or the like installed in the cab 64 (see FIG. 2) of the excavator 60. Display an image.

  The object detection sensor 6 is a device for detecting an object existing around the excavator 60. For example, the object detection sensor 6 is located on the right side of the upper swing body 63 so as to detect an object present in the blind spot of the operator in the cab 64. Attached to the front and back surfaces (see FIG. 2). The object detection sensor 6 may be attached to any one or more of the front, left side, right side, and rear side of the upper swing body 63, or may be attached to all the surfaces.

  In this embodiment, the object detection sensor 6 is an omnidirectional sensor, such as an omnidirectional ultrasonic sensor, an omnidirectional pyroelectric infrared sensor, or the like. Specifically, the object detection sensor 6 radiates ultrasonic waves or infrared light uniformly over the entire detection range, and detects an object existing in the detection range by detecting a reflected wave or reflected light. Further, the object detection sensor 6 may detect the distance to the detected object based on the elapsed time from when the ultrasonic wave or infrared light is emitted until the reflected wave or the reflected light is detected.

  Note that the object detection sensor 6 may be a leaky coaxial cable installed around the excavator 60. In this case, the detection range is formed corresponding to each of the plurality of slots formed in the outer conductor portion.

  In addition, the image generation apparatus 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding objects and a sense of distance can be intuitively grasped. An output image to be generated may be generated, and the output image may be presented to the operator.

  The “processing target image” is an image that is generated based on an input image and that is a target of image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.). When an input image including a horizontal image (for example, an empty portion) is used in an image conversion process with an image captured from above by a camera that captures the image from above, the horizontal image is The input image is projected onto a predetermined spatial model so that it is not unnaturally displayed (for example, the sky part is not treated as being on the ground surface), and then the projected image projected onto the spatial model is changed to another two. It is an image suitable for image conversion processing, which is obtained by reprojecting onto a dimensional plane. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “spatial model” is a plane or curved surface other than the processing target image plane that is the plane on which the processing target image is located (for example, a plane parallel to the processing target image plane or an angle with the processing target image plane). A plane or a curved surface to be formed), and a projection target of an input image composed of one or a plurality of planes or curved surfaces.

  Note that the image generation apparatus 100 may generate an output image by performing image conversion processing on the projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image as it is without being subjected to image conversion processing.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected. FIG. 3A illustrates a relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. FIG. 3B shows the relationship between the excavator 60 and the space model MD when the excavator 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and includes a planar region R1 inside the bottom surface and a curved region R2 inside the side surface.

  FIG. 4 is a diagram illustrating an example of a relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is a plane including the plane area R1 of the space model MD, for example. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for the sake of clarity, the space model MD may be either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the subsequent drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the spatial model MD, or may be an annular area not including the plane area R1 of the spatial model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate association means 10 is a means for associating coordinates on the input image plane where the input image captured by the camera 2 is located, coordinates on the space model MD, and coordinates on the processing target image plane R3. For example, various parameters relating to the camera 2 such as optical center, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc., which are set in advance or input via the input unit 3 And the coordinates on the input image plane, the coordinates on the space model MD, and the processing target image based on the predetermined positional relationship among the input image plane, the spatial model MD, and the processing target image plane R3. The coordinates on the plane R3 are associated with each other, and the corresponding relationship is stored in the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 of the storage unit 4. .

  When the processing target image is not generated, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

  The image generation unit 11 is a unit for generating an output image. For example, by performing scale conversion, affine conversion, or distortion conversion on the processing target image, the coordinates on the processing target image plane R3 and the output image are positioned. Corresponding to the coordinates on the output image plane to be processed, the correspondence relationship is stored in the processing target image / output image correspondence map 42 of the storage unit 4, and the coordinate correspondence means 10 stores the value corresponding to the input image / spatial model With reference to the map 40 and the spatial model / processing target image correspondence map 41, the value of each pixel in the output image (for example, the luminance value, hue value, saturation value, etc.) and the value of each pixel in the input image To generate an output image.

  Further, the image generation means 11 is preset or input via the input unit 3, the optical center of the virtual camera, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, etc. The coordinates on the processing target image plane R3 and the coordinates on the output image plane on which the output image is located are associated with each other on the basis of the various parameters, and the corresponding relationship is stored in the processing target image / output image correspondence map 42 of the storage unit 4. With reference to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 stored and stored in the coordinate matching means 10, the value of each pixel (for example, luminance value, A hue value, a saturation value, etc.) and the value of each pixel in the input image are associated with each other to generate an output image.

  Note that the image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the image generation unit 11 does not generate the processing target image, the image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane in accordance with the applied image conversion processing, and supports the input image / space model. While referring to the map 40, the output image is generated by associating the value of each pixel in the output image (for example, luminance value, hue value, saturation value, etc.) with the value of each pixel in the input image. In this case, the image generation unit 11 omits the association between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence relationship in the processing target image / output image correspondence map 42. .

  The highlighting means 12 is a means for highlighting a predetermined area of the output image. In the present embodiment, when the object detection sensor 6 detects an object, the highlighting display unit 12 specifies the object detection sensor 6 that has detected the object, and emphasizes a predetermined region of the output image according to the specification result. indicate.

  The “predetermined region” is one or a plurality of regions set in advance on the output image, and is a region set based on the detection range of the object detection sensor 6, for example. In the present embodiment, each of the plurality of predetermined areas is set based on the detection range of each of the plurality of object detection sensors 6. For example, one of the plurality of predetermined regions corresponds to an overlapping portion of the detection ranges of the two adjacent object detection sensors 6. Further, another one of the plurality of predetermined regions corresponds to a specific detection range of the plurality of object detection sensors 6. Further, another one of the plurality of predetermined regions is a region obtained by subtracting an overlapping portion of the detection ranges of the two object detection sensors 6 from one detection range of the two adjacent object detection sensors 6. Correspond.

  The predetermined area is set as a fan-shaped area that spreads radially around the position of the excavator 60 when the output image is a viewpoint conversion image in which the periphery of the excavator 60 is looked down on.

  In this embodiment, the highlighting means 12 identifies which object detection sensor 6 has detected an object based on the output of each of the plurality of object detection sensors 6.

  The highlighting means 12 selects one or more predetermined areas to be highlighted from one or more predetermined areas according to the identification result. Specifically, the highlighting means 12 selects one or a plurality of predetermined areas to be highlighted according to the combination of the object detection sensors 6 that have detected the object. For example, the highlighting means 12 reads one or a plurality of predetermined areas to be highlighted by reading the identification number of the predetermined area stored in advance in a ROM or the like in association with the combination of the object detection sensors 6 that detected the object. select.

  The highlighting means 12 highlights the selected one or more predetermined areas. In this embodiment, the highlighting means 12 superimposes and displays the outline of the selected predetermined area on the output image. Further, the highlighting means 12 may display a warning mark at a position corresponding to the selected predetermined area. Alternatively, the highlighting means 12 may make at least one of the luminance and the color tone of the entire selected predetermined region different from the non-selected predetermined region.

  Next, an example of specific processing by the coordinate association unit 10 and the image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 has UVW orthogonal coordinates with the optical center C of the camera 2 as the origin. The space model is represented as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate association unit 10 converts the coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system), so that the origin of the XYZ coordinate system is optically converted. After moving parallel to the center C (the origin of the UVW coordinate system), the X axis is the U axis, the Y axis is the V axis, and the Z axis is the -W axis (the sign "-" indicates that the direction is reversed) This means that the XYZ coordinate system is rotated so that the UVW coordinate system coincides with the + W direction in front of the camera and the XYZ coordinate system in the −Z direction vertically below.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate association unit 10 uses an XYZ coordinate system in parallel with each of the plurality of UVW coordinate systems. It will be moved and rotated.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. Since it is realized by rotating to coincide with the axis, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this transformation in Hamilton's quaternion. it can.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the surface extending between the vector A and the vector B as an axis. If the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It will be expressed as

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

The conjugate quaternion of the quaternion Q is

It shall be represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. For example, an arbitrary point S (sx, sy, sz) can be expressed as an arbitrary unit vector. A point D (ex, ey, ez) when rotated by an angle θ with C (l, m, n) as an axis can be expressed as follows.

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

It will be expressed as

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , A quaternion R representing "rotation to make the X axis coincide with the U axis"

It will be expressed as

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

Since the quaternion R is invariable in each of the cameras 2, the coordinate association unit 10 thereafter performs the above calculation to obtain the coordinates on the space model (XYZ coordinate system). It can be converted into coordinates on the input image plane (UVW coordinate system).

  After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 determines the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the space. An incident angle α formed by a line segment CP ′ connecting an arbitrary coordinate P on the model with a coordinate P ′ represented in the UVW coordinate system and the optical axis G of the camera 2 is calculated.

  In addition, the coordinate association unit 10 includes an intersection E between the plane H and the optical axis G, and a coordinate P ′ in a plane H that is parallel to the input image plane R4 (for example, CCD plane) of the camera 2 and includes the coordinate P ′. Are calculated, and the deviation angle φ formed by the line segment EP ′ connecting the two and the U ′ axis in the plane H, and the length of the line segment EP ′.

  In the optical system of the camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate matching means 10 performs normal projection (h = ftanα) and orthographic projection (h = fsinα). Image height by selecting an appropriate projection method such as stereo projection (h = 2 ftan (α / 2)), equisolid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc. Calculate h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. By dividing, the coordinates P (P ′) on the space model MD can be associated with the coordinates on the input image plane R4.

  If the pixel size per pixel in the U-axis direction of the input image plane R4 is aU and the pixel size per pixel in the V-axis direction of the input image plane R4 is aV, the coordinates P (P The coordinates (u, v) on the input image plane R4 corresponding to ') are

It will be expressed as

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  Note that, when it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate associating means 10 inputs the coordinates P (P ′) on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with coordinates on the image plane R4, or may be associated with coordinates on the input image plane R4 selected by the operator.

  Next, a process of reprojecting coordinates on the curved surface area R2 (coordinates having a component in the Z-axis direction) among the coordinates on the spatial model MD onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between coordinates by the coordinate association means 10, and FIG. 6A shows an input image plane R4 of the camera 2 that employs a normal projection (h = ftanα) as an example. FIG. 6 is a diagram showing a correspondence relationship between the coordinates on the upper surface and the coordinates on the space model MD, and the coordinate associating means 10 displays the coordinates on the input image plane R4 of the camera 2 and the space model MD corresponding to the coordinates. Both the coordinates are made to correspond to each other so that each of the line segments connecting the coordinates passes through the optical center C of the camera 2.

  In the example of FIG. 6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane area R1 of the space model MD, and inputs the image 2 R4 of the camera 2. The upper coordinate K2 is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  FIG. 6B is a diagram showing a correspondence relationship between coordinates on the curved surface region R2 of the space model MD and coordinates on the processing target image plane R3, and the coordinate association unit 10 is positioned on the XZ plane. The parallel line group PL that forms an angle β with the processing target image plane R3 is introduced, and the coordinates on the curved surface region R2 of the spatial model MD and the processing target image corresponding to the coordinates are introduced. The coordinates on the plane R3 are associated with each other such that the coordinates are on one of the parallel line groups PL.

  In the example of FIG. 6B, the coordinate matching means 10 assumes that the coordinate L2 on the curved surface region R2 of the space model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line, and both coordinates are obtained. Make it correspond.

  The coordinate association means 10 can associate the coordinates on the plane area R1 of the spatial model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface area R2. However, in the example of FIG. 6B, since the plane area R1 and the processing target image plane R3 are a common plane, the coordinates L1 on the plane area R1 of the spatial model MD and the processing target image plane R3 The coordinate M1 has the same coordinate value.

  In this way, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image R3. It is stored in the spatial model / processing object image correspondence map 41.

  FIG. 6C is a diagram illustrating a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V that adopts the normal projection (h = ftanα) as an example. In the image generation means 11, each line segment connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates passes through the optical center CV of the virtual camera 2V. In this way, both coordinates are associated.

  In the example of FIG. 6C, the image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane area R1 of the space model MD), The coordinate N2 on the output image plane R5 of the virtual camera 2V is associated with the coordinate M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  Note that when the virtual camera 2V employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the image generating unit 11 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and associates the coordinates on the output image plane R5 with the coordinates on the processing target image R3. Each pixel in the output image while referring to the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 stored in the processing object image / output image correspondence map 42 and stored by the coordinate association means 10. Is associated with the value of each pixel in the input image to generate an output image.

  FIG. 6D is a combination of FIGS. 6A to 6C. The camera 2, the virtual camera 2V, the plane area R1 and the curved area R2 of the space model MD, and the processing target. The mutual positional relationship of image plane R3 is shown.

  Next, the operation of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 7A is a diagram in the case where the angle β is formed between the parallel line group PL positioned on the XZ plane and the processing target image plane R3, and FIG. 7B is the diagram on the XZ plane. It is a figure in case angle (beta) 1 ((beta) 1> (beta)) is formed between the parallel line group PL and the process target image plane R3. Also, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in FIGS. 7A and 7B corresponds to the coordinates Ma to Md on the processing target image plane R3, respectively. Assume that the intervals between the coordinates La to Ld in FIG. 7A are equal to the intervals between the coordinates La to Ld in FIG. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. It shall be. In this case, the Z axis is referred to as a “reprojection axis”.

  As shown in FIGS. 7A and 7B, the distance between the coordinates Ma to Md on the processing target image plane R3 is such that the angle between the parallel line group PL and the processing target image plane R3 is the same. As it increases, it decreases linearly (it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md). On the other hand, since the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is such that only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD is linearly enlarged or out of the image portion on the output image plane R5 (see FIG. 6). It means to be reduced.

  Next, an alternative example of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  FIG. 8A is a diagram in the case where all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3, and FIG. It is a figure in case all the line groups AL extend toward the process target image plane R3 from the starting point T2 (T2> T1) on the Z axis. 8A and 8B, the coordinates La to Ld on the curved surface region R2 of the spatial model MD correspond to the coordinates Ma to Md on the processing target image plane R3 ( In the example of FIG. 8A, the coordinates Mc and Md are not shown because they are outside the region of the processing target image plane R3.) The intervals between the coordinates La to Ld in FIG. It is assumed that the interval between the coordinates La to Ld in 8 (B) is equal. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the processing target image plane R3. It shall be. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in FIGS. 8A and 8B, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (high) between the starting point of the auxiliary line group AL and the origin O. As the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md is larger, the width of reduction of each interval is larger. On the other hand, since the coordinate group on the plane region R1 of the spatial model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image portion is enlarged or reduced nonlinearly.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and does not affect the space model MD. Since the image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 can be linearly or nonlinearly enlarged or reduced, the road surface image in the vicinity of the excavator 60 Without affecting the excavator 60 (virtual image when viewed from directly above), an object located in the periphery of the excavator 60 (an object in the image when viewed in the horizontal direction from the excavator 60) can be quickly and flexibly The visibility of the blind spot area of the excavator 60 can be improved.

  Next, referring to FIG. 9, the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. Processing to be generated (hereinafter referred to as “output image generation processing”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Further, it is assumed that the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved surface area R2), and the processing target image plane R3 is determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD using the coordinate association unit 10 (step S1).

  Specifically, the coordinate association unit 10 obtains an angle formed between the parallel line group PL and the processing target image plane R3, and the parallel line group PL extending from one coordinate on the processing target image plane R3. One point that intersects the curved surface region R2 of the space model MD is calculated, and a coordinate on the curved surface region R2 corresponding to the calculated point is set to one on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3. The coordinates are derived as coordinates, and the corresponding relationship is stored in the space model / processing object image correspondence map 41. It should be noted that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be a value to be entered.

  In addition, when one coordinate on the processing target image plane R3 matches one coordinate on the plane area R1 of the space model MD, the coordinate association unit 10 uses the one coordinate on the plane area R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing object image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate association unit 10 to associate one coordinate on the spatial model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate association unit 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα), is a line segment extending from one coordinate on the space model MD, and the optical center A point where the line segment passing through C intersects the input image plane R4 is calculated, and the coordinates on the input image plane R4 corresponding to the calculated point are set on the input image plane R4 corresponding to the one coordinate on the space model MD. And the corresponding relationship is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3), and all the coordinates are still set. Are determined to be not associated (NO in step S3), the processes in steps S1 and S2 are repeated.

  On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation process, starts the output image generation process, and the image generation unit 11 The coordinates on the processing target image plane R3 are associated with the coordinates on the output image plane R5 (step S4).

  Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image, and is determined by the content of the applied scale conversion, affine conversion, or distortion conversion. The correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 is stored in the processing target image / output image correspondence map 42.

  Alternatively, when generating the output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, when the output image is generated using the virtual camera 2V that employs normal projection (h = ftanα), the image generation unit 11 obtains the coordinates of the optical center CV of the virtual camera 2V and outputs A line segment extending from one coordinate on the image plane R5, a point where a line segment passing through the optical center CV intersects the processing target image plane R3 is calculated, and the coordinates on the processing target image plane R3 corresponding to the calculated point May be derived as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5, and the correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Thereafter, the control unit 1 uses the image generation unit 11 to refer to the input image plane R4 while referring to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. The correspondence between the coordinates on the space model MD and the coordinates on the space model MD, the correspondence between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the coordinates on the processing target image plane R3 and the output image plane R5 And the values of the coordinates on the input image plane R4 corresponding to the coordinates on the output image plane R5 (for example, luminance values, hue values, saturation values, etc.) are obtained. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). Note that when a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 uses each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value (for example, average value, maximum value, minimum value, intermediate value, etc.) may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. .

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6), and all the coordinate values are still associated. If it is determined that it is not attached (NO in step S6), the processes in steps S4 and S5 are repeated.

  On the other hand, if the control unit 1 determines that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

  Note that, when the processing target image is not generated, the image generation apparatus 100 omits the processing target image generation processing, and sets “coordinates on the processing target image plane” in step S4 in the output image generation processing to “on the spatial model”. It shall be read as "coordinates".

  With the above configuration, the image generation apparatus 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the object around the excavator 60 and the excavator 60.

  Further, the image generation apparatus 100 associates coordinates on the processing target image plane R3 from the processing target image plane R3 to the input image plane R4 through the spatial model MD, thereby obtaining the coordinates on the processing target image plane R3. One or more coordinates on R4 can be reliably associated, and compared with a case where coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD (in this case) Can reliably correspond each coordinate on the input image plane R4 to one or a plurality of coordinates on the processing target image plane R3. However, a part of the coordinates on the processing target image plane R3 is part of the input image plane R4. In some cases, the coordinates may not be associated with any of the above coordinates, and in such a case, it is necessary to perform interpolation processing or the like on a part of the coordinates on the processing target image plane R3). It is possible to rapidly generate.

  Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generating apparatus 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Thus, it is possible to realize a desired enlargement or reduction without rewriting the contents of the input image / space model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the space model / processing object image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters relating to scale conversion, affine transformation, or distortion transformation. The desired output image (scale-converted image, affine-transformed image, or distortion-converted image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing object image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

  The image generation apparatus 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects them onto the processing target image plane R3 to generate a processing target image. Then, an image conversion process (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion process, etc.) is performed on the generated processing target image to generate an output image, and the vicinity of the excavator 60 can be seen from above. An image looking down (image in the plane region R1) and an image looking around in the horizontal direction from the excavator 60 (image in the processing target image plane R3) are simultaneously displayed.

  When the image generation apparatus 100 does not generate the processing target image, the output image is generated by performing image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD. Shall.

  The output image is trimmed in a circle so that the image when the excavator 60 performs the turning motion can be displayed without a sense of incongruity, and the center CTR of the circle is on the cylindrical central axis of the space model MD and the excavator 60 is turned. It is generated so as to be on the axis PV, and is displayed so as to rotate about its center CTR in accordance with the turning operation of the excavator 60. In this case, the cylindrical central axis of the space model MD may or may not coincide with the reprojection axis.

  The radius of the space model MD is, for example, 5 meters, and the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is the turning of the shovel 60. When an object (for example, a worker) exists at a position away from the center by a maximum reachable distance (for example, 12 meters) of the excavation attachment E, the object is sufficiently large (for example, 7) It can be set to be displayed.

  Further, the output image may be a CG image of the excavator 60 arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. This is to make the positional relationship between the shovel 60 and the object appearing in the output image easier to understand. Note that a frame image including various kinds of information such as an orientation may be arranged around the output image.

  Next, another example of an output image generated by the image generation apparatus 100 will be described with reference to FIGS.

  FIG. 11 is a top view of the excavator 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 and a total of six object detection sensors 6 installed on the left and right sides of each of the three cameras 2. The three cameras 2 include a left side camera 2L, a right side camera 2R, and a rear camera 2B. The six object detection sensors 6 include a first left side object detection sensor 6L1, a second left side object detection sensor 6L2, a first right side object detection sensor 6R1, a second right side object detection sensor 6R2, and a first rear object detection. A sensor 6B1 and a second rear object detection sensor 6B2 are included. In addition, areas CL, CR, and CB indicated by alternate long and short dashed lines in FIG. 11 indicate imaging areas of the left camera 2L, the right camera 2R, and the rear camera 2B, respectively. In addition, areas ZL1, ZR1, and ZB1 indicated by thick dotted lines in FIG. 11 indicate detection ranges of the first left side object detection sensor 6L1, the first rear object detection sensor 6B1, and the first right side object detection sensor 6R1, respectively. In addition, areas ZL2, ZR2, and ZB2 indicated by thin dotted lines in FIG. 11 indicate detection ranges of the second left side object detection sensor 6L2, the second rear object detection sensor 6B2, and the second right side object detection sensor 6R2, respectively.

  In the present embodiment, the shapes of the regions ZL1, ZB1, ZR1 and the regions ZL2, ZB2, ZR2 are sector shapes that form part of a circle centered on the pivot axis of the excavator 60 and have the same size. However, the regions ZL1, ZL2, ZB1, ZB2, ZR1, and ZR2 may have different shapes and sizes. For example, the regions ZL1, ZL2, ZB1, ZB2, ZR1, and ZR2 may have different fan-shaped central angles. This is because the smaller the predetermined area, the more precisely the area where the presence of the object is estimated can be narrowed down. In this case, the number, installation positions, and detection areas of the object detection sensors 6 are set according to a desired shape and size of each predetermined area.

  In FIG. 11, for the sake of clarity, the regions ZL1, ZB1, ZR1 and the regions ZL2, ZB2, ZR2 are slightly displaced in the radial direction, but this displacement is not essential.

  In this embodiment, as shown in FIG. 11, the detection range of the object detection sensor 6 is narrower than the imaging area of the camera 2, but may be the same as the imaging area of the camera 2 and wider than the imaging area of the camera 2. Also good. Further, in this embodiment, as shown in FIG. 11, the detection range of the object detection sensor 6 is located in the vicinity of the shovel 60 in the imaging region of the camera 2, but even in a region farther from the shovel 60. Good.

  In the present embodiment, as shown in FIG. 11, the regions ZL1, ZB1, and ZR1 are arranged adjacent to each other so as not to overlap each other. Similarly, the regions ZL2, ZB2, and ZR2 are also arranged adjacent to each other so as not to overlap each other. On the other hand, the region ZL1 and the region ZL2 are arranged so as to partially overlap each other. The same applies to the overlapping relationship between the region ZB1 and the region ZB2 and the overlapping relationship between the region ZR1 and the region ZR2. However, the regions ZL1, ZL2, ZB1, ZB2, ZR1, and ZR2 may be arranged so as not to overlap each other.

  FIG. 12 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images.

  The image generation apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the space model MD, and then reprojects them onto the process target image plane R3 to generate a process target image. . The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generating apparatus 100 includes an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1) and an image in which the periphery is viewed from the shovel 60 in the horizontal direction (an image in the processing target image plane R3). Display at the same time. The image displayed at the center of the output image is a CG image 60CG of the excavator 60.

  In FIG. 12, the input image of the right-side camera 2R and the input image of the rear camera 2B each capture a person in the overlapping imaging region of the imaging region of the right-side camera 2R and the imaging region of the rear camera 2B ( (See a region R10 surrounded by a two-dot chain line in the input image of the right camera 2R and a region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.)

  However, if the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image will cause the person in the overlapping imaging area to disappear (in the output image (Refer area | region R12 enclosed with a dashed-dotted line.).

  Therefore, the image generation apparatus 100 associates the region on the input image plane of the rear camera 2B with the coordinate on the input image plane of the right-side camera 2R in the output image portion corresponding to the overlapping imaging region. To prevent the objects in the overlapping imaging area from disappearing.

  FIG. 13 is a diagram for explaining an image disappearance prevention process for preventing the disappearance of an object in the overlapping image capturing area of the image capturing areas of the two cameras.

  FIG. 13A is a diagram illustrating an output image portion corresponding to an overlapping imaging region of the imaging region of the right camera 2R and the imaging region of the rear camera 2B, and corresponds to the rectangular region R13 indicated by the dotted line in FIG. .

  In FIG. 13A, a gray area PR1 is an image area in which the input image portion of the rear camera 2B is arranged, and each coordinate on the output image plane corresponding to the area PR1 has a rear camera. Coordinates on the 2B input image plane are associated.

  On the other hand, a region PR2 filled with white is an image region in which the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the portion PR2 has an input image plane of the right side camera 2R. The upper coordinates are associated.

  In the present embodiment, the region PR1 and the region PR2 are arranged so as to form a stripe pattern, and the boundary line of the portion where the regions PR1 and the region PR2 are alternately arranged in a stripe shape is centered on the turning center of the excavator 60. Determined by concentric circles on a horizontal plane.

  FIG. 13B is a top view showing the state of the space region diagonally right behind the shovel 60, and shows the current state of the space region imaged by both the rear camera 2B and the right side camera 2R. FIG. 13B shows that a rod-shaped three-dimensional object OB exists behind the excavator 60 diagonally to the right.

  FIG. 13C shows a part of an output image generated based on an input image obtained by actually capturing the spatial region shown in FIG. 13B with the rear camera 2B and the right side camera 2R.

  Specifically, the image OB1 is expanded in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the rear camera 2B. Represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

  Further, the image OB2 is expanded in the extension direction of the line connecting the right-side camera 2R and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the right rear camera 2R. Represents a thing. That is, the image OB2 is a part of an image of the three-dimensional object OB displayed when a road surface image in the output image portion is generated using the input image of the right side camera 2R.

  As described above, the image generating apparatus 100 includes the region PR1 in which the coordinates on the input image plane of the rear camera 2B are associated with the region PR2 in which the coordinates on the input image plane of the right-side camera 2R are associated in the overlapping imaging region. And mix. As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 related to one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

  14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12, and FIG. 14 (A) is the output of FIG. An image is shown, and FIG. 14B shows an output image after applying the image disappearance prevention process. In the region R12 surrounded by the alternate long and short dash line in FIG. 14A, the person disappears, whereas in the region R14 surrounded by the alternate long and short dash line in FIG. 14B, the person is displayed without disappearing.

  Next, a process in which the image generation apparatus 100 selects a predetermined area to be highlighted (hereinafter referred to as “predetermined area selection process”) will be described with reference to FIG.

  FIG. 15 is a diagram in which the detection ranges of the six object detection sensors in FIG. 11 are extracted, and shows the relationship between the detection ranges of the six object detection sensors and the predetermined region to be highlighted. In FIG. 15, the detection range of the object detection sensor that has detected the object is indicated by a black dotted line, and the detection range of the object detection sensor that has not detected the object is indicated by a gray dotted line.

  FIG. 15A shows a region HP1 selected as a predetermined region to be highlighted when an object is detected only by the first left-side object detection sensor 6L1. A region HP1 indicated by hatching is a region obtained by subtracting the overlap detection region (AND region) of the region ZL1 and the region ZL2 from the region ZL1.

  FIG. 15B shows a region HP2 selected as a predetermined region to be highlighted when an object is detected by the first left side object detection sensor 6L1 and the second left side object detection sensor 6L2. The region HP2 indicated by vertical stripe hatching is a region obtained by subtracting the overlap detection region (AND region) of the region ZL2 and the region ZB1 from the combined region (OR region) of the region ZL1 and the region ZL2.

  FIG. 15C shows a region HP3 selected as a predetermined region to be highlighted when an object is detected by the second left side object detection sensor 6L2 and the first rear object detection sensor 6B1. Note that the area HP3 indicated by hatching is the overlap detection area (AND area) of the areas ZL1 and ZL2 and the overlap detection area (AND area) of the areas ZB1 and ZB2 from the combined area (OR area) of the areas ZL2 and ZB1. ) Is subtracted.

  FIG. 15D shows a region HP4 selected as a predetermined region to be highlighted when an object is detected by the first rear object detection sensor 6B1 and the second rear object detection sensor 6B2. The region HP4 indicated by hatching is the overlap detection region (AND region) of the region ZL2 and region ZB1, the overlap detection region (AND region) of the region ZB2 and region ZR1, from the combined region (OR region) of the region ZB1 and region ZB2. ) Is subtracted.

  FIG. 15E shows a region HP5 selected as a predetermined region to be highlighted when an object is detected by the second rear object detection sensor 6B2 and the first right side object detection sensor 6R1. The region HP5 indicated by hatching is the overlap detection region (AND region) of the region ZB1 and region ZB2, the overlap detection region (AND region) of the region ZR1 and region ZR2, from the combined region (OR region) of the region ZB2 and region ZR1. ) Is subtracted.

  FIG. 15F shows a region HP6 selected as a predetermined region to be highlighted when an object is detected by the first right side object detection sensor 6R1 and the second right side object detection sensor 6R2. A region HP6 indicated by vertical stripe hatching is a region obtained by subtracting the overlap detection region (AND region) of the region ZB2 and the region ZR1 from the combined region (OR region) of the region ZR1 and the region ZR2.

  FIG. 15G shows a region HP7 selected as a predetermined region to be highlighted when an object is detected only by the second right side object detection sensor 6R2. A region HP7 indicated by hatching is a region obtained by subtracting the overlap detection region (AND region) of the region ZR1 and the region ZR2 from the region ZR2.

  FIG. 15H shows a predetermined area to be highlighted when an object is detected by the first left side object detection sensor 6L1, the second left side object detection sensor 6L2, and the first rear object detection sensor 6B1. The regions HP2 and HP3 to be selected are shown.

  Next, a process in which the image generation apparatus 100 highlights a predetermined area (hereinafter referred to as “highlight display process”) will be described with reference to FIG. FIG. 16 is a flowchart showing the flow of the highlighting process. The image generating apparatus 100 repeatedly executes the highlighting process at a predetermined cycle while the excavator 60 is operating.

  First, the highlighting means 12 in the control unit 1 of the image generating apparatus 100 determines whether or not an object has been detected based on the outputs of the six object detection sensors 6 attached to the excavator 60 (step). S11).

  When it is determined that no object is detected by any of the object detection sensors 6 (NO in step S11), the highlighting display unit 12 ends the current highlighting process.

  On the other hand, if it is determined that an object has been detected (YES in step S11), the highlighting means 12 identifies which object detection sensor 6 has detected the object (step S12).

  Thereafter, the highlighting means 12 selects a predetermined area where the presence of the object is estimated from a plurality of predetermined areas based on the identification result (step S13). Specifically, the highlighting means 12 selects one or a plurality of predetermined areas by reading the identification number of the predetermined area stored in advance in a ROM or the like in association with the combination of the object detection sensors 6 that detected the object. To do.

  Thereafter, the highlighting means 12 highlights the selected one or more predetermined areas (step S14). Specifically, the highlighting means 12 superimposes and displays the outline of the selected one or more predetermined areas on the output image.

  Next, an example of an output image including the highlighted predetermined area will be described with reference to FIG. In FIG. 17, outlines HL1 to HL3 of three selected predetermined areas and warning marks EM1 to EM3 corresponding to the three outlines HL1 to HL3 are superimposed on the output image of FIG. Indicates the case of display.

  As shown in FIG. 17, the contour HL1 and the warning mark EM1 can easily convey to the operator of the excavator 60 the presence of the worker imaged by the left-side camera 2L that exists on the left side of the excavator 60.

  Further, the contour HL2 and the warning mark EM2 can easily convey to the operator of the excavator 60 the presence of the worker imaged by the rear camera 2B existing behind the excavator 60.

  In addition, the contour HL3 and the warning mark EM3 convey to the operator of the excavator 60 the presence of workers imaged by both the rear camera 2B and the right-side camera 2R, which are present obliquely rearward to the right of the excavator 60. it can.

  As described above, the image generation apparatus 100 displays the outline of the predetermined area corresponding to a part of the imaging area of one camera, and indicates the presence of the worker in a part of the imaging area as an operator of the excavator 60. Can be easily communicated to. In addition, the image generation apparatus 100 displays the outline of a predetermined area corresponding to the overlapping imaging area of the two cameras in a superimposed manner so that the operator of the excavator 60 can easily understand the presence of the worker in the overlapping imaging area. it can.

  With the above configuration, the image generation apparatus 100 identifies which object detection sensor 6 has detected an object, narrows down an area where the presence of the object is estimated, and highlights the area. As a result, the image generating apparatus 100 can inform the operator of the status of surrounding objects in an easy-to-understand manner.

  Further, even when an omnidirectional sensor is used as the object detection sensor 6, the image generation apparatus 100 highlights the area after narrowing down the area where the presence of the object is estimated. Therefore, the image generation apparatus 100 emphasizes the region after narrowing down the region where the presence of the object is estimated without using a directional sensor such as a scanning ultrasonic sensor or a scanning optical sensor, or a distance image sensor. Can be displayed. However, the image generation apparatus 100 may use a directivity sensor, a distance image sensor, or the like as the object detection sensor 6.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 employs the cylindrical spatial model MD as the spatial model, but may employ a spatial model having other columnar shapes such as a polygonal column, A spatial model composed of two side surfaces may be adopted, or a spatial model having only side surfaces may be adopted.

  The image generating apparatus 100 is mounted with a camera on a self-propelled excavator having movable members such as a bucket, an arm, a boom, and a turning mechanism, and moves the shovel while presenting surrounding images to the operator. An operation support system that supports the operation of the movable member is configured. However, the image generating apparatus 100 is a work machine that does not have a turning mechanism such as a forklift or asphalt finisher, or a work machine that has a movable member such as an industrial machine or a fixed crane, but does not self-propelled. You may comprise the operation assistance system mounted with a camera and assisting operation of these work machines.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 6 .... Object detection sensor 10 ... Coordinate association means 11 ... Image generation means 12 ... Highlight display means 40 ... Input image / spatial model correspondence map 41 ... Spatial model / processing target image correspondence map 42 ... Processing target image / output image correspondence map 60 ... Excavator 61 ... Lower traveling body 62 ... Turning mechanism 63 ... Upper turning body 64 ... Cab 100 ... Image generating device

Claims (8)

A work machine periphery monitoring device that generates an output image based on an input image captured by a camera attached to the work machine,
An object detection sensor that is attached to the work machine and detects an object existing around the work machine;
Highlighting means for identifying an object detection sensor that has detected the object when the object is detected by the object detection sensor, and highlighting a predetermined region of the output image according to the identification result;
The object detection sensor is composed of a plurality of object detection sensors,
The detection region of one object detection sensor of the plurality of object detection sensors and the detection region of another one object detection sensor of the plurality of object detection sensors have an overlap detection region,
The highlighting means subtracts the overlap detection area from the detection area of the one object detection sensor when the one object detection sensor detects an object and the other one object detection sensor does not detect the object. Highlight the area as the predetermined area,
Perimeter monitoring device for work machines. The output image is a viewpoint conversion image that overlooks the periphery of the work machine,
The predetermined area is a fan-shaped area that spreads radially around the position of the work machine.
The work machine periphery monitoring device according to claim 1. Said highlighting means for highlighting a region that is set based on the detection area of the object detection sensor as the predetermined area,
The periphery monitoring apparatus for work machines according to claim 1 or 2. The camera is composed of a plurality of cameras.
The imaging area of one of the plurality of cameras and the imaging area of the other one of the plurality of cameras have an overlapping imaging area,
The overlap imaging area corresponds to the overlap detection area,
The work machine periphery monitoring device according to any one of claims 1 to 3 .   A work machine periphery monitoring device that generates an output image based on an input image captured by a camera attached to the work machine,
  An object detection sensor that is attached to the work machine and detects an object existing around the work machine;
  Highlighting means for identifying an object detection sensor that has detected the object when the object is detected by the object detection sensor, and highlighting a predetermined region of the output image according to the identification result;
  The output image is a viewpoint conversion image that overlooks the periphery of the work machine,
  The predetermined area is a plurality of areas radially extending around the position of the work machine,
  The camera is composed of a plurality of cameras.
  The imaging area of one of the plurality of cameras and the imaging area of the other one of the plurality of cameras have an overlapping imaging area,
  The predetermined area includes an area corresponding to the overlapping imaging area, and an area corresponding to a part other than the overlapping imaging area among the imaging areas of the one camera.
Perimeter monitoring device for work machines. The highlighting means displays an outline of the predetermined area;
The work machine periphery monitoring device according to any one of claims 1 to 5. The object detection sensor is an ultrasonic sensor or a pyroelectric infrared sensor.
The work machine periphery monitoring device according to any one of claims 1 to 6. The output image is generated using input images of a plurality of cameras.
The work machine periphery monitoring device according to any one of claims 1 to 7.