JP2014181509A - Periphery monitoring apparatus for working machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a periphery monitoring apparatus for a working machine capable of urging an operator to confirm that a person existing in a prescribed range around the working machine leaves the prescribed range.SOLUTION: A periphery monitoring apparatus 100B includes an alarm output section 7 for outputting an alarm to an operator. The periphery monitoring apparatus 100B further includes working machine state determination means 14 for determining a state of a shovel 60, human existence determination means 12 for determining the existence of a person around the shovel 60, and alarm control means 13 for controlling the alarm output section 7. The alarm control means 13 makes the alarm to be output when it is determined that the shovel 60 is not in the state that working is possible and a person exists around the shovel 60, and it is determined that the shovel 60 is in the state that working is possible after that.

Description

  The present invention relates to a work machine periphery monitoring device having a function of determining the presence or absence of a person around a work machine.

  2. Description of the Related Art Conventionally, a periphery monitoring device that emits an alarm sound when an operator is detected within a monitoring range of an obstacle detector mounted on an excavator is known (see, for example, Patent Document 1). Whether or not the worker who entered the work area set around the excavator is a collaborator is judged based on the light emission pattern of the LED attached to the helmet of the worker and outputs an alarm sound. There is known an alarm system that determines whether or not (see, for example, Patent Document 2). In addition, a safety device is known that performs communication between a forklift and an operator who performs work in the vicinity (around) and controls the output of an alarm sound based on this communication (see, for example, Patent Document 3). ).

JP 2008-179940 A JP 2009-193494 A JP 2007-310587 A

  However, all of the techniques of Patent Documents 1 to 3 are based on the premise that an operator who has entered a predetermined range can be reliably detected, and do not assume a case where a worker who works within the predetermined range is missed. Therefore, if the alarm is stopped, the operator such as the excavator determines that the worker who was working within the predetermined range has moved out of the predetermined range, and the operator using the excavator does not perform confirmation by the operator himself / herself. May start.

  In view of the above, the present invention provides a work machine periphery monitoring device that can prompt an operator to confirm that a person who was within a predetermined range around the work machine has left the predetermined range. The purpose is to do.

  In order to achieve the above-described object, a work machine periphery monitoring device according to an embodiment of the present invention is a work machine periphery monitoring device including an alarm output unit that outputs an alarm to an operator. A working machine state determining means for determining a state, a presence / absence determining means for determining presence / absence of a person around the working machine, and an alarm control means for controlling the alarm output unit, wherein the alarm control means When it is determined that the work machine is not in a workable state and it is determined that there is a person around the work machine, an alarm is issued when it is subsequently determined that the work machine is in a workable state. Output.

  With the above-described means, the present invention provides a work machine periphery monitoring device that can prompt an operator to confirm that a person who was within a predetermined range around the work machine has left the predetermined range. be able to.

It is a block diagram which shows roughly the structural example of the image generation apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of the shovel mounted with an image generation apparatus. It is a figure which shows an example of the space model on which an input image is projected. It is FIG. (1) which shows an example of the relationship between a space model and a process target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is a display example (the 1) of an output image. It is a top view of the shovel in which an image generation apparatus is mounted. FIG. 4 is a diagram (No. 1) illustrating input images of three cameras mounted on an excavator and output images generated using the input images. It is a figure for demonstrating the image loss prevention process which prevents the loss | disappearance of the object in the overlap part of each imaging space of two cameras. FIG. 13 is a comparison diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying image loss prevention processing to the output image of FIG. 12. It is FIG. (2) which shows each input image of the three cameras mounted in the shovel, and the output image produced | generated using those input images. It is FIG. (2) which shows an example of the relationship between a space model and a process target image plane. It is a figure explaining the relationship of the two output images switched by the 1st output image switching process. It is a figure explaining the relationship of the three output images switched by the 2nd output image switching process. It is a correspondence table | surface which shows the correspondence of the determination result of a human presence determination means, and the content of an output image. It is a flowchart which shows the flow of a 1st alarm control process. It is a figure which shows an example of transition of the output image displayed during a 1st alarm control process. It is a flowchart which shows the flow of a 2nd alarm control process. It is a figure which shows an example of transition of the output image displayed during a 2nd alarm control process. It is a flowchart which shows the flow of a working machine state determination process. It is a flowchart which shows the flow of a warning control process at the time of a work start. It is a block diagram which shows another structural example of a periphery monitoring apparatus roughly. It is a block diagram which shows schematically the further another structural example of a periphery monitoring apparatus.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically illustrating a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

  The image generation device 100 is an example of a work machine periphery monitoring device that monitors the periphery of the work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, a human detection sensor 6, an alarm. An output unit 7, a gate lock lever 8, and an ignition switch 9 are included. Specifically, the image generation device 100 generates an output image based on an input image captured by the camera 2 mounted on the work machine and presents the output image to the operator. Further, the image generation device 100 switches the content of the output image to be presented based on the output of the human detection sensor 6.

  FIG. 2 is a diagram illustrating a configuration example of an excavator 60 as a work machine on which the image generating apparatus 100 is mounted. The excavator 60 is placed on a crawler-type lower traveling body 61 via a turning mechanism 62. A swing body 63 is mounted so as to be swingable around a swing axis PV.

  The upper swing body 63 includes a cab (operator's cab) 64 on the front left side, a drilling attachment E on the front center, and the camera 2 (right camera 2R, rear camera 2B) on the right and rear surfaces. ) And a person detection sensor 6 (right person detection sensor 6R, rear person detection sensor 6B). In addition, the display part 5 is installed in the position in the cab 64 where the operator is easy to visually recognize. In the cab 64, an alarm output unit 7 (right alarm output unit 7R, rear alarm output unit 7B), a gate lock lever 8, and an ignition switch 9 are installed.

  Next, each component of the image generation apparatus 100 will be described.

  The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. In the present embodiment, the control unit 1 has programs corresponding to, for example, a coordinate association unit 10, an image generation unit 11, a presence / absence determination unit 12, an alarm control unit 13, and a work machine state determination unit 14 described later. The CPU stores the data in the ROM or NVRAM, and causes the CPU to execute processing corresponding to each means while using the RAM as a temporary storage area.

  The camera 2 is a device for acquiring an input image that reflects the surroundings of the excavator 60. In the present embodiment, the camera 2 is, for example, a right side camera 2R and a rear camera 2B that are attached to the right side and the rear side of the upper swing body 63 so as to image a blind spot of the operator in the cab 64 (see FIG. 2). The camera 2 includes an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 2 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front side and the left side), and a wide-angle lens or a fisheye lens is attached so as to capture a wide range. It may be.

  In addition, the camera 2 acquires an input image according to a control signal from the control unit 1 and outputs the acquired input image to the control unit 1. In addition, when the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image obtained by correcting apparent distortion and tilt caused by using these lenses is transmitted to the control unit 1. Output. Further, the camera 2 may output an input image without correcting its apparent distortion and tilt to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

  The input unit 3 is a device that allows an operator to input various types of information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

  The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, or a semiconductor memory.

  The display unit 5 is a device for displaying image information. For example, the display unit 5 is a liquid crystal display, a projector, or the like installed in the cab 64 (see FIG. 2) of the excavator 60. Display an image.

  The human detection sensor 6 is a device for detecting a person existing around the excavator 60. In the present embodiment, the human detection sensor 6 is attached to the right side surface and the rear surface of the upper swing body 63 so as to detect, for example, a person present in the blind spot of the operator in the cab 64 (see FIG. 2). .

  The person detection sensor 6 is a sensor that distinguishes and detects a person from an object other than a person. For example, the person detection sensor 6 is a sensor that detects a change in energy in a corresponding monitoring space, and includes a pyroelectric infrared sensor, a bolometer infrared sensor, Includes a moving object detection sensor using an output signal of an infrared camera or the like. In this embodiment, the human detection sensor 6 uses a pyroelectric infrared sensor, and detects a moving object (moving heat source) as a person. The monitoring space of the right person detection sensor 6R is included in the imaging space of the right camera, and the monitoring space of the rear person detection sensor 6B is included in the imaging space of the rear camera 2B.

  Note that the human detection sensor 6 may be attached to a position other than the right side and the rear side of the upper swing body 63 (for example, the front and left sides), like the camera 2. It may be attached to any one of the left side surface, the right side surface, and the rear surface, or may be attached to all surfaces.

  The alarm output unit 7 is a device that outputs an alarm for the operator of the excavator 60. For example, the alarm output unit 7 is an alarm device that outputs at least one of sound and light, and includes a sound output device such as a buzzer and a speaker, and a light emitting device such as an LED and a flashlight. In the present embodiment, the alarm output unit 7 is a buzzer that outputs an alarm sound. The right side alarm output unit 7 </ b> R attached to the right inner wall of the cab 64 and the rear alarm output unit attached to the rear inner wall of the cab 64. 7B (see FIG. 2).

  The gate lock lever 8 is a device that switches the state of the excavator 60. In the present embodiment, the gate lock lever 8 has a locked state in which the excavator 60 is in a work-impossible state and a unlocked state in which the excavator 60 is in a workable state. The “workable state” means a state where the operator can operate the shovel 60, and the “work not possible state” means a state where the operator cannot operate the shovel 60. The gate lock lever 8 repeatedly outputs a signal representing its current state to the control unit 1 at a predetermined cycle.

  Specifically, the operator raises the gate lock lever 8 and makes it substantially horizontal to bring the gate lock lever 8 into the unlocked state, and depressing the gate lock lever 8 brings the gate lock lever 8 into the locked state.

  In the unlocked state, the gate lock lever 8 blocks an entrance / exit of the cab 64 and restricts the operator from exiting the cab 64, and controls an operation lever and an operation pedal (not shown) in the cab 64 (hereinafter referred to as “operation”). The operation according to “Pedal etc.” is made effective so that the operator can operate the shovel 60.

  On the other hand, in the locked state, the gate lock lever 8 opens the entrance / exit of the cab 64 and allows the operator to leave the cab 64 while invalidating the operation by the operation lever or the like in the cab 64. Prevents the shovel 60 from being operated.

  More specifically, the gate lock lever 8 closes the gate lock valve in the locked state and opens the gate lock valve in the unlocked state. The gate lock valve is a switching valve provided between an oil passage between a control valve (not shown) and an operation lever or the like. The control valve is a flow control valve that controls the flow of hydraulic oil between a hydraulic pump (not shown) and various hydraulic actuators. In the closed state, the gate lock valve shuts off the flow of hydraulic oil between the control valve and the operation lever and disables the operation lever and the like. In the open state, the gate lock valve makes the operation lever and the like effective by communicating hydraulic fluid between the control valve and the operation lever and the like.

  The ignition switch 9 is a device that switches the state of the excavator 60. In the present embodiment, the ignition switch 9 has an off state in which the excavator 60 is in an unworkable state and an on state in which the excavator 60 is in a workable state. The ignition switch 9 repeatedly outputs a signal representing its current state to the control unit 1 repeatedly at a predetermined cycle.

  Specifically, the operator switches the on / off state of the ignition switch 9 by pressing the ignition switch 9.

  In the ON state, the ignition switch 9 starts a control pump that supplies hydraulic oil to an oil passage between the control valve and the operation lever and the like by starting the engine. The ignition switch 9 enables the operation of the excavator 60 by enabling the operation by the operation lever or the like in the cab 64.

  On the other hand, in the off state, the ignition switch 9 stops the control pump by stopping the engine. The ignition switch 9 disables the operation of the operation lever or the like in the cab 64 so that the operator cannot operate the excavator 60.

  In addition, the image generation apparatus 100 generates a processing target image based on the input image, and performs an image conversion process on the processing target image so that the positional relationship with the surrounding objects and a sense of distance can be intuitively grasped. An output image to be generated may be generated, and the output image may be presented to the operator.

  The “processing target image” is an image that is a target of image conversion processing (for example, scale conversion processing, affine conversion processing, distortion conversion processing, viewpoint conversion processing, etc.) that is generated based on the input image. Specifically, the “processing target image” is, for example, an input image by a camera that images the ground surface from above, and includes an image in the horizontal direction (for example, an empty portion) with a wide angle of view. It is an image suitable for image conversion processing generated from the above. More specifically, the input image is projected onto a predetermined spatial model so that the horizontal image is not displayed unnaturally (for example, the sky is not treated as being on the ground surface) It is generated by reprojecting the projection image projected on the spatial model onto another two-dimensional plane. The processing target image may be used as an output image as it is without performing an image conversion process.

  The “space model” is a projection target of the input image. Specifically, the “space model” includes at least one plane or curved surface including a plane or curved surface other than the processing target image plane that is a plane on which the processing target image is located. The plane or curved surface other than the processing target image plane that is the plane on which the processing target image is located is, for example, a plane parallel to the processing target image plane or a plane or curved surface that forms an angle with the processing target image plane. .

  Note that the image generation apparatus 100 may generate an output image by performing image conversion processing on the projection image projected on the space model without generating a processing target image. Further, the projection image may be used as an output image as it is without being subjected to image conversion processing.

  FIG. 3 is a diagram illustrating an example of a spatial model MD onto which an input image is projected, and the left diagram in FIG. 3 illustrates a relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. 3 shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

  As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and includes a planar region R1 inside the bottom surface and a curved region R2 inside the side surface.

  FIG. 4 is a diagram illustrating an example of a relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is a plane including the plane area R1 of the space model MD, for example. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for the sake of clarity, the space model MD may be either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the subsequent drawings. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the spatial model MD, or may be an annular area not including the plane area R1 of the spatial model MD.

  Next, various units included in the control unit 1 will be described.

  The coordinate association unit 10 is a unit for associating coordinates on the input image plane where the input image captured by the camera 2 is located, coordinates on the space model MD, and coordinates on the processing target image plane R3. In the present embodiment, the coordinate association unit 10 includes, for example, various parameters relating to the camera 2 that are set in advance or input via the input unit 3, and predetermined input image planes, space models MD, The coordinates on the input image plane, the coordinates on the space model MD, and the coordinates on the processing target image plane R3 are associated with each other based on the mutual positional relationship of the processing target image plane R3. The various parameters related to the camera 2 are, for example, the optical center of the camera 2, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method, and the like. Then, the coordinate association unit 10 stores these correspondences in the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41 of the storage unit 4.

  When the processing target image is not generated, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

  The image generation means 11 is a means for generating an output image. In the present embodiment, the image generation unit 11 performs, for example, scale conversion, affine transformation, or distortion conversion on the processing target image, so that the coordinates on the processing target image plane R3 and the output image plane on which the output image is located. Match coordinates. Then, the image generation unit 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image. The value of each pixel is, for example, a luminance value, a hue value, a saturation value, and the like.

  Further, the image generation means 11 outputs the output image plane where the coordinates on the processing target image plane R3 and the output image are located based on various parameters relating to the virtual camera set in advance or input via the input unit 3. Correlate with the coordinates above. Various parameters relating to the virtual camera are, for example, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the image generation unit 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image.

  Note that the image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

  Further, when the processing target image is not generated, the image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane according to the applied image conversion processing. Then, the image generation unit 11 generates an output image by associating the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40. In this case, the image generation unit 11 omits the association between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence relationship in the processing target image / output image correspondence map 42. .

  Further, the image generation unit 11 switches the content of the output image based on the determination result of the presence / absence determination unit 12. Details of the output image switching by the image generation means 11 will be described later.

  The presence / absence determination means 12 is a means for determining the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine. In the present embodiment, the presence / absence determination unit 12 determines the presence / absence of a person around the excavator 60 based on the output of the person detection sensor 6.

  Further, the presence / absence determination unit 12 may determine the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine based on an input image captured by the camera 2. Specifically, the presence / absence determination unit 12 may determine the presence / absence of a person around the work machine using an image processing technique such as optical flow or pattern matching. The presence / absence determination unit 12 may determine the presence / absence of a person around the work machine based on an output of an image sensor different from the camera 2.

  Alternatively, the presence / absence determination unit 12 determines the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine based on the output of the human detection sensor 6 and the output of the image sensor such as the camera 2. Also good.

  The alarm control means 13 is a means for controlling the alarm output unit 7. In this embodiment, the alarm control unit 13 outputs an alarm based on the determination result of the presence / absence determination unit 12 or based on the determination result of the presence / absence determination unit 12 and the determination result of the work machine state determination unit 14. The unit 7 is controlled. The details of the control of the alarm output unit 7 by the alarm control means 13 will be described later.

  The work machine state determination means 14 is a means for determining the state of the work machine. In this embodiment, the work machine state determination unit 14 determines whether or not the excavator 60 is in a workable state. Details of the determination of the state of the excavator 60 by the work machine state determination means 14 will be described later.

  Next, an example of specific processing by the coordinate association unit 10 and the image generation unit 11 will be described.

  The coordinate association means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, a Hamilton quaternion.

  FIG. 5 is a diagram for explaining the association between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 is represented as one plane in the UVW orthogonal coordinate system with the optical center C of the camera 2 as the origin. The spatial model is represented as a three-dimensional surface in the XYZ orthogonal coordinate system.

  First, the coordinate matching unit 10 moves the origin of the XYZ coordinate system in parallel to the optical center C (the origin of the UVW coordinate system), then sets the X axis as the U axis, the Y axis as the V axis, and the Z axis. The XYZ coordinate system is rotated so as to match the −W axis. This is because the coordinates on the space model (the coordinates on the XYZ coordinate system) are converted to the coordinates on the input image plane (the coordinates on the UVW coordinate system). Note that the sign “-” of “−W axis” means that the direction of the Z axis is opposite to the direction of the −W axis. This is because the UVW coordinate system has the + W direction in front of the camera, and the XYZ coordinate system has the −Z direction in the vertically downward direction.

  When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system, so the coordinate association unit 10 moves the XYZ coordinate system in parallel with respect to each of the plurality of UVW coordinate systems, and Rotate.

  In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 is the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis. This is realized by rotating to coincide with the axis. Therefore, the coordinate matching means 10 can combine these two rotations into one rotation calculation by describing this conversion in Hamilton's quaternion.

  By the way, the rotation for making one vector A coincide with another vector B corresponds to the process of rotating the vector A and the vector B by the angle formed by using the normal line of the plane formed by the vectors A and B as an axis. . And, if the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

It is represented by

  Further, the unit vector N of the normal line between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

It will be expressed as

  Note that the quaternion has i, j, and k as imaginary units,

In this embodiment, the quaternion Q is represented by t as a real component and a, b, and c as pure imaginary components.

The conjugate quaternion of the quaternion Q is

It is represented by

  The quaternion Q can represent a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b , C can also be used to express a rotational motion with an arbitrary vector as an axis.

  Further, the quaternion Q can be expressed as a single rotation operation by integrating a plurality of continuous rotation operations. Specifically, the quaternion Q is obtained when, for example, an arbitrary point S (sx, sy, sz) is rotated by an angle θ with an arbitrary unit vector C (l, m, n) as an axis. The point D (ex, ey, ez) can be expressed as follows.

Here, in this embodiment, if the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X ′. X '

It is represented by

  In this embodiment, if the quaternion representing the rotation that matches the line connecting the point X ′ on the X axis and the origin to the U axis is Qx, “the Z axis matches the −W axis, , A quaternion R representing "rotation to make the X axis coincide with the U axis"

It is represented by

  As described above, the coordinate P ′ when an arbitrary coordinate P on the space model (XYZ coordinate system) is expressed by a coordinate on the input image plane (UVW coordinate system) is

It is represented by In addition, since the quaternion R is invariant in each of the cameras 2, the coordinate association unit 10 thereafter executes the calculation to convert the coordinates on the space model (XYZ coordinate system) to the input image plane (UVW Can be converted to coordinates on the coordinate system).

  After the coordinates on the space model (XYZ coordinate system) are converted to the coordinates on the input image plane (UVW coordinate system), the coordinate association means 10 is formed by the line segment CP ′ and the optical axis G of the camera 2. The incident angle α is calculated. The line segment CP ′ is a line segment that connects the optical center C (coordinate on the UVW coordinate system) of the camera 2 and a coordinate P ′ that represents an arbitrary coordinate P on the space model in the UVW coordinate system.

  In addition, the coordinate association unit 10 calculates the deflection angle φ and the length of the line segment EP ′ in the plane H that is parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and includes the coordinates P ′. The line segment EP ′ is a line segment connecting the intersection point E between the plane H and the optical axis G and the coordinate P ′, and the deflection angle φ is formed by the U ′ axis and the line segment EP ′ in the plane H. It is an angle to do.

  In the camera optical system, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate correlating means 10 performs normal projection (h = ftan α), orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), and equal solid angle projection (h = 2 fsin (α / 2). )), An appropriate projection method such as equidistant projection (h = fα) is selected to calculate the image height h.

  Thereafter, the coordinate matching means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and is a numerical value corresponding to the pixel size per pixel of the input image plane R4. Divide. As a result, the coordinate association unit 10 can associate the coordinates P (P ′) on the space model MD with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

It is represented by

  In this way, the coordinate association unit 10 associates the coordinates on the space model MD with the coordinates on one or a plurality of input image planes R4 existing for each camera, and coordinates on the space model MD, the camera identifier. And the coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

  Further, since the coordinate association unit 10 calculates the coordinate conversion using the quaternion, unlike the case where the coordinate conversion is calculated using the Euler angle, there is an advantage that no gimbal lock is generated. . However, the coordinate association unit 10 is not limited to the one that calculates the coordinate conversion using the quaternion, and may perform the coordinate conversion using the Euler angle.

  Note that, when it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate associating means 10 inputs the coordinates P (P ′) on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with coordinates on the image plane R4, or may be associated with coordinates on the input image plane R4 selected by the operator.

  Next, a process of reprojecting coordinates on the curved surface area R2 (coordinates having a component in the Z-axis direction) among the coordinates on the spatial model MD onto the processing target image plane R3 on the XY plane will be described.

  FIG. 6 is a diagram for explaining the association between coordinates by the coordinate association means 10. F6A is a diagram illustrating a correspondence relationship between coordinates on the input image plane R4 of the camera 2 that adopts normal projection (h = ftanα) as an example and coordinates on the space model MD. The coordinate associating means 10 is arranged so that each line segment connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the spatial model MD corresponding to the coordinates passes through the optical center C of the camera 2. Associate both coordinates.

  In the example of F6A, the coordinate association means 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane area R1 of the space model MD, and coordinates K2 on the input image plane R4 of the camera 2 Is associated with the coordinate L2 on the curved surface region R2 of the space model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

  In addition, when the camera 2 employs a projection method other than normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the coordinate association unit 10 uses each projection method. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

  Specifically, the coordinate association unit 10 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the space model MD. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

  F6B is a diagram illustrating a correspondence relationship between coordinates on the curved surface region R2 of the space model MD and coordinates on the processing target image plane R3. The coordinate association unit 10 introduces a parallel line group PL that is located on the XZ plane and forms an angle β with the processing target image plane R3. Then, the coordinate association means 10 causes both the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3 corresponding to the coordinates to ride on one of the parallel line groups PL. Associate both coordinates.

  In the example of F6B, the coordinate association means 10 associates both coordinates on the assumption that the coordinate L2 on the curved surface region R2 of the spatial model MD and the coordinate M2 on the processing target image plane R3 are on a common parallel line.

  The coordinate association means 10 can also associate the coordinates on the plane area R1 of the space model MD with the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface area R2. is there. However, in the example of F6B, the plane area R1 and the processing target image plane R3 are a common plane. Therefore, the coordinate L1 on the plane area R1 of the space model MD and the coordinate M1 on the processing target image plane R3 have the same coordinate value.

  In this way, the coordinate association unit 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. And stored in the space model / processing object image correspondence map 41.

  F6C is a diagram illustrating a correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting the normal projection (h = ftanα) as an example. The image generation means 11 causes each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates to pass through the optical center CV of the virtual camera 2V. And correlate both coordinates.

  In the example of F6C, the image generation unit 11 associates the coordinate N1 on the output image plane R5 of the virtual camera 2V with the coordinate M1 on the processing target image plane R3 (plane area R1 of the spatial model MD), and the virtual camera 2V The coordinate N2 on the output image plane R5 is associated with the coordinate M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

  When the virtual camera 2V employs a projection method other than the normal projection (for example, orthographic projection, stereoscopic projection, equisolid angle projection, equidistant projection, etc.), the image generation unit 11 uses each projection method. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

  Specifically, the image generating unit 11 is configured to use a predetermined function (for example, orthographic projection (h = fsin α), stereoscopic projection (h = 2 ftan (α / 2)), or equal solid angle projection (h = 2 fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segment M1-N1 and the line segment M2-N2 do not pass through the optical center CV of the virtual camera 2V.

  In this way, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and sets the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. The image is associated and stored in the processing image / output image correspondence map 42. The image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41. Generate an output image.

  F6D is a diagram in which F6A to F6C are combined, and shows the positional relationship between the camera 2, the virtual camera 2V, the plane area R1 and the curved area R2 of the spatial model MD, and the processing target image plane R3.

  Next, the operation of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  The left figure of FIG. 7 is a figure in case the angle (beta) is formed between the parallel line group PL located on XZ plane, and process target image plane R3. On the other hand, the right diagram in FIG. 7 is a diagram in the case where an angle β1 (β1> β) is formed between the parallel line group PL positioned on the XZ plane and the processing target image plane R3. Each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left diagram of FIG. 7 and the right diagram of FIG. 7 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. Further, the intervals between the coordinates La to Ld in the left diagram in FIG. 7 are equal to the respective intervals between the coordinates La to Ld in the right diagram in FIG. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the processing target image plane R3. To do. In this case, the Z axis is referred to as a “reprojection axis”.

  As shown in the left diagram of FIG. 7 and the right diagram of FIG. 7, the distance between the coordinates Ma to Md on the processing target image plane R3 increases the angle between the parallel line group PL and the processing target image plane R3. Decreases linearly with time. That is, the distance decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md. On the other hand, since the coordinate group on the plane region R1 of the space model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 7, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups is such that only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD is linearly enlarged or out of the image portion on the output image plane R5 (see FIG. 6). It means to be reduced.

  Next, an alternative example of the parallel line group PL introduced by the coordinate association unit 10 to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

  The left diagram in FIG. 8 is a diagram in the case where all the auxiliary line groups AL positioned on the XZ plane extend from the start point T1 on the Z axis toward the processing target image plane R3. On the other hand, the right figure of FIG. 8 is a figure in case all the auxiliary line groups AL are extended toward the process target image plane R3 from the starting point T2 (T2> T1) on a Z-axis. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left diagram of FIG. 8 and the right diagram of FIG. 8 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. In the example of the left diagram in FIG. 8, the coordinates Mc and Md are not shown because they are outside the region of the processing target image plane R3. Further, the respective intervals between the coordinates La to Ld in the left diagram of FIG. 8 are equal to the respective intervals of the coordinates La to Ld in the right diagram of FIG. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the processing target image plane R3. To do. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

  As shown in the left diagram of FIG. 8 and the right diagram of FIG. 8, the distance between the coordinates Ma to Md on the processing target image plane R3 is the distance (height) between the starting point of the auxiliary line group AL and the origin O. As it increases, it decreases non-linearly. That is, the greater the distance between the curved surface region R2 of the space model MD and each of the coordinates Ma to Md, the greater the reduction width of each interval. On the other hand, since the coordinate group on the plane region R1 of the spatial model MD is not converted into the coordinate group on the processing target image plane R3 in the example of FIG. 8, the interval between the coordinate groups does not change. .

  The change in the interval between these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6), as in the case of the parallel line group PL. It means that only the image portion is enlarged or reduced nonlinearly.

  In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and does not affect the space model MD. The image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface area R2 can be linearly or nonlinearly enlarged or reduced. Therefore, the image generation apparatus 100 does not affect the road surface image in the vicinity of the excavator 60 (virtual image when the excavator 60 is viewed from directly above), and does not affect the object (the horizontal direction from the excavator 60). The object in the image when viewing the surroundings can be expanded and contracted quickly and flexibly, and the visibility of the blind spot area of the excavator 60 can be improved.

  Next, referring to FIG. 9, the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. Processing to be generated (hereinafter referred to as “output image generation processing”) will be described. FIG. 9 is a flowchart showing the flow of the processing target image generation process (steps S1 to S3) and the output image generation process (steps S4 to S6). Further, the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved surface area R2), and the processing target image plane R3 is determined in advance.

  First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the space model MD using the coordinate association unit 10 (step S1).

  Specifically, the coordinate association unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3. Then, the coordinate association unit 10 calculates a point where one of the parallel line groups PL extending from one coordinate on the processing target image plane R3 intersects the curved surface region R2 of the space model MD. Then, the coordinate association unit 10 derives the coordinates on the curved surface region R2 corresponding to the calculated point as one coordinate on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3, and the correspondence relationship is derived. It is stored in the spatial model / processing object image correspondence map 41. It should be noted that the angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like. It may be a value to be entered.

  In addition, when one coordinate on the processing target image plane R3 matches one coordinate on the plane area R1 of the space model MD, the coordinate association unit 10 uses the one coordinate on the plane area R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R3, and the correspondence is stored in the space model / processing object image correspondence map 41.

  Thereafter, the control unit 1 causes the coordinate association unit 10 to associate one coordinate on the spatial model MD derived by the above-described processing with a coordinate on the input image plane R4 (step S2).

  Specifically, the coordinate association unit 10 acquires the coordinates of the optical center C of the camera 2 that employs normal projection (h = ftanα). And the coordinate matching means 10 is a line segment extended from one coordinate on the space model MD, and calculates the point where the line segment passing through the optical center C intersects the input image plane R4. Then, the coordinate association unit 10 derives a coordinate on the input image plane R4 corresponding to the calculated point as one coordinate on the input image plane R4 corresponding to the one coordinate on the space model MD, and the correspondence relationship is derived. The image is stored in the input image / space model correspondence map 40.

  Thereafter, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3). And when it determines with the control part 1 having not matched all the coordinates yet (NO of step S3), the process of step S1 and step S2 is repeated.

  On the other hand, when determining that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation process and then starts the output image generation process. And the control part 1 matches the coordinate on process target image plane R3, and the coordinate on output image plane R5 by the image generation means 11 (step S4).

  Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image. Then, the image generation unit 11 determines the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5, which are determined by the contents of the scale conversion, affine transformation, or distortion conversion performed. Store in the output image correspondence map 42.

  Alternatively, when generating the output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

  Alternatively, the image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V when generating an output image using the virtual camera 2V that employs normal projection (h = ftanα). Then, the image generation unit 11 calculates a point that is a line segment extending from one coordinate on the output image plane R5 and that intersects the processing target image plane R3 with a line segment passing through the optical center CV. Then, the image generation unit 11 derives the coordinate on the processing target image plane R3 corresponding to the calculated point as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5. In this way, the image generation unit 11 may store the correspondence relationship in the processing target image / output image correspondence map 42.

  Thereafter, the image generation unit 11 of the control unit 1 refers to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. The image generation unit 11 then associates the coordinates on the input image plane R4 with the coordinates on the space model MD, the correspondence between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the processing target. The correspondence between the coordinates on the image plane R3 and the coordinates on the output image plane R5 is traced. Then, the image generation unit 11 acquires values (for example, luminance values, hue values, saturation values, etc.) possessed by the coordinates on the input image plane R4 corresponding to the respective coordinates on the output image plane R5. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). Note that when a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 uses each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. The statistical value is, for example, an average value, a maximum value, a minimum value, an intermediate value, or the like.

  Thereafter, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6). And when it determines with the control part 1 having not matched all the values of the coordinate yet (NO of step S6), the process of step S4 and step S5 is repeated.

  On the other hand, if the control unit 1 determines that all coordinate values are associated (YES in step S6), the control unit 1 generates an output image and ends the series of processes.

  Note that the image generation apparatus 100 omits the processing target image generation processing when the processing target image is not generated. In this case, “coordinates on the processing target image plane” in step S4 in the output image generation processing is read as “coordinates on the space model”.

  With the above configuration, the image generation apparatus 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the object around the excavator 60 and the excavator 60.

  In addition, the image generation apparatus 100 performs coordinate association so as to go back from the processing target image plane R3 to the input image plane R4 via the spatial model MD. Thereby, the image generation device 100 can reliably correspond each coordinate on the processing target image plane R3 to one or a plurality of coordinates on the input image plane R4. Therefore, the image generation apparatus 100 can quickly generate a higher-quality processing target image as compared with the case where the coordinate association is executed in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD. Can do. When the coordinate association is executed in the order from the input image plane R4 to the processing target image plane R3 via the space model MD, each coordinate on the input image plane R4 is set to one or more on the processing target image plane R3. It is possible to reliably correspond to the coordinates. However, some of the coordinates on the processing target image plane R3 may not be associated with any of the coordinates on the input image plane R4, and in this case, some of the coordinates on the processing target image plane R3. Need to be interpolated.

  Further, when enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generating apparatus 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Thus, it is possible to realize a desired enlargement or reduction without rewriting the contents of the input image / space model correspondence map 40 only by rewriting only the portion related to the curved surface region R2 in the space model / processing object image correspondence map 41. .

  Further, when changing the appearance of the output image, the image generating apparatus 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters relating to scale conversion, affine transformation, or distortion transformation. The desired output image (scale-converted image, affine-transformed image, or distortion-converted image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing object image correspondence map 41.

  Similarly, when changing the viewpoint of the output image, the image generating apparatus 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / processing object image correspondence map 41.

  FIG. 10 is a display example when an output image generated using input images of two cameras 2 (the right side camera 2R and the rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. .

  The image generation apparatus 100 projects the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the space model MD, and then reprojects them onto the processing target image plane R3 to generate a processing target image. To do. Then, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. In this way, the image generating apparatus 100 has an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1), and an image in which the surroundings are viewed in the horizontal direction from the excavator 60 (an image in the processing target image plane R3). And an output image that simultaneously displays Hereinafter, such an output image is referred to as a peripheral monitoring virtual viewpoint image.

  Note that the virtual viewpoint image for periphery monitoring is subjected to image conversion processing (for example, viewpoint conversion processing) on the image projected on the space model MD when the image generation apparatus 100 does not generate a processing target image. Generated by.

  Further, the periphery monitoring virtual viewpoint image is trimmed in a circle so that the image when the excavator 60 performs the turning motion can be displayed without a sense of incongruity, the center CTR of the circle is on the cylindrical center axis of the space model MD, and It is generated so as to be on the pivot axis PV of the excavator 60. Therefore, the peripheral viewpoint virtual viewpoint image is displayed so as to rotate around the center CTR in accordance with the turning operation of the excavator 60. In this case, the cylindrical central axis of the space model MD may or may not coincide with the reprojection axis.

  Note that the radius of the space model MD is, for example, 5 meters. Further, the angle formed between the parallel line group PL and the processing target image plane R3 or the starting point height of the auxiliary line group AL is the maximum reachable distance (for example, 12 meters) of the excavation attachment E from the turning center of the shovel 60. If there is an object (for example, an operator) at a position separated by a certain distance, the display unit 5 is set to display the object sufficiently large (for example, 7 mm or more). obtain.

  Further, the virtual viewpoint image for peripheral monitoring may be a CG image of the excavator 60 so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. . This is to make the positional relationship between the shovel 60 and the object appearing in the output image easier to understand. It should be noted that a frame image including various information such as an orientation may be arranged around the virtual viewpoint image for periphery monitoring.

  Next, the details of the periphery monitoring virtual viewpoint image generated by the image generation apparatus 100 will be described with reference to FIGS.

  FIG. 11 is a top view of the excavator 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three person detection sensors 6 (left side person detection sensor 6L, right side). A person detection sensor 6R and a rear person detection sensor 6B). Note that regions CL, CR, and CB indicated by alternate long and short dashed lines in FIG. 11 indicate imaging spaces of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate monitoring spaces for the left side person detection sensor 6L, the right side person detection sensor 6R, and the rear person detection sensor 6B, respectively. In addition, the excavator 60 includes a display unit 5, three alarm output units 7 (a left side alarm output unit 7L, a right side alarm output unit 7R, and a rear alarm output unit 7B), a gate lock lever, and a gate lock lever. 8 and an ignition switch 9.

  In the present embodiment, the monitoring space of the human detection sensor 6 is narrower than the imaging space of the camera 2, but the monitoring space of the human detection sensor 6 may be the same as the imaging space of the camera 2. It may be wide. The monitoring space of the human detection sensor 6 is located in the vicinity of the shovel 60 in the imaging space of the camera 2, but may be in a region farther from the shovel 60. In addition, the monitoring space of the human detection sensor 6 has an overlapping portion in the portion where the imaging spaces of the camera 2 overlap. For example, in the overlapping portion of the imaging space CR of the right camera 2R and the imaging space CB of the rear camera 2B, the monitoring space ZR of the right person detection sensor 6R overlaps with the monitoring space ZB of the rear person detection sensor 6B. However, the monitoring space of the human detection sensor 6 may be arranged so as not to overlap.

  FIG. 12 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images.

  The image generation apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the space model MD, and then reprojects them onto the process target image plane R3 to generate a process target image. . The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generating apparatus 100 includes an image in which the vicinity of the excavator 60 is looked down from above (an image in the plane region R1) and an image in which the surroundings are viewed in the horizontal direction from the excavator 60 (an image in the processing target image plane R3). A virtual viewpoint image for periphery monitoring to be displayed at the same time is generated. Note that the image displayed at the center of the virtual viewpoint image for periphery monitoring is the CG image 60CG of the excavator 60.

  In FIG. 12, the input image of the right-side camera 2R and the input image of the rear camera 2B each capture a person in an overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B (right side). (See region R10 surrounded by a two-dot chain line in the input image of the direction camera 2R and region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.)

  However, if the coordinates on the output image plane are associated with the coordinates on the input image plane for the camera with the smallest incident angle, the output image loses the person in the overlapping portion (one point in the output image). (See region R12 surrounded by a chain line.)

  Therefore, in the output image portion corresponding to the overlapping portion, the image generating apparatus 100 associates the region on the input image plane of the rear camera 2B with the coordinate on the input image plane of the right-side camera 2R. The area is mixed to prevent the object in the overlapping portion from disappearing.

  FIG. 13 is a diagram for explaining a stripe pattern process which is an example of an image disappearance prevention process for preventing the disappearance of an object in an overlapping portion of the imaging spaces of two cameras.

  F13A is a diagram illustrating an output image portion corresponding to an overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B, and corresponds to a rectangular region R13 indicated by a dotted line in FIG.

  In F13A, a region PR1 filled in gray is an image region in which the input image portion of the rear camera 2B is arranged, and the input image of the rear camera 2B is at each coordinate on the output image plane corresponding to the region PR1. Coordinates on the plane are associated.

  On the other hand, a region PR2 filled with white is an image region in which the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the portion PR2 has an input image plane of the right side camera 2R. The upper coordinates are associated.

  In this embodiment, the region PR1 and the region PR2 are arranged so as to form a stripe pattern (stripe pattern processing), and the boundary line of the portion where the regions PR1 and the region PR2 are alternately arranged in a stripe shape is the swivel of the excavator 60. Defined by concentric circles on a horizontal plane centered on the center.

  F13B is a top view showing the situation of the space area diagonally right behind the excavator 60, and shows the current situation of the space area imaged by both the rear camera 2B and the right-side camera 2R. Further, F13B indicates that a rod-shaped three-dimensional object OB exists on the right rear side of the excavator 60.

  F13C represents a part of an output image generated based on an input image obtained by actually capturing the space area indicated by F13B with the rear camera 2B and the right-side camera 2R.

  Specifically, the image OB1 is expanded in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the rear camera 2B. Represents what was done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

  In addition, the image OB2 is expanded in the extension direction of the line connecting the right camera 2R and the three-dimensional object OB by the viewpoint conversion for generating the road surface image of the three-dimensional object OB in the input image of the right camera 2R. Represents a thing. That is, the image OB2 is a part of an image of the three-dimensional object OB displayed when a road surface image in the output image portion is generated using the input image of the right side camera 2R.

  As described above, the image generating apparatus 100 includes the region PR1 in which the coordinates on the input image plane of the rear camera 2B are associated with the region PR2 in which the coordinates on the input image plane of the right-side camera 2R are associated with each other in the overlapping portion. Mix. As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 related to one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

  FIG. 14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying image loss prevention processing (stripe pattern processing) to the output image of FIG. The output image of FIG. 12 is shown, and the lower part of FIG. 14 shows the output image after applying the image loss prevention process (stripe pattern process). The person disappears in the region R12 surrounded by the alternate long and short dash line in the upper diagram of FIG. 14, whereas the person is displayed without disappearing in the region R14 surrounded by the dashed dotted line in the lower diagram of FIG.

  Note that the image generating apparatus 100 may prevent the disappearance of the object in the overlapping portion by applying a mesh pattern process, an averaging process, or the like instead of the stripe pattern process. Specifically, the image generation apparatus 100 outputs an average value of corresponding pixel values (for example, luminance values) in the input images of the two cameras, corresponding to the overlapping portion, by averaging processing. It is adopted as the pixel value of the image portion. Alternatively, the image generation apparatus 100 performs mesh pattern processing so that, in the output image portion corresponding to the overlapping portion, the region in which the pixel value in the input image of one camera is associated with the pixel value in the input image of the other camera Are arranged so as to form a net pattern (mesh pattern). Thereby, the image generation device 100 prevents the objects in the overlapping portion from disappearing.

  Next, referring to FIGS. 15 to 17, the image generation unit 11 switches the content of the output image based on the determination result of the presence / absence determination unit 12 (hereinafter referred to as “first output image switching process”). .). FIG. 15 is a diagram illustrating input images of the three cameras 2 mounted on the excavator 60 and output images generated using the input images, and corresponds to FIG. FIG. 16 is a diagram illustrating an example of the relationship between the spatial model MD used in the first output image switching process and the processing target image plane R3, and corresponds to FIG. FIG. 17 is a diagram for explaining the relationship between two output images switched in the first output image switching process.

  As shown in FIG. 15, the image generating apparatus 100 projects the input images of the three cameras 2 onto the plane area R1 and the curved area R2 of the space model MD, and then reprojects them onto the processing target image plane R3. To generate a processing target image. The image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, etc.) on the generated processing target image. As a result, the image generation device 100 generates a peripheral monitoring virtual viewpoint image that simultaneously displays an image of the vicinity of the excavator 60 as viewed from above and an image of the periphery of the excavator 60 viewed in the horizontal direction.

  In FIG. 15, the input images of the left camera 2L, the rear camera 2B, and the right camera 2R indicate a state where there are three workers. The output image shows a state where nine workers are present around the excavator 60.

  The height of the plane region R1 of the space model MD is set to a height corresponding to the road surface that is the ground contact surface of the excavator 60. Therefore, in the image in the plane region R1 in FIG. 15, the distance between the ground contact position of each of the nine workers and the CG image 60CG of the shovel 60 is the distance between each of the nine workers and the shovel 60. It accurately represents the actual distance. However, the worker's image is displayed so as to increase as the distance from the ground contact position (foot) increases. In particular, as shown in FIG. 15, the operator's head is displayed significantly larger than the size of the CG image 60CG of the excavator 60. Therefore, the operator of the excavator 60 who sees the output image of FIG. 15 may have an illusion that the distance between the excavator 60 and the operator is larger than the actual distance.

  Therefore, the image generation unit 11 determines the content of the output image when the presence / absence determination unit 12 determines that a person is present in any of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. Switch. In the present embodiment, the image generating means 11 changes the height of the plane region R1 of the spatial model MD from the height corresponding to the road surface to the height corresponding to the height of the human head (hereinafter referred to as “head height”). Switch to). The head height is a preset value, for example, 150 cm. However, the head height may be a dynamically determined value. For example, when the height (height) of the worker existing around the excavator 60 can be detected based on the output of the human detection sensor 6 or the like, the image generation apparatus 100 determines the head height based on the detected height. May be. Specifically, the head height may be determined according to the height of the worker closest to the excavator 60, and the statistical values (maximum value, minimum value, The head height may be determined according to an average value or the like.

  Here, referring to FIG. 16 and FIG. 17, the space model MD of FIG. 4 having a plane area R1 having a height corresponding to the road surface, and the head height reference space model MDM having a plane area R1M of the head height, The relationship will be described. 16 shows the relationship between the space model MD and the processing target image plane R3 including the plane region R1, and the lower diagram in FIG. 16 shows the head height reference space model MDM and the head height reference plane region R1M. The head height reference processing target image plane R3M including the relationship is shown. In FIG. 17, an output screen D1 is a road surface height reference virtual viewpoint image for road monitoring generated using the space model MD, and an output image D2 is a head height generated using the space model MDM. It is a virtual viewpoint image for reference periphery monitoring. The image D3 is an explanatory image showing the difference in size between the road viewpoint-based periphery monitoring virtual viewpoint image and the head height-based periphery monitoring virtual viewpoint image. In addition, the size of the image portion D10 in the road viewpoint-based periphery monitoring virtual viewpoint image corresponds to the size of the head height-based periphery monitoring virtual viewpoint image.

  As shown in FIG. 16, the height of the head height reference plane area R1M and the head height reference processing target image plane R3M is higher than the height of the plane area R1M and processing target image plane R3 corresponding to the height of the road surface. The head height is high by HT.

  As a result, as shown in FIG. 17, the output image D <b> 2 that is the virtual viewpoint image for the periphery monitoring based on the head height has a distance between the body part (head) of the worker existing at the head height HT and the shovel 60. The actual distance between the operator and the excavator 60 is displayed. Thereby, the image generating apparatus 100 can prevent the operator of the shovel 60 who has seen the output image from capturing the distance between the shovel 60 and the worker larger than the actual distance.

  As shown in FIG. 16, a region D10M on the spatial model MDM corresponding to the image portion D10 is included in the head height reference plane region R1M. In other words, the virtual viewpoint image for periphery monitoring based on the head height does not use the input image portion that is re-projected on the annular portion (portion other than the head height reference plane region R1M) of the head height reference processing target image plane R3M. . For this reason, in this embodiment, the image generation apparatus 100 omits the association of coordinates in the region other than the region D10M.

  With the above configuration, the image generation apparatus 100 switches the content of the output image based on the determination result of the presence / absence determination unit 12. Specifically, when it is determined that there is a person around the excavator 60, the image generation device 100 switches the road surface height-based periphery monitoring virtual viewpoint image to the head height-based periphery monitoring virtual viewpoint image. . As a result, when the operator is detected around the excavator 60, the image generating apparatus 100 can more accurately convey the distance between the excavator 60 and the operator to the operator of the excavator 60. If the image generating apparatus 100 determines that there is no person around the excavator 60 after that, the head height-based periphery monitoring virtual viewpoint image is changed to a road surface height-based periphery monitoring virtual viewpoint image. Switch. This is because the operator of the shovel 60 can monitor the periphery of the shovel 60 in a wider range.

  Next, referring to FIG. 18, the image generation means 11 is referred to as another process for switching the contents of the output image based on the determination result of the presence / absence determination means 12 (hereinafter referred to as “second output image switching process”). ). FIG. 18 is a diagram for explaining the relationship between the three output images switched in the second output image switching process.

  The image generation apparatus 100 executes image loss prevention processing such as stripe pattern processing, mesh pattern processing, and averaging processing that prevent the disappearance of objects in the overlapping portions of the imaging spaces of the two cameras. Specifically, the image generation apparatus 100 overlaps the output image portion corresponding to the overlapped portion by mixing regions in which the coordinates on the input image planes of the two cameras are associated with each other by the image loss prevention processing. Prevent disappearance of objects in the part. However, the image loss prevention process has a problem that the visibility of the object in the overlapping portion is lowered.

  Therefore, the image generation unit 11 switches the content of the output image according to the determination result of the presence / absence determination unit 12. In the present embodiment, the image generation unit 11 sets a camera corresponding to a monitoring space determined to have a person as a priority camera and a camera corresponding to a monitoring space determined to have no person as a priority camera. Note that the imaging space of the priority camera and the imaging space of the priority camera have overlapping portions. Then, when the priority camera and the priority camera cannot be determined, the image generation unit 11 generates a peripheral monitoring virtual viewpoint image to which the image loss prevention process is applied. On the other hand, if the image generation unit 11 can determine the priority camera and the priority camera, the image generation unit 11 applies the output image portion corresponding to the overlapped portion on the input image plane of the priority camera without applying the image loss prevention process. Peripheral monitoring virtual viewpoint images are generated by associating the coordinates.

  Specifically, the image generation unit 11 performs peripheral monitoring using image loss prevention processing when it is determined that there is a person in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR. A virtual viewpoint image is generated. This is because none of the cameras can be a priority camera, and none of the cameras can be a priority camera. That is, if any one of the cameras is set as the priority camera, there is a possibility that the person in the overlapping portion may be lost.

  Further, the image generation means 11 is for peripheral monitoring that applies image loss prevention processing even when it is determined that no person is present in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR. A virtual viewpoint image is generated. This is because there is no target to be displayed while using any camera as the priority camera. However, in this case, the image generation unit 11 does not apply the image loss prevention process, and associates the coordinates on the input image plane of one of the cameras with the output image portion corresponding to the overlapping portion, thereby monitoring the periphery. A virtual viewpoint image may be generated. This is because there is no object that may be lost in the first place, and the processing load on the control unit 1 can be reduced by omitting the image loss prevention process.

  The output image D11 of FIG. 18 is a periphery to which image loss prevention processing is generated that is generated when it is determined that there is a person in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR. A virtual viewpoint image for monitoring is shown. In this case, the image generation unit 11 includes a region in which coordinates on the input image plane of the left camera 2L are associated with an output image portion R15 corresponding to an overlapping portion of the left imaging space CL and the rear imaging space CB, An area associated with coordinates on the input image plane of the camera 2B is mixed. In addition, the image generation unit 11 includes a region in which coordinates on the input image plane of the right-side camera 2R are associated with an output image portion R16 corresponding to an overlapping portion of the right-side imaging space CR and the rear imaging space CB, and a rear camera. An area associated with coordinates on the 2B input image plane is mixed. This is to prevent the worker from disappearing in the output image portions R15 and R16.

  On the other hand, if it is determined that there is a person in the left side monitoring space ZL and it is determined that there is no person in the rear side monitoring space ZB, the image generating unit 11 sets the left side camera 2L as the priority camera. The rear camera 2B is a priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the left camera 2L with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB.

  Further, when it is determined that there is a person in the right monitoring space ZR and it is determined that there is no person in the rear monitoring space ZB, the image generation unit 11 sets the right camera 2R as the priority camera, The camera 2B is a priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the right-side camera 2R with the output image portion R16 corresponding to the overlapping portion of the right-side imaging space CR and the rear imaging space CB.

  The output image D12 of FIG. 18 is a side camera generated when it is determined that there is a person in the left side monitoring space ZL and the right side monitoring space ZR, and it is determined that there is no person in the rear monitoring space ZB. The priority viewpoint virtual viewpoint image for surrounding monitoring is shown. In this case, the image generation unit 11 sets the left camera 2L and the right camera 2R as priority cameras, and the rear camera 2B as a priority camera. Then, the image generation means 11 associates the coordinates on the input image plane of the left side camera 2L with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB, and the right side imaging space CR. Coordinates on the input image plane of the right side camera 2L are associated with the output image portion R16 corresponding to the overlapping portion of the rear imaging space CB. As a result, the image generating means 11 associates the coordinates on the input image plane of the left camera 2L with the output image portion R17P, associates the coordinates on the input image plane of the right camera 2R with the output image portion R18P, and outputs the result. The image portion R19N is associated with the coordinates on the input image plane of the rear camera 2B.

  Further, when it is determined that there is no person in the left side monitoring space ZL and it is determined that there is no person in the rear side monitoring space ZB, the image generating unit 11 sets the left side camera 2L as a priority camera, The rear camera 2B is set as the priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB.

  Further, when it is determined that there is no person in the right side monitoring space ZR and it is determined that there is no person in the rear side monitoring space ZB, the image generating unit 11 sets the right side camera 2R as a priority camera, The rear camera 2B is set as the priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB.

  The output image D13 of FIG. 18 is generated when it is determined that there is no person in the left side monitoring space ZL and the right side monitoring space ZR and it is determined that there is no person in the rear monitoring space ZB. The priority viewpoint virtual viewpoint image for surrounding monitoring is shown. In this case, the image generation unit 11 sets each of the left camera 2L and the right camera 2R as a priority camera and the rear camera 2B as a priority camera. Then, the image generation means 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB, and the right side imaging space CR and the rear side. Coordinates on the input image plane of the rear camera 2B are also associated with the output image portion R16 corresponding to the overlapping portion of the imaging space CB. As a result, the image generating means 11 associates the coordinates on the input image plane of the left camera 2L with the output image portion R17N, associates the coordinates on the input image plane of the right camera 2R with the output image portion R18N, and outputs Coordinates on the input image plane of the rear camera 2B are associated with the image portion R19P.

  As shown in FIG. 18, the output image portion R17P corresponds to a portion obtained by adding the output image portion R15 to the output image portion R17N, and the output image portion R18P is obtained by adding the output image portion R16 to the output image portion R18N. It corresponds to the part. The output image portion R19P corresponds to a portion obtained by adding the output image portion R15 and the output image portion R16 to the output image portion R19N.

  FIG. 19 is a correspondence table showing a correspondence relationship between the determination result of the presence / absence determination unit 12 and the content of the output image. A circle indicates that the person is determined to exist by the presence / absence determination means 12, and a mark x indicates that it is determined that no person exists.

  In the pattern A, when it is determined that a person exists only in the left side monitoring space ZL and it is determined that no person exists in the rear side monitoring space ZB and the right side monitoring space ZR, as shown in the output image D12 of FIG. This indicates that a virtual viewpoint image for peripheral monitoring with priority given to the side camera is generated.

  In the pattern B, when it is determined that there is a person only in the right side monitoring space ZR and it is determined that there is no person in the left side monitoring space ZL and the rear monitoring space ZB, as shown in the output image D12 of FIG. This indicates that a virtual viewpoint image for peripheral monitoring with priority given to the side camera is generated.

  The pattern C is a side as shown in the output image D12 of FIG. 18 when it is determined that there is a person in the left side monitoring space ZL and the right side monitoring space ZR and no person is present in the rear monitoring space ZB. This indicates that a virtual viewpoint image for peripheral monitoring with priority given to the direction camera is generated.

  In the pattern D, when it is determined that there is a person in the left side monitoring space ZL and the rear side monitoring space ZB and it is determined that there is no person in the right side monitoring space ZR, the rear side as shown in the output image D13 in FIG. This indicates that a camera-periphered peripheral monitoring virtual viewpoint image is generated.

  Patterns E to G are for preventing image loss as shown in the output image D11 of FIG. 18 when it is determined that a person is present in at least one of the left side monitoring space ZL and the right side monitoring space ZR and the rear monitoring space ZB. This indicates that a virtual viewpoint image for periphery monitoring to which the process is applied is generated.

  In the pattern H, when it is determined that no person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, an image loss prevention process as shown in the output image D11 of FIG. 18 is performed. This indicates that an applied peripheral monitoring virtual viewpoint image is generated.

  With the above configuration, the image generation apparatus 100 switches the content of the output image based on the determination result of the presence / absence determination unit 12. Specifically, when the priority camera and the priority camera cannot be determined, the image generation apparatus 100 generates a peripheral monitoring virtual viewpoint image to which image loss prevention processing is applied. On the other hand, when the image generation unit 11 can determine the priority camera and the priority camera, the image generation unit 11 does not apply the image loss prevention process to the output image portion corresponding to the overlapping portion on the input image plane of the priority camera. A virtual viewpoint image for peripheral monitoring is generated by associating coordinates. As a result, the image generating apparatus 100 can display the workers present in the overlapping portions of the imaging spaces of the two cameras more clearly.

  Further, in the above-described embodiment, the image generation apparatus 100 generates a peripheral viewpoint virtual viewpoint image with priority on the side camera in which the left camera 2L and the right camera 2R are the priority cameras and the rear camera 2B is the priority camera. To do. Alternatively, the image generating apparatus 100 generates a virtual viewpoint image for peripheral monitoring with priority on the rear camera in which the left camera 2L and the right camera 2R are priority cameras and the rear camera 2B is a priority camera. However, the present invention is not limited to this configuration. For example, the image generation device 100 applies the image loss prevention process to the output image portion R16 while associating the coordinates on the input image plane of either the left camera 2L or the rear camera 2B with the output image portion R15. May be. In addition, the image generation apparatus 100 applies the image loss prevention process to the output image portion R15 while associating the coordinates on the input image plane of either the right camera 2R or the rear camera 2B with the output image portion R16. May be.

  The image generation apparatus 100 may combine the first output image switching process and the second output image switching process.

  In the above-described embodiment, the image generation apparatus 100 associates the monitoring space of one human detection sensor with the imaging space of one camera. However, the monitoring space of one human detection sensor is set in the imaging space of a plurality of cameras. You may make it respond | correspond, and you may make the monitoring space of a some human detection sensor respond | correspond to the imaging space of one camera.

  In the above-described embodiment, the image generation apparatus 100 switches the content of the output image at the moment when the determination result of the presence / absence determination unit 12 changes. However, the present invention is not limited to this configuration. For example, the image generation apparatus 100 may set a predetermined delay time from when the determination result of the presence / absence determination unit 12 changes until the content of the output image is switched. This is to suppress frequent switching of the contents of the output image.

  Next, referring to FIG. 20 and FIG. 21, the alarm control unit 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination unit 12 (hereinafter referred to as “first alarm control process”). ). FIG. 20 is a flowchart showing the flow of the first alarm control process, and FIG. 21 shows an example of the transition of the output image displayed during the first alarm control process. The alarm control means 13 repeatedly executes this first alarm control process at a predetermined cycle.

  First, the presence / absence determination unit 12 determines whether or not there is a person around the excavator 60 (step S11). At this time, the image generation unit 11 generates and displays a road surface height reference virtual viewpoint image as shown in an output image D21 of FIG. 21, for example.

  When it is determined that there is a person around the excavator 60 (YES in step S <b> 11), the presence / absence determination unit 12 outputs a detection signal to the alarm control unit 13. Upon receiving the detection signal, the alarm control means 13 outputs an alarm start signal for starting an alarm to the alarm output unit 7, and outputs an alarm from the alarm output unit 7 (step S12).

  In this embodiment, the alarm output unit 7 outputs an alarm sound. Then, for example, as shown in an output image D22 in FIG. 21, the image generation unit 11 switches the road surface height-based periphery monitoring virtual viewpoint image to the head height-based periphery monitoring virtual viewpoint image and displays it. Specifically, the presence / absence determination unit 12 notifies the alarm control unit 13 of a detection signal when it is determined that the worker P1 exists in the rear monitoring space ZB. And the alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear side alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let Further, the image generation means 11 superimposes and displays an alarm stop button G1 on the head height reference periphery monitoring virtual viewpoint image. The alarm stop button G1 is a software button configured by cooperation with a touch panel as the input unit 3. The operator can stop the alarm sound by pressing the alarm stop button G1. Note that the alarm stop button G1 may be a hardware button installed in the vicinity of the display unit 5. In this case, the image generation unit 11 may superimpose and display a text message indicating that the alarm can be stopped by pressing a hardware button as the alarm stop button G1 on the virtual viewpoint image for peripheral monitoring. Further, even when it is determined that there is a person around the excavator 60, the image generation unit 11 directly uses the road viewpoint-based periphery monitoring virtual viewpoint image as shown in the output image D23 of FIG. The alarm stop button G1 may be superimposed and displayed while being used.

  On the other hand, when it is determined that there is no person around the excavator 60 (NO in step S11), the presence / absence determination unit 12 does not output a detection signal to the alarm control unit 13. Therefore, the alarm control means 13 does not output an alarm start signal to the alarm output unit 7. Further, the alarm control means 13 does not output an alarm stop signal for stopping the alarm to the alarm output unit 7 even when the alarm has already been output.

  Thereafter, the alarm control means 13 determines whether or not the alarm stop button G1 has been pressed (step S13). If it is determined that the alarm stop button G1 has been pressed (YES in step S13), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7 and outputs an alarm output from the alarm output unit 7. Stop (step S14). Further, when displaying the head height-based periphery monitoring virtual viewpoint image, the image generation means 11 converts the head height-based periphery monitoring virtual viewpoint image into the road surface height-based periphery monitoring virtual viewpoint image. Switch to the image for display.

  On the other hand, if it is determined that the alarm stop button G1 is not pressed (NO in step S13), the alarm control means 13 does not output an alarm stop signal to the alarm output unit 7. That is, the alarm output by the alarm control unit 13 does not stop until the alarm stop button G1 is pressed even if the human presence determination unit 12 subsequently determines that there is no person around the excavator 60. . In addition, when the head height-based periphery monitoring virtual viewpoint image is displayed, the image generation unit 11 continues to display the head height-based periphery monitoring virtual viewpoint image.

  The alarm control means 13 changes the content of the alarm when the presence / absence determination means 12 determines that there is no person around the excavator 60 even before the alarm stop button G1 is pressed. May be. Specifically, the alarm control means 13 may change the intensity, level, output interval, etc. of the alarm sound that has already been output, or the intensity, emission color, emission interval of the alarm lamp that has already emitted light. Etc. may be changed. This is because the operator who receives the warning can distinguish between the case where it is determined by the presence / absence determination unit 12 that a person is present and the case where it is determined that there is no person.

  With the above configuration, the image generation apparatus 100 outputs an alarm when it is determined that there is a person in the monitoring space, and does not stop the alarm until the operator indicates an intention to stop the alarm. That is, the image generating apparatus 100 stops the alarm even when a person who exists in the monitoring space leaves the monitoring space or fails to detect a person existing in the monitoring space. There is nothing. Therefore, the image generating apparatus 100 can prompt the operator to confirm that the worker who was present in the monitoring space around the excavator 60 has left the monitoring space.

  Further, the image generation apparatus 100 stops the alarm in response to the worker being stationary in the monitoring space in the configuration in which the human presence determination unit 12 determines the presence or absence of the worker (moving body) based on the moving body detection sensor. Can be prevented. Further, the image generation apparatus 100 is configured such that the presence / absence determination unit 12 uses the optical flow to determine the presence / absence of the worker (moving object), and stops the alarm in response to the worker being stationary in the monitoring space. Can be prevented. Note that the moving object detection sensor and the optical flow do not set a stationary object as a detection target, and do not distinguish whether the worker has stopped in the monitoring space and that the worker has left the monitoring space. Therefore, in the configuration in which the alarm is stopped when it is determined that there is no person, there is a possibility that the alarm may be stopped despite the presence of the worker in the monitoring space. On the other hand, the image generating apparatus 100 can alert the operator of the excavator 60 by not stopping the alarm when there is a possibility that an operator exists in the monitoring space.

  Next, referring to FIGS. 22 to 25, the alarm control unit 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination unit 12 and the determination result of the work machine state determination unit 14 ( Hereinafter, “second alarm control processing” will be described. FIG. 22 is a flowchart showing the flow of the second alarm control process, and FIG. 23 shows an example of the transition of the output image displayed during the second alarm control process. FIG. 24 is a flowchart showing a flow of a process in which the work machine state determination means 14 determines the state of the work machine (hereinafter referred to as “work machine state determination process”), and FIG. 5 is a flowchart showing a flow of a process (hereinafter referred to as “work start alarm control process”) in which the alarm control means 13 controls the alarm output unit 7 at the time of switching from a workable state to a workable state. The alarm control unit 13 repeatedly executes the second alarm control process at a predetermined cycle, and the work machine state determination unit 14 repeatedly executes the work machine state determination process at a predetermined cycle. Moreover, the alarm control means 13 performs a work start time alarm control process at the time of switching from the work impossible state to the work available state.

  First, the person presence / absence determination means 12 determines whether or not there is a person around the excavator 60 (step S21). At this time, when the display unit 5 is activated, the image generation unit 11 generates and displays a road viewpoint-based periphery monitoring virtual viewpoint image as shown in the output image D31 of FIG. 23, for example. .

  When it is determined that there is a person around the excavator 60 (YES in step S21), the presence / absence determination unit 12 refers to the result of determination by the work machine state determination unit 14 (see FIG. 24 described later) (step S22). ). In this case, the presence / absence determination means 12 may fix the gate lock lever 8 in the locked state. Specifically, the presence / absence determination means 12 may prevent the operator from pulling up the gate lock lever 8 and may prevent the lock release state even if the gate lock lever 8 is pulled up. .

  When it is determined by the work machine state determination unit 14 that the excavator 60 is in a workable state (YES in step S22), the human presence determination unit 12 outputs a detection signal to the alarm control unit 13. Upon receiving the detection signal, the alarm control means 13 outputs an alarm start signal for outputting an alarm to the alarm output unit 7, and outputs an alarm from the alarm output unit 7 (step S23).

  In this embodiment, the alarm output unit 7 outputs an alarm sound. Specifically, the presence / absence determination unit 12 outputs a left side detection signal to the alarm control unit 13 when it is determined that a person (here, workers P10 to P12) exists in the left side monitoring space ZL. Notice. And the alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear side alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let In this case, the control unit 1 activates the display unit 5 when the display unit 5 is not activated. Then, as shown in the output image D22 of FIG. 21, the image generation means 11 switches the road surface height-based periphery monitoring virtual viewpoint image to the head height-based periphery monitoring virtual viewpoint image and presses the alarm stop button G1. Superimposed display. Further, as shown in the output image D23 of FIG. 21, the image generation means 11 may superimpose and display the alarm stop button G1 on the periphery monitoring virtual viewpoint image based on the road surface height. In this case, the alarm control means 13 may prohibit the work by the shovel 60 until the alarm stop button G1 is pressed. Specifically, the alarm control means 13 may close the gate lock valve, shut off the flow of hydraulic oil between the control valve and the operation lever, and invalidate the operation lever.

  When it is determined by the work machine state determination unit 14 that the excavator 60 is not in a workable state (NO in step S22), the human presence determination unit 12 does not output a detection signal to the alarm control unit 13, The value of the human detection flag prepared in the NVRAM or the like is set to “1” (ON) (step S24). The person detection flag has an initial value “0” (off), and the detection flag value “1” (on) indicates that a person has been detected, and the detection flag value “0” ( (Off) indicates that no person is detected. At this time, since the alarm control means 13 does not receive the detection signal, the alarm start signal is not output to the alarm output unit 7 and the alarm output unit 7 does not output an alarm. Further, the alarm control means 13 may output an alarm stop signal to the alarm output unit 7 when the alarm start signal has already been output to the alarm output unit 7. Further, when the display unit 5 is activated, the image generation unit 11 is the same as the case where it is determined that there is no person even though it is determined that there is a person around the excavator 60. Display the output image. Specifically, the image generation unit 11 displays a virtual viewpoint image for periphery monitoring based on the road surface height, for example, as shown in the output image D32 of FIG. However, the image generation means 11 may switch the road viewpoint-based periphery monitoring virtual viewpoint image to the head height-based periphery monitoring virtual viewpoint image.

  On the other hand, when it is determined that there is no person around the excavator 60 (NO in step S21), the presence / absence determination unit 12 does not refer to the determination result by the work machine state determination unit 14, and the alarm control unit 13 No detection signal is output.

  In this case, when the alarm has already been output, the image generation unit 11 is the same as the case where it is determined that there is a person even though it is determined that there is no person around the excavator 60. Display the output image. Specifically, the image generation unit 11 displays a head height reference periphery monitoring virtual viewpoint image, for example, as shown in the output image D33 of FIG. 23 together with the alarm stop button G1. However, the image generation means 11 may switch the periphery monitoring virtual viewpoint image based on the head height to a road monitoring based viewpoint visual viewpoint image as shown in the output image D34 of FIG.

  Thereafter, when the alarm has already been output, the alarm control means 13 monitors whether or not the alarm stop button G1 has been pressed (step S25). If it is determined that the alarm stop button G1 has been pressed (YES in step S25), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7 and outputs an alarm output from the alarm output unit 7. Stop (step S26). In addition, when the image generation means 11 displays the head height-based periphery monitoring virtual viewpoint image (see the output image D33), the head height-based periphery monitoring virtual viewpoint image is displayed on the road surface height. It switches to the reference | standard periphery monitoring virtual viewpoint image (refer the output image D31), and displays it.

  On the other hand, if it is determined that the alarm stop button G1 has not been pressed (NO in step S25), the alarm control means 13 does not output an alarm stop signal to the alarm output unit 7. That is, the alarm output by the alarm control unit 13 does not stop until the alarm stop button G1 is pressed even if the human presence determination unit 12 subsequently determines that there is no person around the excavator 60. . Further, when displaying the head height-based periphery monitoring virtual viewpoint image (see the output image D33), the image generation means 11 displays the head height-based periphery monitoring virtual viewpoint image. Let it continue.

  The alarm control means 13 changes the content of the alarm when the presence / absence determination means 12 determines that there is no person around the excavator 60 even before the alarm stop button G1 is pressed. May be. Specifically, the alarm control means 13 may change the intensity, level, output interval, etc. of the alarm sound that has already been output, or the intensity, emission color, emission interval of the alarm lamp that has already emitted light. Etc. may be changed. This is because the operator who receives the warning can distinguish between the case where it is determined by the presence / absence determination unit 12 that a person is present and the case where it is determined that there is no person.

  Next, with reference to FIG. 24, work machine state determination processing by the work machine state determination unit 14 will be described.

  First, the work machine state determination means 14 determines whether or not the ignition switch 9 is in an ON state based on the output of the ignition switch 9 (step S31).

  When it is determined that the ignition switch 9 is in the on state (YES in step S31), the work machine state determination unit 14 determines whether the gate lock lever 8 is in the unlocked state based on the output of the gate lock lever 8. Determination is made (step S32).

  When it is determined that the gate lock lever 8 is in the unlocked state (YES in step S32), the work machine state determining unit 14 determines that the excavator 60 is in a workable state (step S33).

  On the other hand, when it is determined that the ignition switch 9 is not in the ON state (NO in step S31), or when it is determined that the gate lock lever 8 is not in the unlocked state (NO in step S32), the work machine state determination means 14 Determines that the excavator 60 is not in a workable state, that is, the excavator 60 is in a work-impossible state (step S34).

  Thereafter, the work machine state determination unit 14 sets the value of the work enable flag prepared in the NVRAM or the like according to the determination result. The presence / absence determination means 12 refers to the value of the work enable flag in step S22 of the second alarm control process shown in FIG. 22, and determines the presence or absence of an alarm output based on the referred value. Note that the work available flag has an initial value of “0” (off), and the work available flag value “1” (on) indicates that work is possible, and the work available flag value “0”. "(Off) indicates that the work is not possible.

  The work machine state determination means 14 may determine whether or not the excavator 60 is in a workable state based only on the state of either the gate lock lever 8 or the ignition switch 9.

  Next, with reference to FIG. 25, the alarm control process at the time of the work start by the alarm control means 13 is demonstrated.

  First, the alarm control means 13 refers to the value of the person detection flag (step S41). When it is determined that the value of the human detection flag is “1” ON (YES in step S41), the alarm control unit 13 outputs an alarm start signal to the alarm output unit 7, and the alarm output unit 7 An alarm is output (step S42). Specifically, the alarm control unit 13 outputs an alarm even when the presence / absence determination unit 12 determines that there is no person around the excavator 60 at the present time. In this embodiment, the alarm control means 13 outputs an alarm start signal to the left alarm output unit 7L, the rear alarm output unit 7B, and the right alarm output unit 7R, and the alarm is output from all three alarm output units. Output sound. In this case, the alarm control means 13 may prohibit the work by the excavator 60 until the alarm output is stopped. Specifically, the alarm control means 13 may close the gate lock valve, shut off the flow of hydraulic oil between the control valve and the operation lever, and invalidate the operation lever.

  Thereafter, the alarm control means 13 measures the elapsed time after the alarm is started, and determines whether or not a predetermined time (for example, 2 seconds) has elapsed after the alarm is started (step S43).

  If it is determined that the predetermined time has not elapsed after the alarm is started (NO in step S43), the alarm control unit 13 waits until the predetermined time elapses.

  On the other hand, when it is determined that a predetermined time has elapsed after the start of the alarm (YES in step S43), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7 and outputs an alarm from the alarm output unit 7 Is stopped (step S44). Further, the alarm control means 13 resets the value of the person detection flag to “0” (off) (step S45). The alarm control unit 13 may not stop the alarm until the alarm stop button G1 is pressed.

  With the above configuration, the image generation apparatus 100 does not output an alarm when the excavator 60 is in a work-impossible state even when it is determined that there is a person in the monitoring space. Therefore, the image generating apparatus 100 can prevent an unnecessary alarm from being output when the excavator 60 is in a work-impossible state.

  On the other hand, even when the image generating apparatus 100 determines that there is no person in the monitoring space at the present time, the worker has entered the monitoring range while the excavator 60 is in an unworkable state. Outputs an alarm. Therefore, the image generating apparatus 100 can prompt the operator to confirm that the worker who was present in the monitoring space around the excavator 60 has left the monitoring space.

  Also, in the above-described work start time alarm control process, the alarm control means 13 outputs an alarm as long as the value of the person detection flag is “1” (ON). However, the present invention is not limited to this configuration. For example, the alarm control means 13 may not output an alarm if the elapsed time from the time when the value of the human detection flag is set to “1” (ON) is a predetermined time or more. Alternatively, the presence / absence determination unit 12 sets the value of the human detection flag to “0” (off) when the elapsed time from the time when the value of the human detection flag is set to “1” (on) reaches a predetermined time. It may be reset. This is to prevent an alarm from being output based on a too old detection record.

  Further, when the presence / absence determination unit 12 determines the presence / absence of the worker (moving body) based on the moving body detection sensor, the image generation apparatus 100 enters the monitoring space and stops still when the excavator 60 is in a work impossible state. It is possible to prevent the worker who is working from being detected when the excavator 60 is ready to work and the operator of the excavator 60 starts working with the excavator 60 without noticing the worker.

  In addition, when the presence / absence determination unit 12 determines the presence / absence of the worker (moving object) using the optical flow, the image generation apparatus 100 enters the monitoring space and stops still when the excavator 60 is in a work-impossible state. It is possible to prevent the worker who is working from being detected when the excavator 60 is ready to work and the operator of the excavator 60 starts working with the excavator 60 without noticing the worker.

  Note that the moving object detection sensor and the optical flow do not set a stationary object as a detection target, and do not distinguish whether the worker has stopped in the monitoring space and that the worker has left the monitoring space. Therefore, when it is determined that there is no person in the monitoring space at the present time, in a configuration in which an alarm is not output in any case, the work that enters the monitoring space and is stationary before the excavator 60 enters the workable state. There is a risk that the excavator 60 may be allowed to work despite the presence of a person. On the other hand, the image generating apparatus 100 can alert the operator of the excavator 60 by outputting an alarm when there is a possibility that an operator exists in the monitoring space.

  Further, in the second alarm control process described above, the image generation apparatus 100 displays the output image on the display unit 5 even when the excavator 60 is in a work-impossible state, but does not display the output image on the display unit 5. May be. This is because when the excavator 60 is in a work-impossible state, there is no possibility that the excavator 60 and the operator come into contact with each other, and the operator does not need to monitor the periphery of the excavator 60 through the output image.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the image generation apparatus 100 employs the cylindrical spatial model MD as the spatial model, but may employ a spatial model having other columnar shapes such as a polygonal column, A spatial model composed of two side surfaces may be adopted, or a spatial model having only side surfaces may be adopted.

  The image generating apparatus 100 is mounted on a self-propelled excavator with movable members such as a bucket, an arm, a boom, and a turning mechanism together with a camera and a human detection sensor. The image generating apparatus 100 constitutes an operation support system that supports the movement of the shovel and the operation of these movable members while presenting the surrounding image to the operator. However, the image generation apparatus 100 may be mounted together with a camera and a human detection sensor on a work machine that does not have a turning mechanism such as a forklift or an asphalt finisher. Alternatively, the image generation apparatus 100 may be mounted together with a camera and a human detection sensor on a work machine that has a movable member but does not self-propel, such as an industrial machine or a fixed crane. The image generation apparatus 100 may constitute an operation support system that supports the operation of these work machines.

  Further, the periphery monitoring device has been described by taking the image generation device 100 including the camera 2 and the display unit 5 as an example, but may be configured as a device that does not include an image display function by the camera 2, the display unit 5, and the like. For example, as shown in FIG. 26, the periphery monitoring device 100A as a device that executes the first alarm control process includes a camera 2, a storage unit 4, a display unit 5, a gate lock lever 8, an ignition switch 9, and coordinate matching means. 10, the image generation means 11 and the work machine state determination means 14 may be omitted. Further, as shown in FIG. 27, the periphery monitoring device 100B as a device that executes the second voice transmission support process and the work start warning control process includes a camera 2, an input unit 3, a storage unit 4, a display unit 5, and coordinates. The association unit 10 and the image generation unit 11 may be omitted.

  DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input part 4 ... Memory | storage part 5 ... Display part 6 .... -Human detection sensor 6L ... Left side person detection sensor 6R ... Right side person detection sensor 6B ... Back person detection sensor 7 ... Alarm output unit 7L ... Left side alarm output unit 7B ... Rear side warning output unit 7R ... right side warning output unit 8 ... gate lock lever 9 ... ignition switch 10 ... coordinate association means 11 ... image generation means 12 ... human existence determination Means 13: Alarm control means 14 ... Work machine state determination means 40 ... Input image / space model correspondence map 41 ... Spatial model / processing object image correspondence map 42 ... Processing object image / output image Compatible maps 60 ... excavator 61 ... lower traveling body 62 ... turning mechanism 63 ... upper swing body 64 ... cab 100 ... image generating apparatus 100A, 100B ... environment monitoring device

Claims (5)

A work machine periphery monitoring device including an alarm output unit for outputting an alarm to an operator,
Work machine state determination means for determining the state of the work machine;
Human presence determination means for determining the presence or absence of a person around the work machine;
Alarm control means for controlling the alarm output unit,
When it is determined that the work machine is not in a workable state and it is determined that there is a person around the work machine, the alarm control unit determines that the work machine is in a workable state thereafter. An alarm is output when
Perimeter monitoring device for work machines. The presence / absence determination means determines the presence / absence of a person around the work machine by detecting a moving object based on an output of a moving object detection sensor or an image sensor attached to the work machine.
The work machine periphery monitoring device according to claim 1. The work machine state determining means determines that the work machine is not in a workable state when the ignition switch is in an off state, or when the gate lock lever is in a locked state, and the ignition switch is in an on state. And when the gate lock lever is unlocked, it is determined that the work machine is in a workable state.
The periphery monitoring apparatus for work machines according to claim 1 or 2. When the warning control means determines that the work machine is not in a workable state and it is determined that there is a person around the work machine, the work machine is determined to be in a workable state. The content of the alarm to be output is different from the content of the alarm to be output when it is determined that the work machine is in a workable state and it is determined that there is a person around the work machine.
The work machine periphery monitoring device according to any one of claims 1 to 3. The alarm control means determines that the work machine is not in a workable state, and the work machine is in a workable state before a predetermined time elapses after it is determined that there is a person around the work machine. Output an alarm when it is determined that
The work machine periphery monitoring device according to any one of claims 1 to 4.