WO2014148202A1

WO2014148202A1 - Periphery monitoring device for work machine

Info

Publication number: WO2014148202A1
Application number: PCT/JP2014/054287
Authority: WO
Inventors: 芳永清田
Original assignee: 住友重機械工業株式会社
Priority date: 2013-03-19
Filing date: 2014-02-24
Publication date: 2014-09-25
Also published as: JP6352592B2; JP2014181510A

Abstract

A periphery monitoring device (100A) for a shovel (60) provided with a notification unit (20) capable of being seen from a periphery is provided with: a work-machine-state determination means (14) for determining whether the shovel (60) is in a state in which work can be performed; a person's presence determination means (12) for determining whether a person is present in the periphery of the shovel (60); a notification-unit control means (15) for controlling the notification unit (20); and a work-permision determination means (16) for determining whether the shovel (60) is permitted to perform work. The notification-unit control means (15) sets a notification state of the notification unit (20) such that the notification state in cases when it is determined that the shovel (60) is in a state in which work can be performed, is different to the notification state in cases when it is determined that the shovel (60) is not in a state in which work can be performed. The work-permission determination means (16) determines, on the basis of a determination result from the person's presence determination means (12), whether the shovel (60) is permitted to perform work.

Description

Peripheral monitoring device for work machine

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to a work machine peripheral monitoring device having a function of notifying people around the work machine of its state.

DESCRIPTION OF RELATED ART The hydraulic excavator provided with the indicator for notifying the surrounding person of the starting state, turning state, driving | running | working state, etc. of a hydraulic excavator is known (for example, refer patent document 1).

JP 2002-327469 A

However, the display device of Patent Document 1 merely informs the status of the hydraulic excavator from the hydraulic excavator side to surrounding people. Therefore, a worker who works around the hydraulic excavator is limited to the fact that the hydraulic excavator or its operator is aware of the presence of the operator even if the status of the hydraulic excavator is known by looking at the indicator. There is no reason to continue working safely around the hydraulic excavator. Thus, in the indicator of Patent Document 1, the state of the hydraulic excavator when the operator of the hydraulic excavator notices the presence of the operator or when the hydraulic excavator and the operator approach each other The information that the operator wants to know, such as

In view of the above-mentioned points, it is desirable to provide a work machine peripheral monitoring device capable of more appropriately informing a person present around the work machine the state of the work machine.

The work machine peripheral monitoring device according to an embodiment of the present invention is a work machine peripheral monitoring device provided with a notification unit visible from the periphery of the work machine, and whether or not the work machine is in a workable state Work machine state judgment means for judging, person presence / absence judgment means for judging presence / absence of a person around the work machine, notification unit control means for controlling the notification unit, and work for determining the permission / prohibition of work by the work machine The notification unit control means is configured to determine whether the work machine is determined to be in the work enable state or not determined to be in the work enable state; The notification state is changed to a different state, and the work permission determination means determines the permission of the work by the work machine based on the determination result of the person existence determination means.

According to the above-described means, the work machine peripheral monitoring device can be provided which can more appropriately notify the person present around the work machine the state of the work machine.

It is a block diagram showing roughly an example of composition of an image generation device concerning an example of the present invention. It is a figure which shows the structural example of the shovel by which an image generation apparatus is mounted. It is a figure which shows an example of the space model by which an input image is projected. FIG. 6 is a diagram showing an example of a relationship between a space model and a processing target image plane. It is a figure for demonstrating matching with the coordinate on an input image plane, and the coordinate on a space model. It is a figure for demonstrating the matching between the coordinates by a coordinate matching means. It is a figure for demonstrating the effect | action of a parallel line group. It is a figure for demonstrating the effect | action of an auxiliary line group. It is a flowchart which shows the flow of a process target image generation process and an output image generation process. It is an example of an output image. It is a top view of a shovel by which an image generating device is carried. It is an example of the figure which shows the input image of each of 3 cameras mounted in the shovel, and the output image produced | generated using those input images. It is a figure for demonstrating the image loss prevention process which prevents the loss | disappearance of the object in the overlapping part of imaging space of each of two cameras. FIG. 13 is a contrast diagram showing the difference between the output image shown in FIG. 12 and the output image obtained by applying the image loss prevention process to the output image of FIG. 12. It is another example of the figure which shows the input image of each of 3 cameras mounted in the shovel, and the output image produced | generated using those input images. FIG. 7 shows another example of the relationship between the spatial model and the image plane to be processed. It is a figure explaining the relation of two output pictures switched by the 1st output picture change processing. It is a figure explaining the relationship of three output images switched by 2nd output image switching process. It is a corresponding | compatible table which shows the correspondence of the determination result of a person existence determination means, and the content of an output image. It is a flow chart which shows the flow of the 1st alarm control processing. It is a figure which shows an example of transition of the output image displayed during 1st alarm control processing. It is a flow chart which shows the flow of the 2nd alarm control processing. It is a figure which shows an example of transition of the output image displayed during a 2nd alarm control process. It is a flow chart which shows a flow of work machine state judging processing. It is a flowchart which shows the flow of alarm control processing at the time of work start. It is a flowchart which shows the flow of a 1st alerting | reporting part control process. It is a flowchart which shows the flow of work permission decision processing. It is a flowchart which shows the flow of 2nd alerting | reporting part control processing. It is a block diagram which shows roughly another structural example of a periphery monitoring apparatus.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generation apparatus 100 according to an embodiment of the present invention.

The image generation device 100 is an example of a work machine peripheral monitoring device that monitors the periphery of a work machine, and includes a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, a human detection sensor 6, and an alarm. The output unit 7, the gate lock lever 8, the ignition switch 9, and the notification unit 20 are provided. Specifically, the image generating apparatus 100 generates an output image based on an input image captured by the camera 2 mounted on a work machine, and presents the output image to the operator. Further, the image generation device 100 switches the content of the output image to be presented based on the output of the human detection sensor 6.

FIG. 2 is a view showing a configuration example of a shovel 60 as a working machine on which the image generating apparatus 100 is mounted. The shovel 60 is an upper portion of a crawler type lower traveling body 61 via a turning mechanism 62. The swing body 63 is rotatably mounted around the swing axis PV.

The upper swing body 63 also has a cab (driver's cab) 64 on the front left side thereof and an excavating attachment E at the front center thereof, and the camera 2 (right side camera 2R, rear camera 2B on its right side and rear side) And a notification unit 20 (a left direction notification unit 20L, a right direction notification unit 20R, and a rear notification unit 20B). In addition, the display part 5 is installed in the position in which the operator in the cab 64 visually recognizes. In the cab 64, an alarm output unit 7 (right side alarm output unit 7R, rear alarm output unit 7B), a gate lock lever 8 and an ignition switch 9 are installed.

Next, each component of the image generation apparatus 100 will be described.

The control unit 1 is a computer provided with a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile random access memory (NVRAM), and the like. In the present embodiment, the control unit 1 includes, for example, coordinate association unit 10, image generation unit 11, person existence determination unit 12, alarm control unit 13, work machine condition determination unit 14, notification unit control unit 15, and the like described later. A program corresponding to each of the work permission determination means 16 is stored in the ROM or NVRAM, and the CPU is caused to execute processing corresponding to each means while using the RAM as a temporary storage area.

The camera 2 is a device for acquiring an input image that reflects the surroundings of the shovel 60. In the present embodiment, the camera 2 is, for example, the right side camera 2R and the rear camera 2B attached to the right side surface and the rear surface of the upper swing body 63 so as to be able to capture an area which becomes a blind spot of the operator in the cab 64 (see FIG. 2)). In addition, the camera 2 includes an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 2 may be attached to a position other than the right side surface and the rear surface of the upper swing body 63 (for example, the front surface and the left side surface), and a wide angle lens or a fisheye lens is attached to capture a wide range. It may be

Further, the camera 2 acquires an input image according to a control signal from the control unit 1, and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the corrected input image corrected for apparent distortion and tilt caused by using these lenses is sent to the control unit 1. Output. In addition, the camera 2 may output an input image not corrected for the apparent distortion or tilt to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion or tilt.

The input unit 3 is a device for enabling an operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

The storage unit 4 is a device for storing various information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

The display unit 5 is a device for displaying image information, and is, for example, a liquid crystal display or a projector installed in a cab 64 (see FIG. 2) of the shovel 60. Display an image.

The human detection sensor 6 is a device for detecting a person present around the shovel 60. In the present embodiment, the human detection sensor 6 is attached to the right side surface and the rear surface of the upper swing body 63, for example, so as to be able to detect a person present in an area in the cab 64 which will be the blind spot of the operator (see FIG. 2). .

The human detection sensor 6 is a sensor that distinguishes and detects a human from an object other than a human, for example, a sensor that detects an energy change in the corresponding monitoring space, and includes a pyroelectric infrared sensor, a bolometer infrared sensor, It includes a moving body detection sensor using an output signal of an infrared camera or the like. In the present embodiment, the human detection sensor 6 uses a pyroelectric infrared sensor, and detects a moving body (moving heat source) as a human. Also, the monitoring space of the right side person detection sensor 6R is included in the imaging space of the right side camera, and the monitoring space of the rear person detection sensor 6B is included in the imaging space of the rear camera 2B.

The person detection sensor 6 may be attached to a position other than the right side surface and the rear surface (for example, the front surface and the left surface) of the upper swing body 63 as in the camera 2. It may be attached to any one of the left side surface, the right side surface, and the rear surface, and may be attached to all the surfaces.

The alarm output unit 7 is a device that outputs an alarm to the operator of the shovel 60. For example, the alarm output unit 7 is an alarm device that outputs at least one of sound and light, and includes a sound output device such as a buzzer and a speaker, and a light emitting device such as an LED and a flash light. In the present embodiment, the alarm output unit 7 is a buzzer for outputting an alarm sound, and the right side alarm output unit 7R attached to the right inner wall of the cab 64 and the rear alarm output unit attached to the rear inner wall of the cab 64 7B (see FIG. 2).

The gate lock lever 8 is a device that switches the state of the shovel 60. In the present embodiment, the gate lock lever 8 has a locked state in which the shovel 60 can not be operated and a unlocked state in which the shovel 60 can be operated. Note that the “work enable state” means a state in which the operator can operate the shovel 60, and the “work inoperable state” means a state in which the operator can not operate the shovel 60. The gate lock lever 8 repeatedly outputs a signal representing the current state of itself to the control unit 1 at a predetermined cycle.

Specifically, the operator pulls up the gate lock lever 8 to make it substantially horizontal to bring the gate lock lever 8 into the unlocked state, and pushes the gate lock lever 8 into the locked state by pushing it down.

In the unlocked state, the gate lock lever 8 blocks the entry / exit opening of the cab 64 to restrict an operator's withdrawal from the cab 64, and an operation lever and an operation pedal (not shown) in the cab 64 (hereinafter referred to as “operation Operation by the lever etc.) is enabled to allow the operator to operate the shovel 60.

On the other hand, in the locked state, the gate lock lever 8 opens the entrance of the cab 64 and allows the operator to withdraw from the cab 64 while invalidating the operation by the operation lever or the like in the cab 64 to operate the operator. Prevents the shovel 60 from operating.

More specifically, the gate lock lever 8 closes the gate lock valve in the locked state, and opens the gate lock valve in the unlocked state. The gate lock valve is a switching valve provided between an oil passage between a control valve (not shown) and an operating lever or the like. The control valve is a flow control valve that controls the flow of hydraulic fluid between a hydraulic pump (not shown) and various hydraulic actuators. In the closed state, the gate lock valve shuts off the flow of hydraulic fluid between the control valve and the operation lever or the like to invalidate the operation lever or the like. Further, in the open state, the gate lock valve allows hydraulic fluid to communicate between the control valve and the operation lever or the like to make the operation lever or the like effective.

The ignition switch 9 is a device that switches the state of the shovel 60. In the present embodiment, the ignition switch 9 has an off state in which the shovel 60 is in an operation impossible state, and an on state in which the shovel 60 is in an operation enabled state. The ignition switch 9 repeatedly outputs a signal representing the current state of itself to the control unit 1 at a predetermined cycle.

Specifically, the operator switches the ignition switch 9 between the on state and the off state by pressing the ignition switch 9.

In the on state, the ignition switch 9 starts the control pump to supply the hydraulic fluid to the oil passage between the control valve and the operation lever or the like by starting the engine. Then, the ignition switch 9 enables an operation of the shovel 60 by enabling an operation by an operation lever or the like in the cab 64.

On the other hand, in the off state, the ignition switch 9 stops the control pump by stopping the engine. Then, the ignition switch 9 invalidates the operation by the operation lever or the like in the cab 64 so that the operator can not operate the shovel 60.

The notification unit 20 is a device that notifies the surrounding people of the state of the shovel 60. In the present embodiment, the notification unit 20 is an indicator lamp configured by an LED, a flash light, etc., and is attached to the left side surface, the right side surface, and the rear surface of the upper swing body 63 so as to be visible to surrounding people (FIG. 2). reference.). In addition, the notification unit 20 may be attached to an arbitrary position such as the outer surface of the shovel 60 including the front surface of the upper swing body 63 or the like so as to be visible to the surrounding people. In addition, the notification unit 20 may be configured to be attached to the camera 2.

Further, the image generation apparatus 100 generates the processing target image based on the input image, and performs image conversion processing on the processing target image so that the positional relationship with the surrounding objects and the sense of distance can be intuitively grasped. After the output image to be generated is generated, the output image may be presented to the operator.

The “processing target image” is an image to be generated based on an input image and to be a target of image conversion processing (for example, scale conversion processing, affine conversion processing, distortion conversion processing, viewpoint conversion processing, and the like). Specifically, the “processing target image” is, for example, an input image by a camera that captures the ground surface from above, and an input image including an image in the horizontal direction (for example, a sky part) due to its wide angle of view. An image suitable for image conversion processing, generated from More specifically, the input image is projected onto a predetermined spatial model so that the horizontal image is not displayed unnaturally (for example, the sky portion is not treated as being on the ground surface). It is generated by reprojecting the projection image projected on the space model to another two-dimensional plane. Note that the processing target image may be used as an output image as it is without performing image conversion processing.

The "space model" is a projection target of the input image. Specifically, the “space model” is configured by one or more planes or curved surfaces including at least a plane or curved surface other than the processing target image plane which is a plane on which the processing target image is located. A plane or a curved surface other than the processing target image plane which is a plane on which the processing target image is located is, for example, a plane parallel to the processing target image plane or a plane or a curved surface forming an angle with the processing target image plane. .

The image generation apparatus 100 may generate an output image by performing an image conversion process on the projection image projected on the space model without generating the processing target image. In addition, the projection image may be used as an output image as it is without performing image conversion processing.

FIG. 3 is a view showing an example of a space model MD on which an input image is projected, and the left view of FIG. 3 shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from the side. 3 shows the relationship between the shovel 60 and the space model MD when the shovel 60 is viewed from above.

As shown in FIG. 3, the space model MD has a semi-cylindrical shape, and has a flat region R1 inside the bottom and a curved region R2 inside the side.

FIG. 4 is a diagram showing an example of the relationship between the space model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the space model MD. Although FIG. 4 shows the space model MD not in a semi-cylindrical shape as shown in FIG. 3 but in a cylindrical shape for clarity, the space model MD has either a semi-cylindrical shape or a cylindrical shape. It may be The same applies to the following figures. Further, as described above, the processing target image plane R3 may be a circular area including the plane area R1 of the space model MD, or may be an annular area not including the plane area R1 of the space model MD.

Next, various units included in the control unit 1 will be described.

The coordinate associating means 10 is a means for associating the coordinates on the input image plane where the input image captured by the camera 2 is located, the coordinates on the space model MD, and the coordinates on the processing object image plane R3. In the present embodiment, the coordinate associating unit 10 includes, for example, various parameters related to the camera 2 set in advance or input through the input unit 3, and an input image plane, a space model MD, and the like, which are determined in advance. The coordinates on the input image plane, the coordinates on the space model MD, and the coordinates on the processing image plane R3 are associated based on the mutual positional relationship between the processing object image plane R3. The various parameters relating to the camera 2 are, for example, the optical center of the camera 2, focal length, CCD size, optical axis direction vector, camera horizontal direction vector, projection method and the like. Then, the coordinate correlating means 10 stores the correspondence between them in the input image / space model correspondence map 40 of the storage unit 4 and the space model / processing target image correspondence map 41.

Note that, when the coordinate association unit 10 does not generate the processing target image, the association between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the space model of the corresponding relationship and the processing target The storage in the image correspondence map 41 is omitted.

The image generation unit 11 is a unit for generating an output image. In the present embodiment, the image generation unit 11 performs, for example, scale conversion, affine conversion, or distortion conversion on the processing target image to obtain the coordinates on the processing target image plane R3 and the output image on which the output image is located. Correspond to the coordinates. Then, the image generation unit 11 stores the correspondence in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / process target image correspondence map 41. Generate an output image. The value of each pixel is, for example, a luminance value, a hue value, a saturation value or the like.

Further, the image generation unit 11 outputs an image plane on which coordinates on the image plane R3 to be processed and an output image are located based on various parameters related to the virtual camera set in advance or input through the input unit 3. Correspond with the upper coordinates. The various parameters relating to the virtual camera are, for example, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the image generation unit 11 stores the correspondence in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / process target image correspondence map 41. Generate an output image.

The image generation unit 11 may generate the output image by changing the scale of the processing target image without using the concept of the virtual camera.

Further, when the image to be processed is not generated, the image generation unit 11 associates the coordinates on the space model MD with the coordinates on the output image plane according to the applied image conversion processing. Then, while referring to the input image / space model correspondence map 40, the image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image to generate an output image. In this case, the image generation unit 11 omits the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane and the storage of the correspondence in the processing target image / output image correspondence map 42. .

Further, the image generation unit 11 switches the content of the output image based on the determination result of the person presence / absence determination unit 12. The details of the switching of the output image by the image generation unit 11 will be described later.

The human presence / absence determination means 12 is a means for determining the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine. In the present embodiment, the human presence / absence determination means 12 determines the presence / absence of a person around the shovel 60 based on the output of the human detection sensor 6.

In addition, the human presence / absence determination means 12 may determine the presence / absence of a person in each of a plurality of monitoring spaces set around the work machine based on the input image captured by the camera 2. Specifically, the human presence / absence judgment means 12 may judge presence / absence of a person around the work machine using an image processing technology such as optical flow, pattern matching and the like. The human presence / absence determination means 12 may determine the presence / absence of a person around the work machine based on the output of an image sensor other than the camera 2.

Alternatively, the human presence / absence determination means 12 determines the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine based on the output of the human detection sensor 6 and the output of the image sensor such as the camera 2 It is also good.

The alarm control unit 13 is a unit that controls the alarm output unit 7. In the present embodiment, the alarm control means 13 outputs an alarm based on the judgment result of the person existence / nonexistence judgment means 12 or based on the judgment result of the person existence / nonexistence judgment means 12 and the judgment result of the work machine state judgment means 14 Control unit 7; The control of the alarm output unit 7 by the alarm control means 13 will be described in detail later.

The working machine state determination means 14 is a means for determining the state of the working machine. In the present embodiment, the work machine state determination means 14 determines whether the shovel 60 is in the work enable state. In addition, about the determination of the state of the shovel 60 by the working machine state determination means 14, the detail is mentioned later.

The notification unit control unit 15 is a unit that controls the notification unit 20. In the present embodiment, the notification unit control means 15 is based on the determination result of the work machine state determination means 14 or based on the determination result of the human presence / absence determination means 12 and the determination result of the work machine state determination means 14. The notification unit 20 is controlled. The control of the notification unit 20 by the notification unit control means 15 will be described in detail later.

The work permission determination means 16 is a means for determining whether to permit the work of the shovel 60. In the present embodiment, the work permission determination means 16 determines the permission of the work of the shovel 60 based on the determination result of the human existence determination means 12. In addition, about the determination of the permissibility of the operation | work of the shovel 60 by the work permission determination means 16, the detail is mentioned later.

Next, an example of specific processing by the coordinate matching unit 10 and the image generation unit 11 will be described.

The coordinate correlating means 10 can associate the coordinates on the input image plane with the coordinates on the space model using, for example, the quaternion of Hamilton.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the space model. The input image plane of the camera 2 is represented as a plane in the UVW orthogonal coordinate system whose origin is the optical center C of the camera 2. A space model is represented as a solid plane in an XYZ orthogonal coordinate system.

First, the coordinate correlating means 10 moves the origin of the XYZ coordinate system parallel to the optical center C (the origin of the UVW coordinate system), then sets the X axis as the U axis, the Y axis as the V axis, and the Z axis Rotate the XYZ coordinate system so that each matches the -W axis. This is to convert coordinates on the space model (coordinates on the XYZ coordinate system) into coordinates on the input image plane (coordinates on the UVW coordinate system). The sign “−” of “−W axis” means that the direction of the Z axis is opposite to the direction of the −W axis. This is because the UVW coordinate system has the + W direction in front of the camera, and the XYZ coordinate system has the -Z direction in the vertically downward direction.

In the case where a plurality of cameras 2 exist, each of the cameras 2 has a separate UVW coordinate system, so the coordinate correlating means 10 moves the XYZ coordinate system parallel to each of the plurality of UVW coordinate systems and Rotate.

The above-mentioned conversion is performed so that the Z axis coincides with the -W axis after parallel movement of the XYZ coordinate system so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system. It is realized by rotating to coincide with the axis. Therefore, the coordinate correlating means 10 can combine these two rotations into one rotation operation by describing this conversion by the quaternion of Hamilton.

By the way, the rotation for making one vector A coincide with another vector B corresponds to the processing of rotating by an angle formed by the vector A and the vector B around the normal of the surface where the vector A and the vector B extend. . Then, assuming that the angle is θ, from the inner product of the vector A and the vector B, the angle θ is

Is represented by

Also, the unit vector N of the normal to the surface where the vector A and the vector B extend is the outer product of the vector A and the vector B

Will be represented by

When quaternions are i, j, k as imaginary units, respectively,

In the present embodiment, the quaternion Q has a real component as t and pure imaginary components as a, b, and c, respectively.

And the quaternion of the quaternion Q is

Is represented by

The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, c while setting the real component t to 0 (zero), and t, a, b The components c and c can represent rotational motion about an arbitrary vector.

Furthermore, the quaternion Q can be expressed as one rotation operation by integrating a plurality of consecutive rotation operations. Specifically, for example, the quaternion Q is obtained by rotating an arbitrary point S (sx, sy, sz) by an angle θ with an arbitrary unit vector C (l, m, n) as an axis. The point D (ex, ey, ez) can be expressed as follows.

Here, in the present embodiment, assuming that a quaternion representing a rotation that causes the Z axis to coincide with the -W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X '. X 'is

Is represented by

Further, in the present embodiment, assuming that a quaternion representing a rotation that causes the line connecting point X ′ on the X axis and the origin to coincide with the U axis be Qx, “the Z axis coincides with the −W axis. , A quaternion R representing "rotation to align the X axis with the U axis",

Is represented by

As described above, coordinates P ′ when arbitrary coordinates P on the space model (XYZ coordinate system) are expressed by coordinates on the input image plane (UVW coordinate system) are:

Is represented by Further, since the quaternion R is invariant for each of the cameras 2, the coordinate correlating means 10 executes coordinates in the input image plane (UVW coordinate system) simply by executing this operation thereafter. Coordinate system) can be converted to coordinates.

After converting the coordinates on the space model (XYZ coordinate system) to the coordinates on the input image plane (UVW coordinate system), the coordinate correlating means 10 forms a line segment CP ′ and the optical axis G of the camera 2 The incident angle α is calculated. The line segment CP ′ is a line connecting the optical center C (coordinates on the UVW coordinate system) of the camera 2 and coordinates P ′ that represent arbitrary coordinates P on the space model in the UVW coordinate system.

Further, the coordinate associating unit 10 calculates the argument φ in the plane H parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and including the coordinate P ′ and the length of the line segment EP ′. The line segment EP 'is a line connecting the intersection point E between the plane H and the optical axis G and the coordinates P', and the argument φ forms the U 'axis in the plane H and the line segment EP' Is the angle at which

In the camera optical system, the image height h is usually a function of the incident angle α and the focal length f. For this reason, the coordinate associating unit 10 performs the normal projection (h = ftan α), the orthogonal projection (h = fsinα), the solid projection (h = 2 ftan (α / 2)), the iso-stereographic projection (h = 2 f sin (α / 2) The image height h is calculated by selecting an appropriate projection method such as)), equidistant projection (h = fα) or the like.

Thereafter, the coordinate associating unit 10 decomposes the calculated image height h into U component and V component on the UV coordinate system by the argument angle φ, and has a numerical value corresponding to the pixel size per pixel of the input image plane R4. Divide Accordingly, the coordinate associating unit 10 can associate the coordinates P (P ′) on the space model MD with the coordinates on the input image plane R4.

Incidentally, when the pixel size per one pixel in the U axis direction of the input image plane R4 and a _U, the pixel size per one pixel in the V axis direction of the input image plane R4 and a _V, coordinates P of the space model MD The coordinates (u, v) on the input image plane R4 corresponding to (P ′) are

Is represented by

In this way, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on one or more input image plane R4 existing for each camera, and coordinates on the space model MD, a camera identifier , And coordinates on the input image plane R4 are stored in the input image / space model correspondence map 40 in association with each other.

Further, since the coordinate correlating means 10 calculates transformation of coordinates using quaternions, it has an advantage that gimbal lock is not generated unlike the case of calculating transformation of coordinates using Euler angles. . However, the coordinate correlating means 10 is not limited to one that calculates transformation of coordinates using a quaternion, and may calculate the transformation of coordinates using Euler angles.

In addition, when the correspondence to the coordinates on the plurality of input image planes R4 is possible, the coordinate associating unit 10 inputs the coordinates P (P ′) on the space model MD with respect to the camera having the smallest incident angle α. It may be made to correspond to the coordinates on the image plane R4, or may be made to correspond to the coordinates on the input image plane R4 selected by the operator.

Next, among the coordinates on the space model MD, a process of re-projecting coordinates on the curved surface region R2 (coordinates having components in the Z-axis direction) on the processing target image plane R3 on the XY plane will be described.

FIG. 6 is a diagram for explaining the association between the coordinates by the coordinate association means 10. As shown in FIG. F6A is a diagram showing the correspondence between coordinates on the input image plane R4 of the camera 2 adopting coordinates (h = ftan α) as an example and coordinates on the space model MD. The coordinate correlating means 10 causes each of the line segments connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the space model MD corresponding to the coordinates to pass through the optical center C of the camera 2, Correspond both coordinates.

In the example of F6A, the coordinate associating unit 10 associates the coordinate K1 on the input image plane R4 of the camera 2 with the coordinate L1 on the plane region R1 of the space model MD, and the coordinate K2 on the input image plane R4 of the camera 2 Are associated with the coordinate L2 on the curved surface area R2 of the space model MD. At this time, the line segment K1-L1 and the line segment K2-L2 both pass through the optical center C of the camera 2.

When the camera 2 adopts a projection method other than the normal projection (for example, orthographic projection, stereographic projection, equi-solid-angle projection, equidistant projection, etc.), the coordinate correlating means 10 sets the respective projection methods. Accordingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the space model MD.

Specifically, the coordinate correlating means 10 is configured to set a predetermined function (for example, orthogonal projection (h = fsin α), solid projection (h = 2 ftan (α / 2)), isosolid angle projection (h = 2 f sin (α /) 2) The coordinates on the input image plane are associated with the coordinates on the space model MD based on the equidistant projection (h = fα) and the like. In this case, the line segment K1-L1 and the line segment K2-L2 do not pass through the optical center C of the camera 2.

F6B is a diagram showing the correspondence between the coordinates on the curved surface area R2 of the space model MD and the coordinates on the processing object image plane R3. The coordinate correlating means 10 is a parallel line group PL located on the XZ plane, and introduces a parallel line group PL which forms an angle β with the processing target image plane R3. Then, the coordinate associating unit 10 causes both the coordinates on the curved surface region R2 of the space model MD and the coordinates on the processing target image plane R3 corresponding to the coordinates to be on one of the parallel line group PL. , Correspond both coordinates.

In the example of F6B, the coordinate associating unit 10 associates both coordinates on the assumption that the coordinate L2 on the curved surface area R2 of the space model MD and the coordinate M2 on the processing target image plane R3 lie on a common parallel line.

Note that the coordinate associating unit 10 can associate the coordinates on the plane region R1 of the space model MD with the coordinates on the processing object image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved region R2. is there. However, in the example of F6B, the planar region R1 and the processing target image plane R3 are common to each other. Therefore, the coordinate L1 on the plane region R1 of the space model MD and the coordinate M1 on the processing target image plane R3 have the same coordinate value.

Thus, the coordinate associating unit 10 associates the coordinates on the space model MD with the coordinates on the processing object image plane R3, and associates the coordinates on the space model MD and the coordinates on the processing object image plane R3. Are stored in the space model / processing target image correspondence map 41.

F6C is a diagram showing the correspondence relationship between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5 of the virtual camera 2V adopting a normal projection (h = ftan α) as an example. The image generation unit 11 causes each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates to pass through the optical center CV of the virtual camera 2V. And associate both coordinates.

In the example of F6C, the image generation unit 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (the plane region R1 of the space model MD). The coordinate N2 on the output image plane R5 is associated with the coordinate M2 on the processing object image plane R3. At this time, the line segment M1-N1 and the line segment M2-N2 both pass the optical center CV of the virtual camera 2V.

When the virtual camera 2V adopts a projection method other than the normal projection (for example, orthographic projection, stereographic projection, iso-stereographic projection, equidistant projection, etc.), the image generation unit 11 selects one of the projection systems. Accordingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the image generation unit 11 generates a predetermined function (for example, an orthogonal projection (h = fsin α), a solid projection (h = 2 ftan (α / 2)), an isostatic projection (h = 2 f sin (α / 2) The coordinates on the output image plane R5 are associated with the coordinates on the processing object image plane R3 on the basis of equidistant projection (h = fα) etc.)). In this case, the line segment M1-N1 and the line segment M2-N2 never pass the optical center CV of the virtual camera 2V.

Thus, the image generation unit 11 associates the coordinates on the output image plane R5 with the coordinates on the processing object image plane R3, and coordinates on the output image plane R5 and the coordinates on the processing object image plane R3. It associates and stores in the processing target image / output image correspondence map 42. Then, the image generation unit 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / space model correspondence map 40 and the space model / process target image correspondence map 41. Generate an output image.

F6D is a diagram combining F6A to F6C, and shows the mutual positional relationship between the camera 2, the virtual camera 2V, the plane region R1 and the curved region R2 of the space model MD, and the processing target image plane R3.

Next, with reference to FIG. 7, an operation of the parallel line group PL introduced by the coordinate associating unit 10 in order to associate the coordinates on the space model MD and the coordinates on the processing target image plane R3 will be described.

The left drawing of FIG. 7 is a view in the case where an angle β is formed between the parallel line group PL located on the XZ plane and the processing object image plane R3. On the other hand, FIG. 7 right figure is a view in the case where an angle β1 (β1> β) is formed between the parallel line group PL located on the XZ plane and the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the space model MD in the left side of FIG. 7 and the right side of FIG. 7 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. Further, the interval of each of the coordinates La to Ld in the left drawing of FIG. 7 is equal to the interval of each of the coordinates La to Ld in the right drawing of FIG. Although the parallel line group PL is present on the XZ plane for the purpose of explanation, in reality, it is present so as to radially extend from all points on the Z axis toward the processing object image plane R3. Do. Note that the Z axis in this case is referred to as "reprojection axis".

As shown in FIG. 7 left view and FIG. 7 right view, the distance between the parallel line group PL and the processing target image plane R3 increases the distance between the coordinates Ma to Md on the processing target image plane R3. It decreases linearly as That is, the distance between the curved surface area R2 of the space model MD and each of the coordinates Ma to Md decreases uniformly regardless of the distance. On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted to the coordinate group on the processing object image plane R3 in the example of FIG. 7, so the interval between the coordinate groups does not change. .

Among the image portions on the output image plane R5 (see FIG. 6), the change in the distance between these coordinate groups is linearly expanded or only the image portion corresponding to the image projected on the curved region R2 of the space model MD. It means to be reduced.

Next, with reference to FIG. 8, an alternative example of the parallel line group PL introduced by the coordinate associating unit 10 in order to associate the coordinates on the space model MD with the coordinates on the processing target image plane R3 will be described.

The left side of FIG. 8 is a diagram in the case where all the auxiliary line groups AL located on the XZ plane extend from the starting point T1 on the Z axis toward the processing object image plane R3. On the other hand, the right side of FIG. 8 is a diagram in the case where all the auxiliary line groups AL extend from the start point T2 (T2> T1) on the Z axis toward the processing object image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the space model MD in the left drawing of FIG. 8 and the right drawing of FIG. 8 corresponds to each of the coordinates Ma to Md on the processing target image plane R3. In the example of the left figure of FIG. 8, the coordinates Mc and Md are not shown because they are outside the area of the processing target image plane R3. Further, the interval of each of the coordinates La to Ld in the left drawing of FIG. 8 is equal to the interval of each of the coordinates La to Ld in the right drawing of FIG. Although the auxiliary line group AL is present on the XZ plane for the purpose of explanation, in reality, the auxiliary line group AL is present so as to radially extend from any one point on the Z axis toward the processing object image plane R3. Do. As in FIG. 7, the Z axis in this case is referred to as “reprojection axis”.

As shown in FIG. 8 left and FIG. 8 right, each distance between coordinates Ma to Md on the processing object image plane R3 is the distance (height) between the start point of the auxiliary line group AL and the origin O Decreases non-linearly as That is, the larger the distance between the curved surface area R2 of the space model MD and each of the coordinates Ma to Md, the larger the reduction width of each space. On the other hand, the coordinate group on the plane region R1 of the space model MD is not converted to the coordinate group on the processing target image plane R3 in the example of FIG. 8, so the interval between the coordinate groups does not change. .

The change in the distance between these coordinate groups corresponds to the image projected on the curved region R2 of the space model MD in the image portion on the output image plane R5 (see FIG. 6), as in the parallel line group PL. It means that only the image part is expanded or reduced non-linearly.

In this manner, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the space model MD, and the space model MD is generated. The image portion (for example, a horizontal image) of the output image corresponding to the image projected onto the curved surface region R2 of the image data of the image data may be expanded or reduced linearly or non-linearly. Therefore, the image generating apparatus 100 does not affect the road surface image in the vicinity of the shovel 60 (a virtual image when the shovel 60 is viewed from directly above), and an object located around the shovel 60 (horizontal direction from the shovel 60) It is possible to quickly and flexibly enlarge or reduce the object (in the image when the surroundings are viewed), and to improve the visibility of the blind area of the shovel 60.

Next, referring to FIG. 9, a process in which the image generation apparatus 100 generates a processing target image (hereinafter referred to as “processing target image generation processing”), and an output image using the generated processing target image. A process of generating (hereinafter, referred to as “output image generation process”) will be described. FIG. 9 is a flowchart showing a flow of processing target image generation processing (steps S1 to S3) and output image generation processing (steps S4 to S6). Further, the arrangement of the camera 2 (input image plane R4), the space model (plane area R1 and curved area R2), and the processing target image plane R3 are determined in advance.

First, the control unit 1 causes the coordinate associating unit 10 to associate the coordinates on the processing target image plane R3 with the coordinates on the space model MD (step S1).

Specifically, the coordinate associating unit 10 acquires an angle formed between the parallel line group PL and the processing target image plane R3. Then, the coordinate associating unit 10 calculates a point at which one of the parallel line group PL extending from one coordinate on the processing target image plane R3 intersects the curved surface region R2 of the space model MD. Then, the coordinate associating unit 10 derives the coordinates on the curved surface area R2 corresponding to the calculated point as one coordinate on the curved surface area R2 corresponding to the one coordinate on the processing target image plane R3, and the correspondence relationship It is stored in the space model / processing target image correspondence map 41. The angle formed between the parallel line group PL and the processing target image plane R3 may be a value stored in advance in the storage unit 4 or the like, and the operator dynamically uses the input unit 3 via the input unit 3. It may be a value to be input.

Further, when one coordinate on the processing target image plane R3 coincides with one coordinate on the plane region R1 of the space model MD, the coordinate associating unit 10 processes the one coordinate on the plane region R1 as the processing target image. It is derived as one coordinate corresponding to the one coordinate on the plane R 3, and the correspondence is stored in the spatial model-processing target image correspondence map 41.

After that, the control unit 1 causes the coordinate associating unit 10 to associate one coordinate on the space model MD derived by the above-described processing with the coordinate on the input image plane R4 (step S2).

Specifically, the coordinate associating unit 10 acquires the coordinates of the optical center C of the camera 2 that adopts the normal projection (h = f tan α). The coordinate associating unit 10 is a line segment extending from one coordinate on the space model MD, and calculates a point at which the line segment passing through the optical center C intersects with the input image plane R4. Then, the coordinate associating unit 10 derives the coordinates on the input image plane R4 corresponding to the calculated point as one coordinate on the input image plane R4 corresponding to the one coordinate on the space model MD, and the correspondence relationship The input image / space model correspondence map 40 is stored.

Thereafter, the control unit 1 determines whether all the coordinates on the processing target image plane R3 are associated with the coordinates on the space model MD and the coordinates on the input image plane R4 (step S3). Then, when it is determined that all the coordinates have not been associated yet (NO in step S3), the control unit 1 repeats the processing in step S1 and step S2.

On the other hand, when it is determined that all the coordinates are associated (YES in step S3), the control unit 1 ends the processing target image generation processing and then starts the output image generation processing. Then, the control unit 1 causes the image generation unit 11 to associate the coordinates on the processing target image plane R3 with the coordinates on the output image plane R5 (step S4).

Specifically, the image generation unit 11 generates an output image by performing scale conversion, affine conversion, or distortion conversion on the processing target image. Then, the image generation unit 11 processes the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5, which is determined by the contents of the applied scale conversion, affine conversion, or distortion conversion. Store in the output image correspondence map 42.

Alternatively, when generating an output image using the virtual camera 2V, the image generation unit 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the adopted projection method, The correspondence relationship may be stored in the processing target image / output image correspondence map 42.

Alternatively, when generating an output image using the virtual camera 2V adopting normal projection (h = ftan α), the image generation unit 11 acquires the coordinates of the optical center CV of the virtual camera 2V. Then, the image generation unit 11 is a line segment extending from one coordinate on the output image plane R5, and calculates a point at which the line segment passing through the optical center CV intersects the processing target image plane R3. Then, the image generation unit 11 derives the coordinates on the processing target image plane R3 corresponding to the calculated point as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5. In this manner, the image generation unit 11 may store the correspondence in the processing target image / output image correspondence map 42.

After that, the image generation unit 11 of the control unit 1 refers to the input image / space model correspondence map 40, the space model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. Then, the image generation means 11 corresponds the correspondence between the coordinates on the input image plane R4 and the coordinates on the space model MD, the correspondence between the coordinates on the space model MD and the coordinates on the processing object image plane R3, and the processing object The correspondence between the coordinates on the image plane R3 and the coordinates on the output image plane R5 is traced. Then, the image generation unit 11 acquires values (for example, luminance value, hue value, saturation value, etc.) of the coordinates on the input image plane R4 corresponding to the respective coordinates on the output image plane R5, The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation unit 11 generates each of the plurality of coordinates on the plurality of input image planes R4. A value-based statistic may be derived and adopted as the value of its one coordinate on the output image plane R5. The statistical value is, for example, an average value, a maximum value, a minimum value, an intermediate value, and the like.

Thereafter, the control unit 1 determines whether or not the values of all the coordinates on the output image plane R5 are associated with the values of the coordinates on the input image plane R4 (step S6). Then, when it is determined that the values of all the coordinates are not associated yet (NO in step S6), the control unit 1 repeats the processes of step S4 and step S5.

On the other hand, when it is determined that the values of all the coordinates are associated (YES in step S6), the control unit 1 generates an output image and terminates this series of processing.

When the image generation apparatus 100 does not generate the processing target image, the processing target image generation process is omitted. In this case, the "coordinates on the processing target image plane" in step S4 in the output image generation process can be read as "coordinates on the space model".

With the above configuration, the image generating apparatus 100 can generate the processing target image and the output image that can make the operator intuitively grasp the positional relationship between the object around the shovel 60 and the shovel 60.

Further, the image generation apparatus 100 performs coordinate association so as to trace back to the input image plane R4 from the processing target image plane R3 through the space model MD. Thereby, the image generation device 100 can reliably make each coordinate on the processing target image plane R3 correspond to one or more coordinates on the input image plane R4. Therefore, the image generation apparatus 100 generates a processing target image of better quality more quickly than the case of executing coordinate matching in the order from the input image plane R4 to the processing target image plane R3 via the space model MD. Can. When matching of coordinates is performed from the input image plane R4 to the processing target image plane R3 via the space model MD, each coordinate on the input image plane R4 corresponds to one or more coordinates on the processing target image plane R3. It can be made to correspond surely to coordinates. However, a part of the coordinates on the processing target image plane R3 may not be associated with any coordinates on the input image plane R4, and in this case, a part of the coordinates on the processing target image plane R3 It is necessary to perform interpolation processing etc.

Further, in the case of enlarging or reducing only the image corresponding to the curved surface region R2 of the space model MD, the image generation device 100 changes the angle formed between the parallel line group PL and the processing target image plane R3. Desired enlargement or reduction can be realized without rewriting the content of the input image / space model correspondence map 40 only by rewriting only the part related to the curved region R2 in the space model / processing target image correspondence map 41. .

Further, when changing the appearance of the output image, the image generation apparatus 100 only changes the values of various parameters related to scale conversion, affine conversion, or distortion conversion to rewrite the processing target image / output image correspondence map 42. A desired output image (scale conversion image, affine conversion image or distortion conversion image) can be generated without rewriting the contents of the input image / space model correspondence map 40 and the space model / processing target image correspondence map 41.

Similarly, when changing the viewpoint of the output image, the image generation apparatus 100 changes the values of various parameters of the virtual camera 2 V and rewrites the processing object image / output image correspondence map 42 to change the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the space model / process target image correspondence map 41.

FIG. 10 is a display example when an output image generated using input images of two cameras 2 (right-side camera 2R and rear camera 2B) mounted on the shovel 60 is displayed on the display unit 5 .

The image generation apparatus 100 projects the input image of each of the two cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and reprojects the image on the processing target image plane R3 to generate the processing target image. Do. Then, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like) on the generated processing target image. In this manner, the image generating apparatus 100 displays an image (image in the flat region R1) looking down from above the vicinity of the shovel 60 and an image (image in the processing target image plane R3) looking horizontally from the shovel 60 And an output image to be displayed simultaneously. Hereinafter, such an output image is referred to as a peripheral monitoring virtual viewpoint image.

When the image generation apparatus 100 does not generate the processing target image, the peripheral monitoring virtual viewpoint image performs an image conversion process (for example, a viewpoint conversion process) on the image projected on the space model MD. Generated by

In addition, the virtual viewpoint image for periphery monitoring is trimmed in a circular shape so that an image when the shovel 60 performs a turning operation can be displayed without a sense of incongruity, and the center CTR of the circle is on the cylinder central axis of the space model MD and It is generated so as to be on the pivot axis PV of the shovel 60. Therefore, in accordance with the turning operation of the shovel 60, the periphery monitoring virtual viewpoint image is displayed so as to rotate about the center CTR. In this case, the cylinder central axis of the space model MD may or may not coincide with the reprojection axis.

The radius of the space model MD is, for example, 5 meters. In addition, the angle formed by the parallel line group PL with the processing target image plane R3, or the starting point height of the auxiliary line group AL, is the maximum reach distance of the excavating attachment E from the turning center of the shovel 60 (for example, 12 meters) Is set so that when an object (for example, an operator) is present at a position far away from the display, the object is displayed on the display unit 5 sufficiently large (for example, 7 millimeters or more). obtain.

Furthermore, the virtual viewpoint image for periphery monitoring may be arranged such that the CG image of the shovel 60 is such that the front of the shovel 60 coincides with the upper side of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. . This is to make it easier to understand the positional relationship between the shovel 60 and the object appearing in the output image. Note that, in the periphery monitoring virtual viewpoint image, a frame image including various information such as an orientation may be arranged around it.

Next, the details of the periphery monitoring virtual viewpoint image generated by the image generation apparatus 100 will be described with reference to FIGS.

FIG. 11 is a top view of a shovel 60 on which the image generating apparatus 100 is mounted. In the embodiment shown in FIG. 11, the shovel 60 has three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three human detection sensors 6 (left side human detection sensor 6L, right side) It includes a police detection sensor 6R, a rear detection sensor 6B, and three notification units 20 (left direction notification unit 20L, right direction notification unit 20R, and rear notification unit 20B). Regions CL, CR, and CB indicated by alternate long and short dash lines in FIG. 11 indicate imaging spaces of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, areas ZL, ZR, and ZB indicated by dotted lines in FIG. 11 indicate monitoring spaces of the left side detection sensor 6L, the right side detection sensor 6R, and the rear detection sensor 6B, respectively. In the cab 64, the shovel 60 includes the display unit 5, three alarm output units 7 (a left direction alarm output unit 7L, a right direction alarm output unit 7R, and a rear alarm output unit 7B), and a gate lock lever. 8 and an ignition switch 9 are provided.

In the present embodiment, although the monitoring space of the human detection sensor 6 is narrower than the imaging space of the camera 2, the monitoring space of the human detection sensor 6 may be the same as the imaging space of the camera 2. It may be wide. In addition, although the monitoring space of the human detection sensor 6 is located near the shovel 60 in the imaging space of the camera 2, the monitoring space may be in an area farther from the shovel 60. Also, the monitoring space of the human detection sensor 6 has an overlapping portion in the portion where the imaging space of the camera 2 overlaps. For example, in the overlapping portion of the imaging space CR of the right side camera 2R and the imaging space CB of the rear camera 2B, the monitoring space ZR of the right side person detection sensor 6R overlaps with the monitoring space ZB of the rear person detection sensor 6B. However, the monitoring space of the human detection sensor 6 may be arranged so that duplication does not occur.

FIG. 12 shows an input image of each of the three cameras 2 mounted on the shovel 60 and an output image generated using the input images.

The image generation apparatus 100 projects the input image of each of the three cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and reprojects the image on the processing target image plane R3 to generate the processing target image. . Further, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like) on the generated processing target image. As a result, the image generating apparatus 100 displays an image (image in the flat region R1) looking down near the shovel 60 from above and an image (image in the processing target image plane R3) looking horizontally from the shovel 60 A virtual viewpoint image for periphery monitoring to be simultaneously displayed is generated. The image displayed at the center of the periphery monitoring virtual viewpoint image is a CG image 60 CG of the shovel 60.

In FIG. 12, the input image of the right side camera 2R and the input image of the rear camera 2B respectively capture a person in the overlapping portion of the imaging space of the right side camera 2R and the imaging space of the rear camera 2B (right side A region R10 surrounded by a two-dot chain line in the input image of the two-way camera 2R and a region R11 surrounded by a two-dot chain line in the input image of the rear camera 2B.

However, assuming that the coordinates on the output image plane correspond to the coordinates on the input image plane related to the camera with the smallest incident angle, the output image will cause the person in the overlapping part to disappear (a point in the output image See the region R12 enclosed by a dashed line).

Therefore, in the output image portion corresponding to the overlapping portion, in the output image portion corresponding to the overlapping portion, the area to which the coordinates on the input image plane of the rear camera 2B are associated is associated with the coordinates on the input image plane of the right side camera 2R. The area is mixed, and the object in the overlapping part is prevented from disappearing.

FIG. 13 is a diagram for explaining stripe pattern processing which is an example of the image loss prevention processing for preventing the loss of an object in the overlapping portion of the imaging space of each of two cameras.

F13A is a diagram showing an output image portion corresponding to an overlapping portion of the imaging space of the right side camera 2R and the imaging space of the rear camera 2B, and corresponds to a rectangular area R13 shown by a dotted line in FIG.

Also, in F13A, the area PR1 filled with gray is an image area where the input image portion of the rear camera 2B is arranged, and the input image of the rear camera 2B is displayed at each coordinate on the output image plane corresponding to the area PR1. Coordinates on the plane are associated.

On the other hand, the area PR2 filled with white is an image area where the input image portion of the right side camera 2R is arranged, and each coordinate on the output image plane corresponding to the area PR2 is the input image plane of the right side camera 2R The upper coordinates are associated.

In the present embodiment, the area PR1 and the area PR2 are arranged to form a stripe pattern (stripe pattern), and the boundary line between the area PR1 and the area PR2 alternately arranged in a stripe is the turning center of the shovel 60. Defined by concentric circles on a horizontal plane centered on

F13B is a top view showing the situation of the space area at the diagonally right back of the shovel 60, and shows the current situation of the space area imaged by both the rear camera 2B and the right side camera 2R. Further, F13B indicates that a rod-like three-dimensional object OB is present obliquely rearward to the right of the shovel 60.

F13C shows a part of an output image generated based on an input image obtained by actually imaging the space area indicated by F13B with the rear camera 2B and the right side camera 2R.

Specifically, in the image OB1, the image of the solid object OB in the input image of the rear camera 2B is extended in the extension direction of the line connecting the rear camera 2B and the solid object OB by viewpoint conversion for generating a road surface image Represents what was That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the rear camera 2B.

In the image OB2, the image of the solid object OB in the input image of the right side camera 2R is expanded in the extension direction of the line connecting the right side camera 2R and the solid object OB by viewpoint conversion for generating a road surface image. Represent the That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated using the input image of the right side camera 2R.

As described above, in the overlapping portion, the image generation device 100 combines the area PR1 with which the coordinates on the input image plane of the rear camera 2B are associated and the area PR2 with which the coordinates on the input image plane of the right side camera 2R are associated. Mix As a result, the image generation apparatus 100 displays both the two images OB1 and OB2 of one solid object OB on the output image, and prevents the solid object OB from disappearing from the output image.

14 is a contrast diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image loss prevention process (stripe pattern process) to the output image of FIG. The output image of FIG. 12 is shown, and the lower part of FIG. 14 shows an output image after the image loss prevention process (stripe pattern process) is applied. While a person disappears in a region R12 surrounded by an alternate long and short dash line in the upper diagram of FIG. 14, a human is displayed without disappearing in a region R14 enclosed by an alternate long and short dash line in the lower diagram of FIG.

The image generation apparatus 100 may prevent the disappearance of the object in the overlapping portion by applying mesh pattern processing, averaging processing, or the like instead of the stripe pattern processing. Specifically, the image generating apparatus 100 outputs an average value of the values (for example, luminance values) of corresponding pixels in the input images of the two cameras by averaging, and corresponds to the overlapping portion. Adopted as the pixel value of the image part. Alternatively, in the output image portion corresponding to the overlapping portion, the image generation apparatus 100 performs mesh pattern processing on a region to which values of pixels in the input image of one camera are associated and values of pixels in the input image of the other camera. Are arranged so as to form a mesh pattern (mesh pattern). Thereby, the image generation device 100 prevents the loss of the object in the overlapping portion.

Next, with reference to FIG. 15 to FIG. 17, the process of switching the content of the output image based on the determination result of the person existence / non-existence determination means 12 (hereinafter referred to as “first output image switching process”) ). FIG. 15 is a view showing input images of the three cameras 2 mounted on the shovel 60 and an output image generated using the input images, and corresponds to FIG. FIG. 16 is a diagram showing an example of the relationship between the space model MD used in the first output image switching process and the processing target image plane R3, and corresponds to FIG. FIG. 17 is a diagram for explaining the relationship between two output images switched in the first output image switching process.

As shown in FIG. 15, the image generation apparatus 100 projects the input image of each of the three cameras 2 onto the plane region R1 and the curved region R2 of the space model MD, and reprojects it on the processing object image plane R3. Generate an image to be processed. Further, the image generation apparatus 100 generates an output image by performing image conversion processing (for example, scale conversion, affine conversion, distortion conversion, viewpoint conversion processing, and the like) on the generated processing target image. As a result, the image generation device 100 generates a periphery monitoring virtual viewpoint image that simultaneously displays an image of the vicinity of the shovel 60 as viewed from above and an image of the periphery of the shovel 60 viewed in the horizontal direction.

Further, in FIG. 15, the input images of the left side camera 2L, the rear side camera 2B, and the right side camera 2R show a state in which three workers are present. The output image also shows that nine workers are present around the shovel 60.

The height of the plane region R1 of the space model MD is set to a height corresponding to the road surface which is the ground contact surface of the shovel 60. Therefore, in the image in the plane region R1 of FIG. 15, the distance between the contact positions of the nine workers and the CG image 60CG of the shovel 60 is between the respective ones of the nine workers and the shovel 60. It accurately represents the actual distance. However, the image of the worker is displayed so as to be larger as it goes away from the ground contact position (under the foot). In particular, as shown in FIG. 15, the head of the worker is displayed significantly larger than the size of the CG image 60 CG of the shovel 60. Therefore, the operator of the shovel 60 who saw the output image of FIG. 15 may be illusive that the distance between the shovel 60 and the worker is larger than the actual distance.

Therefore, the image generation unit 11 determines the content of the output image when it is determined by the human existence / nonexistence determination unit 12 that a person is present in any of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. Switch. In the present embodiment, the image generation means 11 sets the height of the plane region R1 of the space model MD to a height corresponding to the height of a person's head from the height corresponding to the road surface (hereinafter referred to as “head height”). Switch to the The head height is a value set in advance, and is, for example, 150 cm. However, the head height may be a value determined dynamically. For example, when the height (height) of the worker present around the shovel 60 can be detected based on the output of the human detection sensor 6 or the like, the image generation device 100 determines the head height based on the detected height. May be Specifically, the height of the head may be determined according to the height of the worker closest to the shovel 60, and the statistical values (maximum value, minimum value, height values of heights of a plurality of workers present around the shovel 60) The head height may be determined according to the average value or the like.

Here, referring to FIGS. 16 and 17, space model MD of FIG. 4 having plane area R1 of a height corresponding to the road surface, and head height reference space model MDM having plane area R1M of the head height Explain the relationship between The upper diagram in FIG. 16 shows the relationship between the space model MD and the processing target image plane R3 including the flat region R1, and the lower diagram in FIG. 16 shows the head height reference space model MDM and the head height reference plane region R1M. And a head height reference processing target image plane R3M including the Further, in FIG. 17, an output screen D1 is a virtual viewpoint image for perimeter monitoring based on a road surface height generated using a space model MD, and an output image D2 is a head height generated using a space model MDM It is a virtual viewpoint image for perimeter surveillance of the standard. The image D3 is an explanatory image representing the difference in size between the road surface height-based perimeter monitoring virtual viewpoint image and the head height-based perimeter monitoring virtual viewpoint image. Further, the size of the image portion D10 in the periphery monitoring virtual viewpoint image based on the road surface height corresponds to the size of the periphery monitoring virtual viewpoint image based on the head height.

As shown in FIG. 16, the heights of the head height reference plane region R1M and the head height reference processing target image plane R3M are compared with the heights of the plane region R1M and the processing target image plane R3 corresponding to the height of the road surface. , Head height HT only.

As a result, as shown in FIG. 17, in the output image D2, which is a virtual viewpoint image for perimeter monitoring based on head height, the distance between the body portion (head) of the worker present at the head height HT and the shovel 60 is , Is displayed to represent the actual distance between the worker and the shovel 60. Thus, the image generation device 100 can prevent the operator of the shovel 60 who has viewed the output image from capturing the distance between the shovel 60 and the worker larger than the actual distance.

As shown in FIG. 16, a region D10M on the space model MDM corresponding to the image portion D10 is included in the head height reference plane region R1M. That is, the virtual viewpoint image for perimeter monitoring based on head height does not use the input image portion reprojected to the annular portion (portion other than head height reference plane region R1M) of head height based processing target image plane R3M . Therefore, in the present embodiment, the image generating apparatus 100 omits the association of the coordinates in the area other than the area D10M.

With the above configuration, the image generation apparatus 100 switches the content of the output image based on the determination result of the person presence / absence determination unit 12. Specifically, when it is determined that a person is present around the shovel 60, the image generating apparatus 100 switches the virtual viewpoint image for perimeter monitoring based on the road surface height to the virtual viewpoint image for perimeter monitoring based on the head height. . As a result, when the image generation device 100 detects a worker around the shovel 60, the image generation device 100 can more accurately convey the distance between the shovel 60 and the worker to the operator of the shovel 60. If the image generating apparatus 100 subsequently determines that there is no person around the shovel 60, the virtual viewpoint image for perimeter monitoring based on head height is converted to a virtual viewpoint image for perimeter monitoring based on road surface height. Switch. This is to allow the operator of the shovel 60 to monitor the surroundings of the shovel 60 more extensively.

Next, referring to FIG. 18, another process of switching the content of the output image based on the determination result of the person presence / absence determination means 12 (hereinafter, referred to as “second output image switching process”) will be described. Will be described. FIG. 18 is a diagram for explaining the relationship between three output images switched in the second output image switching process.

The image generation apparatus 100 executes image loss prevention processing such as stripe pattern processing, mesh pattern processing, and averaging processing for preventing the loss of an object in an overlapping portion of the imaging space of each of the two cameras. Specifically, the image generation apparatus 100 performs overlap by combining the coordinates on the input image plane of each of the two cameras in the output image portion corresponding to the overlapping portion by the image loss prevention process. Prevent the disappearance of objects in parts. However, the image loss prevention process has a problem that the visibility of the object in the overlapping portion is reduced.

Therefore, the image generation unit 11 switches the content of the output image according to the determination result of the person presence / absence determination unit 12. In the present embodiment, the image generation unit 11 sets a camera corresponding to the monitoring space determined to have a person as a priority camera, and sets a camera corresponding to the monitoring space determined to have no person as a priority camera. Note that the imaging space of the priority camera and the imaging space of the priority camera have an overlapping portion. Then, when the priority camera and the camera to be prioritized can not be determined, the image generation unit 11 generates a virtual viewpoint image for periphery monitoring to which the image loss prevention process is applied. On the other hand, when the image generation unit 11 can determine the priority camera and the priority camera, the image loss prevention process is not applied to the output image portion corresponding to the overlapping portion on the input image plane of the priority camera. A virtual viewpoint image for monitoring the periphery is generated by correlating the coordinates of.

Specifically, the image generation unit 11 monitors the periphery by applying the image loss prevention process when it is determined that a person is present in all of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. Virtual viewpoint images for This is because any camera can not be a priority camera, and neither camera can be a priority camera. That is, when any camera is used as a priority camera, there is a possibility that the person in the overlapping portion may be lost.

In addition, the image generation means 11 is for peripheral monitoring to which the image loss prevention process is applied even when it is determined that no person exists in all of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. Generate a virtual viewpoint image. This is because there is no target to be displayed while using any camera as a priority camera. However, in this case, without applying the image loss prevention process, the image generation unit 11 monitors the periphery by correlating the coordinates on the input image plane of one of the cameras with the output image portion corresponding to the overlapping portion. Virtual viewpoint images may be generated. This is because there is no object that may be lost in the first place, and the processing load of the control unit 1 can be reduced by omitting the image loss prevention process.

The output image D11 of FIG. 18 is generated when it is determined that a person is present in all of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. 7 shows a monitoring virtual viewpoint image. In this case, the image generation unit 11 sets the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB to the area where the coordinates on the input image plane of the left side camera 2L are associated An area to which coordinates on the input image plane of the camera 2B are associated is mixed. Further, the image generation unit 11 is configured such that a region on the input image plane of the right side camera 2R is associated with the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB An area to which coordinates on the input image plane of 2B are associated is mixed. This is to prevent the loss of the worker at the output image portions R15 and R16.

On the other hand, when it is determined that a person is present in the left-side monitoring space ZL and it is determined that a person is not present in the backward monitoring space ZB, the image generation unit 11 sets the left-side camera 2L as a priority camera, The rear camera 2B is set as a priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the left side camera 2L with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB.

When it is determined that a person is present in the right side surveillance space ZR, and it is determined that a person is not present in the rear surveillance space ZB, the image generation means 11 sets the right side camera 2R as a priority camera, The camera 2B is set as a priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the right side camera 2R with the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB.

The output image D12 of FIG. 18 is a side camera that is generated when it is determined that a person is present in the left side monitoring space ZL and the right side monitoring space ZR, and it is determined that a person is not present in the rear monitoring space ZB. The priority virtual observation image for periphery monitoring is shown. In this case, the image generation unit 11 sets each of the left side camera 2L and the right side camera 2R as a priority camera, and sets the rear camera 2B as a priority camera. Then, the image generation unit 11 associates the coordinates on the input image plane of the left side camera 2L with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB, Coordinates on the input image plane of the right side camera 2R are associated with the output image portion R16 corresponding to the overlapping portion of the rear imaging space CB. As a result, the image generation unit 11 associates the output image portion R17P with the coordinates on the input image plane of the left side camera 2L, associates the output image portion R18P with the coordinates on the input image plane of the right side camera 2R, and outputs Coordinates on the input image plane of the rear camera 2B are associated with the image portion R19N.

In addition, when it is determined that a person does not exist in the left side monitoring space ZL and the person is determined to exist in the rear monitoring space ZB, the image generation unit 11 sets the left side camera 2L as a priority camera, The rear camera 2B is set as a priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB.

When it is determined that no person is present in the right side surveillance space ZR and it is determined that a person is present in the backward surveillance space ZB, the image generation unit 11 sets the right side camera 2R as a priority camera, The rear camera 2B is set as a priority camera. That is, the image generation unit 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB.

An output image D13 of FIG. 18 is a rear camera generated when it is determined that a person does not exist in the left side surveillance space ZL and the right side surveillance space ZR and it is determined that a person exists in the rear surveillance space ZB. The priority virtual observation image for periphery monitoring is shown. In this case, the image generation unit 11 sets each of the left side camera 2L and the right side camera 2R as a priority camera, and sets the rear camera 2B as a priority camera. Then, the image generation unit 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB, and the right side imaging space CR and the rear Coordinates on the input image plane of the rear camera 2B are also associated with the output image portion R16 corresponding to the overlapping portion of the imaging space CB. As a result, the image generation unit 11 associates the output image portion R17N with the coordinates on the input image plane of the left side camera 2L, associates the output image portion R18N with the coordinates on the input image plane of the right side camera 2R, and outputs Coordinates on the input image plane of the rear camera 2B are associated with the image portion R19P.

As shown in FIG. 18, the output image portion R17P corresponds to a portion obtained by adding the output image portion R15 to the output image portion R17N, and the output image portion R18P has the output image portion R16 added to the output image portion R18N. It corresponds to the part. The output image portion R19P corresponds to a portion obtained by adding the output image portion R15 and the output image portion R16 to the output image portion R19N.

FIG. 19 is a correspondence table showing the correspondence between the determination result of the human presence / absence determination means 12 and the content of the output image. A circle indicates that it is determined by the human presence / absence determination unit 12 that a person is present, and a cross indicates that it is determined that a person is not present.

As shown in the output image D12 of FIG. 18, when it is determined that a person is present only in the left-side monitoring space ZL and it is determined that a person is not present in the backward monitoring space ZB and the right-side monitoring space ZR. It represents that the virtual camera viewpoint image for periphery surveillance of a side camera priority is generated.

Pattern B is as shown in the output image D12 of FIG. 18 when it is determined that a person is present only in the right side surveillance space ZR and it is determined that a person does not exist in the left direction surveillance space ZL and the rear surveillance space ZB. It represents that the virtual camera viewpoint image for periphery surveillance of a side camera priority is generated.

The pattern C is a side as shown in the output image D12 of FIG. 18 when it is determined that a person is present in the left side monitoring space ZL and the right side monitoring space ZR and it is determined that a person is not present in the rear monitoring space ZB. It represents that a virtual camera viewpoint image with peripheral camera priority is generated.

In the pattern D, when it is determined that a person is present in the rear monitoring space ZB and it is determined that no person is present in the left side monitoring space ZL and the right side monitoring space ZR, a rear as shown in the output image D13 of FIG. It represents that a camera-preferred peripheral monitoring virtual viewpoint image is generated.

In the patterns E to G, when it is determined that a person is present in at least one of the left side monitoring space ZL and the right side monitoring space ZR and the rear monitoring space ZB, the image loss prevention as shown in the output image D11 of FIG. It represents that a peripheral viewpoint virtual viewpoint image to which the processing is applied is generated.

When it is determined that there is no person in all of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR, the pattern H performs the image loss prevention process as shown in the output image D11 of FIG. Indicates that the applied peripheral monitoring virtual viewpoint image is generated.

With the above configuration, the image generation apparatus 100 switches the content of the output image based on the determination result of the person presence / absence determination unit 12. Specifically, when the image generation apparatus 100 can not determine the priority camera and the camera to be prioritized, it generates a virtual viewpoint image for periphery monitoring to which the image loss prevention process is applied. On the other hand, when the image generation unit 11 can determine the priority camera and the prioritized camera, the image generation unit 11 does not apply the image loss prevention process to the output image portion corresponding to the overlapping portion on the input image plane of the priority camera. By associating the coordinates, a peripheral viewpoint virtual viewpoint image is generated. As a result, the image generation apparatus 100 can display the worker present in the overlapping portion of the imaging space of each of the two cameras more clearly.

Further, in the above-described embodiment, the image generation device 100 generates a virtual viewpoint image for perimeter monitoring with side camera priority, with the left side camera 2L and the right side camera 2R as priority cameras and the rear camera 2B as priority cameras. Do. Alternatively, the image generating apparatus 100 generates a rear-view-preferred peripheral viewpoint virtual viewpoint image with the rear-side camera 2L and the right-side camera 2R as priority cameras and the rear camera 2B as a priority camera. However, the present invention is not limited to this configuration. For example, the image generation apparatus 100 applies the image loss prevention process to the output image portion R16 while associating the output image portion R15 with the coordinates on the input image plane of either the left side camera 2L or the rear camera 2B. May be In addition, the image generation apparatus 100 applies the image loss prevention process to the output image portion R15 while associating the output image portion R16 with the coordinates on the input image plane of either the right side camera 2R or the rear camera 2B. May be

Further, the image generation apparatus 100 may combine the first output image switching process and the second output image switching process.

Further, in the above-described embodiment, the image generation device 100 associates the monitoring space of one human detection sensor with the imaging space of one camera, but monitors the monitoring space of one human detection sensor in the imaging spaces of a plurality of cameras. It may be made to correspond, and the monitoring space of a plurality of person detection sensors may be made to correspond to the imaging space of one camera.

Further, in the above-described embodiment, the image generation apparatus 100 switches the content of the output image at the moment when the determination result of the person presence / absence determination means 12 changes. However, the present invention is not limited to this configuration. For example, the image generation apparatus 100 may set a predetermined delay time before the content of the output image is switched after the determination result of the person existence determination unit 12 changes. This is to suppress frequent switching of the content of the output image.

Next, with reference to FIGS. 20 and 21, the alarm control unit 13 controls the alarm output unit 7 based on the determination result of the human presence / absence determination unit 12 (hereinafter, referred to as “first alarm control processing”). Will be described. FIG. 20 is a flowchart showing the flow of the first alarm control process, and FIG. 21 shows an example of the transition of the output image displayed during the first alarm control process. The alarm control means 13 repeatedly executes this first alarm control process at a predetermined cycle.

First, the human presence / absence determination means 12 determines whether or not a person is present around the shovel 60 (step S11). At this time, the image generation unit 11 generates and displays, for example, a virtual viewpoint image for perimeter monitoring based on the road surface height as shown in the output image D21 of FIG.

When it is determined that a person is present around the shovel 60 (YES in step S11), the person presence / absence determination means 12 outputs a detection signal to the alarm control means 13. The alarm control means 13 having received the detection signal outputs an alarm start signal for starting an alarm to the alarm output unit 7, and causes the alarm output unit 7 to output an alarm (step S12).

In the present embodiment, the alarm output unit 7 outputs an alarm sound. Then, the image generation means 11 switches the virtual viewpoint image for perimeter monitoring based on the road surface height to the virtual viewpoint image for perimeter monitoring based on the head height, for example, as shown in the output image D22 of FIG. Specifically, the human presence / absence determination means 12 notifies the alarm control means 13 of a detection signal when it is determined that the worker P1 is present in the backward monitoring space ZB. The alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let In addition, the image generation unit 11 superimposes and displays the alarm stop button G1 on the periphery monitoring virtual viewpoint image based on head height. The alarm stop button G1 is a software button configured in cooperation with the touch panel as the input unit 3. The operator can stop the alarm sound by pressing the alarm stop button G1. The alarm stop button G1 may be a hardware button installed in the vicinity of the display unit 5. In this case, the image generation unit 11 may superimpose a text message indicating that the alarm can be stopped by pressing the hardware button as the alarm stop button G1 on the periphery monitoring virtual viewpoint image. Further, even if it is determined that a person is present around the shovel 60, the image generation means 11 does not change the virtual viewpoint image for perimeter monitoring based on the road surface height as shown in the output image D23 of FIG. The alarm stop button G1 may be superimposed and displayed while being used.

On the other hand, when it is determined that a person does not exist around the shovel 60 (NO in step S11), the person presence / absence determination unit 12 does not output a detection signal to the alarm control unit 13. Therefore, the alarm control means 13 does not output an alarm start signal to the alarm output unit 7. Further, even when the alarm control means 13 has already output an alarm, it does not output an alarm stop signal for stopping the alarm to the alarm output unit 7.

Thereafter, the alarm control means 13 determines whether or not the alarm stop button G1 has been pressed (step S13). When it is determined that the alarm stop button G1 is pressed (YES in step S13), the alarm control unit 13 outputs an alarm stop signal to the alarm output unit 7, and the alarm output from the alarm output unit 7 is output. It is stopped (step S14). In addition, when displaying the virtual viewpoint image for perimeter monitoring based on head height, the image generation unit 11 displays the virtual viewpoint image for perimeter monitoring based on head height as a virtual viewpoint for perimeter monitoring based on road surface height. Switch to images and display.

On the other hand, when it is determined that the alarm stop button G1 is not pressed (NO in step S13), the alarm control unit 13 does not output an alarm stop signal to the alarm output unit 7. That is, even if the human presence / absence determination means 12 subsequently determines that there is no person around the shovel 60, the warning output by the warning control means 13 does not stop until the warning stop button G1 is pressed. . Further, when displaying the peripheral monitoring virtual viewpoint image based on the head height, the image generation unit 11 continues the display of the peripheral monitoring virtual viewpoint image based on the head height.

The alarm control means 13 changes the content of the alarm when the human existence / nonexistence determination means 12 determines that there is no person around the shovel 60 even before the alarm stop button G1 is pressed. May be Specifically, the alarm control means 13 may change the intensity, height, output interval, etc. of the alarm sound that has already been output, or the intensity of the alarm lamp that has already emitted light, emission color, emission interval You may change etc. This is to allow the operator who receives the alarm to distinguish between the case where it is determined by the person existence determination means 12 that there is a person and the case where it is determined that there is no person.

With the above configuration, the image generating apparatus 100 outputs an alarm when it is determined that a person is present in the monitoring space, and does not stop the alarm until the operator indicates the intention to stop the alarm. That is, the image generation apparatus 100 stops the alarm even when a person who has been present in the monitoring space has left the monitoring space or when a person who has been present in the monitoring space has failed to be detected. There is nothing to do. Therefore, the image generating apparatus 100 can urge the operator to confirm that the worker present in the monitoring space around the shovel 60 has left the monitoring space.

Further, in the configuration in which the human presence / absence determination means 12 determines the presence or absence of the worker (moving body) based on the moving body detection sensor, the image generation device 100 causes the alarm to stop in response to the worker standing still in the monitoring space. Can be prevented. Further, in the configuration in which the human presence / absence determination means 12 determines the presence / absence of the worker (moving body) using the optical flow, the image generation device 100 stops the alarm in response to the worker being stopped in the monitoring space. It is possible to prevent it. Note that the moving object detection sensor and the optical flow do not distinguish between a stationary object in the monitoring space and a leaving of the worker from the monitoring space without setting a stationary object as a detection target. Therefore, in the configuration in which the alarm is stopped when it is determined that a person does not exist, the alarm may be stopped although the worker is present in the monitoring space. On the other hand, the image generating apparatus 100 can call attention of the operator of the shovel 60 by not stopping the alarm when there is a possibility that a worker is present in the monitoring space.

Next, with reference to FIG. 22 to FIG. 25, the alarm control means 13 controls the alarm output unit 7 based on the judgment result of the human presence / absence judgment means 12 and the judgment result of the work machine state judgment means 14 ( Hereinafter, the “second alarm control process” will be described. FIG. 22 is a flowchart showing the flow of the second alarm control process, and FIG. 23 shows an example of the transition of the output image displayed during the second alarm control process. FIG. 24 is a flowchart showing a flow of processing (hereinafter referred to as “working machine state determination processing”) for the working machine state determination means 14 to determine the state of the working machine, and FIG. 4 is a flowchart showing a flow of processing (hereinafter referred to as “work start alarm control processing”) in which the alarm control means 13 controls the alarm output unit 7 at the time of switching from the state to the work available state. The alarm control means 13 repeatedly executes the second alarm control process at a predetermined cycle, and the work machine state determination means 14 repeatedly executes the work machine state judgment process at a predetermined cycle. Further, the alarm control means 13 executes an alarm control process at the start of operation when switching from the operation impossible state to the operation enable state.

First, the human existence determining means 12 determines whether or not a person exists around the shovel 60 (step S21). At this time, when the display unit 5 is activated, the image generation unit 11 generates and displays, for example, a virtual viewpoint image for perimeter monitoring based on the road surface height as shown in the output image D31 of FIG. .

When it is determined that a person is present around the shovel 60 (YES in step S21), the human presence / absence determination means 12 refers to the result of determination by the working machine state determination means 14 (see FIG. 24 described later) (step S22). ). In this case, the person existence determination means 12 may fix the gate lock lever 8 in the locked state. Specifically, the human presence / absence determination means 12 may not allow the operator to pull up the gate lock lever 8, and may not be in the unlocked state even if the gate lock lever 8 is pulled up. .

When it is determined by the work machine state determination means 14 that the shovel 60 is in the work enable state (YES in step S22), the human presence / absence determination means 12 outputs a detection signal to the alarm control means 13. The alarm control means 13 having received the detection signal outputs an alarm start signal for outputting an alarm to the alarm output unit 7, and causes the alarm output unit 7 to output an alarm (step S23).

In the present embodiment, the alarm output unit 7 outputs an alarm sound. Specifically, when it is determined that a person (here, workers P10 to P12) is present in the left direction monitoring space ZL, the human presence / absence determination means 12 sends a left direction detection signal to the alarm control means 13 Notice. The alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let In this case, the control unit 1 activates the display unit 5 when the display unit 5 is not activated. Then, the image generation unit 11 switches the virtual viewpoint image for perimeter monitoring on the road surface height standard to the virtual viewpoint image for perimeter monitoring on the head height standard, as shown in the output image D22 of FIG. Display superimposed. Further, as shown in the output image D23 of FIG. 21, the image generation means 11 may superimpose and display the alarm stop button G1 on the virtual viewpoint image for perimeter monitoring based on the road surface height. In this case, the alarm control means 13 may prohibit the work by the shovel 60 until the alarm stop button G1 is pressed. Specifically, the alarm control means 13 may close the gate lock valve and shut off the flow of hydraulic fluid between the control valve and the operation lever or the like to invalidate the operation lever or the like.

When it is determined by the work machine state determination means 14 that the shovel 60 is not in the work enable state (NO in step S22), the human presence / absence determination means 12 does not output a detection signal to the alarm control means 13. The value of the human detection flag prepared in the NVRAM or the like is set to "1" (on) (step S24). The human detection flag is initially set to a value “0” (off), and the human detection flag value “1” (on) represents that a human is detected, and the human detection flag value “0”. “(Off)” indicates that a person is not detected. At this time, since the alarm control means 13 does not receive the detection signal, it does not output the alarm start signal to the alarm output unit 7 and does not output the alarm from the alarm output unit 7. Further, when the alarm control unit 13 has already output the alarm start signal to the alarm output unit 7, the alarm control unit 13 may output the alarm stop signal to the alarm output unit 7. Further, the image generation means 11 is similar to the case where it is determined that there is no person even though it is determined that there is a person around the shovel 60 when the display unit 5 is activated. Display the output image. Specifically, the image generation unit 11 displays, for example, a peripheral viewpoint virtual viewpoint image based on the road surface height as shown in the output image D32 of FIG. However, the image generation unit 11 may switch the virtual viewpoint image for perimeter monitoring based on the road surface height to the virtual viewpoint image for perimeter monitoring based on the head height.

On the other hand, when it is determined that a person does not exist around the shovel 60 (NO in step S21), the human presence / absence determination means 12 does not refer to the result of determination by the working machine state determination means 14, and the alarm control means 13 No detection signal is output.

In this case, in the case where the warning has already been output, the image generation means 11 is similar to the case where it is determined that a person is present despite the fact that no person is present around the shovel 60. Display the output image. Specifically, the image generation unit 11 displays a virtual viewpoint image for perimeter monitoring based on head height as shown in the output image D33 of FIG. 23, for example, together with the alarm stop button G1. However, the image generation unit 11 may switch the virtual viewpoint image for perimeter monitoring based on head height to the virtual viewpoint image for perimeter monitoring based on road surface height as shown in the output image D34 of FIG.

Thereafter, when the alarm is already output, the alarm control means 13 monitors whether the alarm stop button G1 is pressed (step S25). When it is determined that the alarm stop button G1 is pressed (YES in step S25), the alarm control unit 13 outputs an alarm stop signal to the alarm output unit 7, and the alarm output from the alarm output unit 7 is output. It is stopped (step S26). In addition, when displaying the virtual viewpoint image for perimeter monitoring based on head height (see output image D33), the image generation unit 11 displays the virtual viewpoint image for perimeter monitoring based on head height on the road surface height. It switches to the reference | standard periphery monitoring virtual viewpoint image (refer the output image D31), and is displayed.

On the other hand, when it is determined that the alarm stop button G1 is not pressed (NO in step S25), the alarm control unit 13 does not output an alarm stop signal to the alarm output unit 7. That is, even if the human presence / absence determination means 12 subsequently determines that there is no person around the shovel 60, the warning output by the warning control means 13 does not stop until the warning stop button G1 is pressed. . In addition, when displaying the virtual viewpoint image for perimeter monitoring based on head height (see output image D33), the image generation means 11 displays the virtual viewpoint image for perimeter monitoring based on that head height. To continue.

Next, with reference to FIG. 24, the working machine state determination processing by the working machine state determination means 14 will be described.

First, the work machine state determination means 14 determines whether the ignition switch 9 is in the on state based on the output of the ignition switch 9 (step S31).

When it is determined that the ignition switch 9 is in the on state (YES in step S31), the work machine state determination unit 14 determines whether the gate lock lever 8 is in the unlocked state based on the output of the gate lock lever 8. It determines (step S32).

If it is determined that the gate lock lever 8 is in the unlocked state (YES in step S32), the work machine state determination unit 14 determines that the shovel 60 is in the work available state (step S33).

On the other hand, when it is determined that the ignition switch 9 is not in the on state (NO in step S31), or when it is determined that the gate lock lever 8 is not in the unlocking state (NO in step S32) It is determined that the shovel 60 is not in the workable state, that is, the shovel 60 is in the work impossible state (step S34).

After that, the work machine state determination unit 14 sets the value of the work available flag prepared in the NVRAM or the like according to the determination result. The human presence / absence determination means 12 refers to the value of the operation possible flag in step S22 of the second alarm control process shown in FIG. 22, and determines the presence / absence of alarm output based on the referred value. Note that the work enable flag is initially set to the value “0” (off), and the work enable flag value “1” (on) indicates that work is possible, and the work enable flag value “0”. “(Off)” indicates that the work can not be performed.

The work machine state determination means 14 may determine whether the shovel 60 is in the work-enabled state based on only one of the gate lock lever 8 and the ignition switch 9.

Next, with reference to FIG. 25, the alarm control process at the work start time by the alarm control means 13 will be described.

First, the alarm control means 13 refers to the value of the human detection flag (step S41). When it is determined that the value of the human detection flag is “1” on (YES in step S41), the alarm control unit 13 outputs an alarm start signal to the alarm output unit 7, and the alarm output unit 7 An alarm is output (step S42). Specifically, the alarm control means 13 outputs an alarm even when it is determined by the human presence / absence determination means 12 that there is no person around the shovel 60 at the present time. In this embodiment, the alarm control unit 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and generates an alarm from all three alarm output units. Make the sound output. In this case, the alarm control means 13 may prohibit the work by the shovel 60 until the output of the alarm is stopped. Specifically, the alarm control means 13 may close the gate lock valve and shut off the flow of hydraulic fluid between the control valve and the operation lever or the like to invalidate the operation lever or the like.

Thereafter, the alarm control means 13 measures an elapsed time after the alarm start, and determines whether a predetermined time (for example, 2 seconds) has elapsed after the alarm start (step S43).

If it is determined that the predetermined time has not elapsed after the start of the alarm (NO in step S43), the alarm control unit 13 stands by until the predetermined time elapses.

On the other hand, when it is determined that the predetermined time has elapsed after the start of the alarm (YES in step S43), the alarm control unit 13 outputs an alarm stop signal to the alarm output unit 7, and the alarm output from the alarm output unit 7 Are stopped (step S44). Further, the alarm control means 13 resets the value of the human detection flag to "0" (off) (step S45). The alarm control means 13 may not stop the alarm until the alarm stop button G1 is pressed.

With the above configuration, the image generation apparatus 100 does not output an alarm when the shovel 60 is in the operation impossible state even when it is determined that a person is present in the monitoring space. Therefore, the image generation apparatus 100 can prevent an unnecessary alarm from being output when the shovel 60 is in the operation impossible state.

On the other hand, even if it is determined that the image generation device 100 does not have a person in the monitoring space at the present time, the case where the worker enters the monitoring range while the shovel 60 is in the operation impossible state Output an alarm to. Therefore, the image generating apparatus 100 can urge the operator to confirm that the worker present in the monitoring space around the shovel 60 has left the monitoring space.

Further, in the above-described work start alarm control process, the alarm control means 13 outputs an alarm as long as the value of the human detection flag is “1” (on). However, the present invention is not limited to this configuration. For example, the alarm control unit 13 may not output an alarm if the elapsed time from the time when the value of the human detection flag is set to “1” (on) is equal to or longer than a predetermined time. Alternatively, the human presence / absence determination means 12 sets the value of the human detection flag to “0” (off) when the elapsed time from the time of setting the value of the human detection flag to “1” (on) reaches a predetermined time. It may be reset. This is to prevent an alarm from being output based on a detection record that is too old.

Further, when the human presence / absence determination means 12 determines the presence / absence of the worker (moving body) based on the moving body detection sensor, the image generating apparatus 100 enters into the monitoring space when the shovel 60 is in the operation impossible state and stands still. It fails to detect the worker who is doing work when the shovel 60 becomes ready to work, and it is possible to prevent the operator of the shovel 60 from starting the work by the shovel 60 without the worker being aware of it.

Further, when the human existence determining means 12 determines the presence or absence of the worker (moving body) using the optical flow, the image generating apparatus 100 enters into the monitoring space when the shovel 60 is in the operation impossible state and stands still It fails to detect the worker who is doing work when the shovel 60 becomes ready to work, and it is possible to prevent the operator of the shovel 60 from starting the work by the shovel 60 without the worker being aware of it.

Note that the moving object detection sensor and the optical flow do not distinguish between a stationary object in the monitoring space and a leaving of the worker from the monitoring space without setting a stationary object as a detection target. Therefore, in the configuration in which no alarm is output in any case when it is determined that there is no person in the monitoring space at the current time, the work is stopped by entering the monitoring space before the shovel 60 becomes ready to work. There is a possibility that work of the shovel 60 may be permitted despite the presence of a person. On the other hand, the image generating apparatus 100 can call attention of the operator of the shovel 60 by outputting an alarm when there is a possibility that a worker is present in the monitoring space.

Further, in the second alarm control process described above, the image generation device 100 causes the display unit 5 to display the output image even when the shovel 60 is in the operation impossible state, but does not display the output image on the display unit 5. May be When the shovel 60 is in the inoperable state, there is no possibility that the worker comes in contact with the shovel 60, and the operator does not have to monitor the periphery of the shovel 60 through the output image.

Next, with reference to FIG. 26, about the processing (hereinafter referred to as “first notification part control processing”) in which the notification part control means 15 controls the notification part 20 based on the determination result of the person presence / absence determination means 12. explain. FIG. 26 is a flowchart showing the flow of the first notification unit control process executed by the image generation apparatus 100 mounted on the shovel 60 shown in FIG. 11, and the notification unit control means 15 repeats this process at a predetermined cycle. The first notification unit control process is executed.

First, the notification unit control means 15 refers to the determination result of the work machine state determination means 14 and determines whether or not the shovel 60 is in the work available state (step S51).

When it is determined that the shovel 60 is in the work enable state (YES in step S51), the notification unit control means 15 outputs a work enable state signal to the notification unit 20, and the shovel 60 is in the work enable state. The notification is made (step S52). In the present embodiment, the notification unit control means 15 causes the three indicator lamps as the notification unit 20 to emit light in red to notify surrounding people that the shovel 60 is ready for work. Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In addition, below, this state of the alerting | reporting part 20 is made into a "operation possible alerting | reporting state."

On the other hand, when it is determined that the shovel 60 is not in the work enable state (NO in step S51), the notification unit control means 15 outputs the operation impossible state signal to the notification unit 20, and the shovel 60 is in the operation impossible state. To notify that (step S53). In the present embodiment, the notification unit control unit 15 causes the three indicator lamps as the notification unit 20 to emit light in green and the shovel 60 is not in the work enable state, that is, the shovel 60 is in the operation impossible state. Inform people of Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In addition, below, this state of the alerting | reporting part 20 is made into a "operation impossible alerting | reporting state."

Next, with reference to FIG. 27, processing for determining whether or not work is permitted by the shovel 60 based on the determination result of the human presence / absence determination means 12 (hereinafter referred to as “work permission determination process”). Will be explained. FIG. 27 is a flow chart showing a flow of the work permission determination process executed by the image generation apparatus 100 mounted on the shovel 60 shown in FIG. 11, and the work permission determination means 16 repeatedly performs this work permission at a predetermined cycle. Execute the decision process.

First, the work permission determination means 16 refers to the determination result of the work machine state determination means 14 to determine whether the shovel 60 is in the work enable state (step S61).

When it is determined that the shovel 60 is in the work ready state (YES in step S61), the work permission determination means 16 refers to the determination result of the person existence determination means 12 and determines whether or not a person exists around the shovel 60. It is determined (step S62).

If it is determined that a person is present around the shovel 60 (YES in step S62), the work permission determination unit 16 prohibits the work by the shovel 60 (step S63). Specifically, the work permission determination means 16 outputs, for example, a work prohibition signal to the gate lock valve, shuts off the flow of hydraulic oil between the control valve and the operation lever, etc., and invalidates the operation lever, etc. Prohibits the work by the shovel 60. In this case, the work permission determination means 16 prohibits the work by the shovel 60 even when the gate lock lever 8 is in the unlocked state. However, the work permission determination unit 16 may release the prohibition of the work by the shovel 60 when a predetermined condition is satisfied, such as when the alarm output by the alarm control unit 13 is stopped.

On the other hand, when it is determined that there is no person around the shovel 60 (NO in step S62), the work permission determination unit 16 permits the work by the shovel 60 (step S64). Specifically, the work permission determination means 16 outputs, for example, a work permission signal to the gate lock valve, causes the hydraulic fluid to communicate between the control valve and the control lever, and makes the control lever or the like effective. Thus, the work by the shovel 60 is permitted.

In addition, when it determines with the shovel 60 not being in an operation possible state (NO of step S61), the operation | use permission determination means 16 complete | finishes this operation permission determination process, without determining the permission of the operation by the shovel 60. This is because the work by the shovel 60 is already prohibited.

With the above configuration, the image generating apparatus 100 can notify persons around the shovel 60 whether or not the shovel 60 is in a workable state. In addition, when a person is present around the shovel 60, the image generating apparatus 100 can prohibit the work by the shovel 60 even if the shovel 60 is in a workable state. In addition, even if a person who looks at the notification unit 20 in the work ready notification state approaches the shovel 60 even if a person is present around the shovel 60, the work by the shovel 60 is prohibited. It can recognize that it does not start moving unexpectedly. Therefore, a person who looks at the notification unit 20 in the work availability notification state can safely approach the shovel 60. In addition, the person who saw the notification unit 20 in the work non-permission notification state starts the shovel 60 unexpectedly because the work by the shovel 60 is prohibited regardless of whether there is a person around the shovel 60 or not. Can recognize that there is no Therefore, a person who looks at the notification unit 20 in the inoperable notification state can approach the shovel 60 with confidence. In this manner, the image generating apparatus 100 can more appropriately notify a person present around the shovel 60 the state of the shovel 60.

Next, referring to FIG. 28, the process of controlling informing unit 20 based on the determination result of human presence / absence determination means 12 and the determination result of work machine state determination means 14 of notification unit control means 15 (hereinafter referred to as “ The second notification unit control process will be described. FIG. 28 is a flow chart showing the flow of the second notification unit control process executed by the image generation apparatus 100 mounted on the shovel 60 shown in FIG. 11, and the notification unit control means 15 repeats this process at a predetermined cycle. The second notification unit control process is executed. Further, in parallel with the second notification unit control processing by the notification unit control unit 15, the work permission determination unit 16 repeatedly executes the work permission determination processing at a predetermined cycle.

First, the notification unit control means 15 refers to the determination result of the work machine state determination means 14 and determines whether or not the shovel 60 is in the work enable state (step S71).

When it is determined that the shovel 60 is in the work-enabled state (YES in step S71), the informing unit control means 15 refers to the determination result of the person presence / absence determination means 12 and determines whether or not a person exists around the shovel 60. It is determined (step S72).

When it is determined that a person is present around the shovel 60 (YES in step S72), the notification unit control unit 15 outputs a work possible / person detection state signal to the notification unit 20, and the shovel 60 is in a work possible state. And notifies that it is in a state where a person is detected by the person detection sensor 6 (hereinafter referred to as "person detection state") (step S73). In the present embodiment, the notification unit control unit 15 causes the three indicator lamps as the notification unit 20 to emit light in red, and notifies surrounding people that the shovel 60 is in the operation enable state and in the human detection state. Do. Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In addition, below, let this state of the alerting | reporting part 20 be a "working possible / person detection alerting | reporting state." Further, in this case, the work by the shovel 60 is prohibited by the work permission determination means 16. However, the work permission determination unit 16 may release the prohibition of the work by the shovel 60 when a predetermined condition is satisfied, such as when the alarm output by the alarm control unit 13 is stopped.

On the other hand, when it is determined that a person does not exist around the shovel 60 (NO in step S72), the notification unit control means 15 outputs a work possible / non-detected state signal to the notification unit 20, and the shovel 60 It is informed that it is in a state in which it is possible to work and that no person is detected by the person detection sensor 6 (hereinafter referred to as "person non-detection state") (step S74). In the present embodiment, the notification unit control unit 15 causes the three indicator lamps as the notification unit 20 to emit light in yellow so as to notify surrounding people that the shovel 60 is in the operation enable state and in the human non-detection state. Make it Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In addition, below, let this state of the alerting | reporting part 20 be a "working possible / person non-detection alerting | reporting state." Further, in this case, the work by the shovel 60 is permitted by the work permission determination means 16.

When it is determined that the shovel 60 is not in the work enable state (NO in step S71), the informing unit control means 15 outputs the operation impossible state signal to the informing unit 20 to notify the informing unit 20 of the operation impossible informing state Then, the operator is informed that the shovel 60 is in the inoperable state (step S75). In the present embodiment, the notification unit control means 15 causes the three indicator lamps as the notification unit 20 to emit light in green to notify surrounding people that the shovel 60 is in an operation impossible state. Further, the notification unit control means 15 may turn on or turn off the indicator lamp.

Further, even when the notification unit control means 15 determines that the shovel 60 is not in the work-enabled state, whether or not a person exists around the shovel 60 with reference to the determination result of the person existence / non-existence determination means 12 It may be determined.

In this case, when it is determined that a person is present around the shovel 60, the notification unit control means 15 outputs an operation impossible / person detection state signal to the notification unit 20, and the shovel 60 is in the operation impossible state and the person is It may be informed that it is in the detection state. Specifically, the notification unit control unit 15 causes the three indicator lamps as the notification unit 20 to emit light in orange, for example, to notify surrounding people that the shovel 60 is in the operation impossible state and in the human detection state. You may do so. Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In the following, this state of the notification unit 20 will be referred to as “a state where the operation is not possible / person detection notification”.

In addition, when it is determined that a person is not present around the shovel 60, the notification unit control means 15 outputs an operation impossible / person non-detection state signal to the notification unit 20, and the shovel 60 is in an operation impossible state It may be informed that it is a person non-detection state. Specifically, for example, the notification unit control unit 15 causes the three indicator lamps as the notification unit 20 to emit light in blue, and the surrounding person is in a state in which the shovel 60 is in an operation impossible state and in a person non-detection state. It may be notified. Further, the notification unit control means 15 may turn on or turn off the indicator lamp. In addition, below, let this state of the alerting | reporting part 20 be "the operation impossible / person non-detection alerting | reporting state."

As described above, the informing unit control means 15 sets three informing states (operation impossible notification state, operation possible / person detection notification state, and operation possible / not person detection notification state) in each of the notification units 20 (indicator lamps). The individual notification units 20 are controlled to be distinguishable. In addition, the notification unit control means 15 has four notification states (operation impossible / person detection notification state, operation not possible / person non-detection notification state, operation possible / person detection notification state, and the like) in each notification unit 20 (indicator lamp) The individual notification units 20 may be controlled so as to be able to distinguish between work possible / person non-detection notification state).

Alternatively, the image generation apparatus 100 may include three additional notification units (not shown) for notifying persons around the shovel 60 of the determination result of the person existence / non-existence determination means 12 separately from the three notification units 20. It may be provided. In this case, the notification unit control means 15 controls each notification state of the notification unit 20 so as to distinguish between the operation impossible notification state and the operation possible notification state, so that the person detection state and the person non-detection state can be distinguished. Control the notification status of each additional notification unit.

With the above configuration, the image generating apparatus 100 can notify people around the shovel 60 whether or not the shovel 60 is in a workable state, and at the same time, whether or not the shovel 60 is in a person detection state We can notify people around 60. As a result, a person who looks at the notification unit 20 can know whether or not he / she is detected by the shovel 60, independently of whether the shovel 60 is in the work enable state. In addition, when a person is present around the shovel 60, the image generating apparatus 100 can prohibit the work by the shovel 60 even if the shovel 60 is in a workable state. In addition, even if a person who sees the notification unit 20 in the work ready / person detection / notification state approaches the shovel 60 temporarily, the work by the shovel 60 is prohibited when there is a person around the shovel 60 , It can be recognized that the shovel 60 does not start moving unexpectedly. Therefore, a person who looks at the notification unit 20 in the work enable / person detection / notification state can safely approach the shovel 60. In addition, a person who sees the notification unit 20 in the inoperable / human detecting / not informing / non-human detecting / not informing state prohibits the operation by the shovel 60 regardless of whether there is a person around the shovel 60 or not. Because of this, it can be recognized that the shovel 60 does not start suddenly. Therefore, a person who looks at the notification unit 20 in the work non-permission / person detection / notification state or the work non-permission / person non-detection / notification state can safely approach the shovel 60. In this manner, the image generating apparatus 100 can more appropriately notify a person present around the shovel 60 the state of the shovel 60.

Further, in the above-described embodiment, the notification unit 20 notifies the plurality of notification states in a distinguishable manner by changing the light emission color, but changes the character information to be displayed to notify the plurality of notification states in a distinguishable manner. May be

Although the preferred embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and various modifications and substitutions may be made to the above-described embodiments without departing from the scope of the present invention. It can be added.

For example, in the above-described embodiment, the image generation apparatus 100 adopts a cylindrical space model MD as a space model, but may adopt a space model having another columnar shape such as a polygonal column, etc. A space model composed of two sides may be adopted, or a space model having only sides may be adopted.

In addition, the image generation apparatus 100 is mounted on a self-propelled shovel together with a camera and a human detection sensor while having movable members such as a bucket, an arm, a boom, and a turning mechanism. Then, the image generation device 100 configures an operation support system that supports the movement of the shovel and the operation of the movable members while presenting the surrounding image to the operator. However, the image generating apparatus 100 may be mounted on a working machine having no turning mechanism such as a forklift, an asphalt finisher, etc. together with the camera and the human detection sensor. Alternatively, the image generating apparatus 100 may be mounted together with the camera and the human detection sensor on a working machine having movable members but not self-propelled, such as industrial machines or fixed cranes. And the image generation apparatus 100 may comprise the operation assistance system which assists operation of those working machines.

In addition, although the periphery monitoring device has been described as an example of the image generation device 100 including the camera 2 and the display unit 5, the periphery monitoring device may be configured as a device not including an image display function by the camera 2, the display unit 5 or the like. For example, as shown in FIG. 29, the periphery monitoring device 100A as a device that executes the first notification unit control process or the second notification unit control process is a camera 2, an input unit 3, a storage unit 4, a display unit 5, coordinates. The association unit 10, the image generation unit 11, and the alarm control unit 13 may be omitted.

The present application claims priority based on Japanese Patent Application No. 2013-057400 filed on March 19, 2013, and the entire contents of this Japanese patent application are incorporated herein by reference.

DESCRIPTION OF SYMBOLS 1 ... Control part 2 ... Camera 2L ... Left-hand side camera 2 R .. Right-hand side camera 2 B .. Rear view camera 3 ... Input part 4 ... Storage part 5 ... Display part 6 ..・ Human detection sensor 6L ・・・ Left side detection sensor 6R ・・・ Right side detection sensor 6B ・・・ Rear person detection sensor 7 ・・・ Alarm output part 7L ・・・ Left side alarm output part 7B ・・・Rear side alarm output unit 7R Right side alarm output unit 8 Gate lock lever 9 Ignition switch 10 Coordinate matching unit 11 Image generation unit 12 Human presence / absence judgment Means 13 · · · Alarm control means 14 · · · work machine state determination means 15 · · · notification unit control means 16 · · · work permission decision means 20 · · · notification unit 40 · · · · image corresponding to the input image · space model 41 ··· Space model · processing target image correspondence map 42 ... processing target image · output image correspondence map 60 ... shovel 61 ··· lower traveling body 62 · · · turning mechanism 63 · · · upper turning body 64 ... Cab 100 ... Image generation device 100A ... Peripheral monitoring device

Claims

A peripheral monitoring device for a working machine including a notification unit that is visible from the periphery of the working machine,
A working machine state determination unit that determines whether the working machine is in a workable state;
A human presence / absence determination means for determining the presence or absence of a person around the work machine;
Notification unit control means for controlling the notification unit;
Operation permission determination means for determining permission or rejection of the work by the work machine;
The notification unit control means makes the notification state of the notification unit different depending on whether the work machine is determined to be in the workable state or not determined to be in the workable state.
The work permission determination means determines the permission of the work by the work machine based on the determination result of the person existence determination means.
Peripheral monitoring device for work machines.
The notification unit control means makes the notification state of the notification unit different depending on whether it is determined that there is a person around the work machine and it is determined that there is no person around the work machine. ,
The work machine peripheral monitoring device according to claim 1.
The work permission determination means prohibits the work by the work machine when it is determined that a person is present around the work machine even when it is determined that the work machine is in the work enable state. Do,
The work machine peripheral monitoring device according to claim 1.
When the ignition switch is in the off state or the gate lock lever is in the lock state, the work machine state determination unit determines that the work machine is not in the operation enable state, and the ignition switch is in the on state. Further, when the gate lock lever is in the unlocked state, it is determined that the work machine is in the operable state.
The work machine peripheral monitoring device according to claim 1.
The notification unit control means is configured such that a person present around the work machine can identify a state in which the work machine is determined to be in a workable state and a person is determined to be present around the work machine. When the notification unit is operated, the work permission determination means determines that the work machine is in the work enable state, and it is determined that a person is present around the work machine. Permit work by work machine,
The work machine peripheral monitoring device according to claim 1.