JP6321977B2 - Perimeter monitoring equipment for heavy machinery - Google Patents

Description

  The present invention relates to a heavy equipment periphery monitoring device that is mounted on a heavy machine such as a crane truck, and images the situation around the heavy machine and presents it to a driver.

  In recent years, a plurality of cameras are installed around a vehicle, and images obtained by the plurality of cameras are coordinate-transformed and synthesized, and the situation around the vehicle is expressed as a single overhead image and presented to the driver. A peripheral monitoring system has been put into practical use. Such a peripheral monitoring system is applied not only to passenger cars but also to heavy machinery.

  For example, in the monitor device for a work machine described in Patent Document 1, a single image is obtained by performing coordinate conversion on images around a heavy machine taken by a plurality of cameras installed around the heavy machine (for example, a crane truck). Are displayed in the same orientation while shifting the display range.

Patent No. 5066198

  However, the monitor device described in Patent Document 1 always displays an image in the same direction on the display unit even when the arm is extended or swiveled while the crane is working. .

  In such a conventional monitor device, even when the work area is changed by adjusting the length and direction of the arm during the work, it is always displayed in the same direction. Further, for example, when the work area is changed by turning the arm, the position of the area where attention should be paid in the displayed image changes. In particular, when the arm is extended during work, attention should be paid to the position immediately below the tip of the arm moving in a direction away from the crane vehicle. In such a case, since it becomes impossible to monitor the range that needs to be visually recognized, it is difficult for the driver of the crane vehicle to immediately recognize the direction and position of the obstacle even when viewing the displayed image. It was.

  The present invention has been made in view of the above problems, and by displaying an image around a heavy machine in a form according to the working state of the heavy machine (for example, the length and direction of the arm of a crane truck), the presence of an obstacle is present. An object of the present invention is to provide a heavy equipment peripheral monitoring device that can immediately recognize the direction and position of the heavy equipment.

  The heavy equipment periphery monitoring device according to the present invention is capable of confirming obstacles existing around heavy equipment with high visibility.

  That is, the periphery monitoring device for heavy machinery according to the present invention includes an arm that can change its length and elevation angle, and an upper swing body that can be swung right and left by a driver and a lower travel body that can travel and move. An imaging unit that is attached to a plurality of surroundings of the lower traveling body and captures a plurality of images including the ground around the heavy machinery, and converts a plurality of images captured by the imaging unit, respectively. A virtual image generation unit that combines and generates one virtual image that is predicted to be observed from a virtual imaging unit that is virtually installed in a predetermined position above the heavy machine in a predetermined direction And a display unit for displaying the virtual image, which is installed on the upper swing body so as to be visible to the driver, in the peripheral monitoring device for heavy machinery, the length of the arm, the elevation angle of the arm, The pivot angle of the arm. A heavy machine attitude measurement unit that measures each of the above, and an icon superimposition display unit that superimposes an icon simulating the attitude of the heavy machine that is predicted to be observed from the virtual imaging unit on the virtual image. The virtual image generation unit is predicted to be observed from a virtual imaging unit installed in a direction corresponding to the posture at a position corresponding to the posture of the heavy machinery measured by the heavy machinery posture measurement unit. An image is generated, and the icon superimposing display unit superimposes an icon corresponding to the posture of the heavy machine on the virtual image, and displays the virtual image on which the icon is superimposed on the display unit.

  According to the peripheral monitoring apparatus for heavy machinery according to the present invention configured as described above, a plurality of images including the ground around the heavy machinery captured by the imaging unit attached to the periphery of the lower traveling body of the heavy machinery, The virtual image generation unit is predicted to be observed from a virtual imaging unit installed in a position and direction according to the arm turning angle, the arm length, and the arm elevation angle measured by the heavy equipment attitude measurement unit. The icon superimposing display unit superimposes an icon corresponding to the attitude of the heavy machine on the virtual image, and the driver of the heavy machine converts the virtual image on which the icon is superimposed. Since the information is displayed on the visible display unit, the heavy equipment driver can immediately recognize the position and direction of the obstacle in the heavy equipment work area without depending on the posture change of the heavy equipment during work.

  According to the heavy equipment periphery monitoring device of the present invention, it is possible to immediately recognize the position and direction of an obstacle in the work area of the heavy equipment without depending on the posture change of the heavy equipment.

It is an external view explaining the parameter regarding the name of each part of the crane vehicle which is an example of a heavy machine, and the attitude | position of a heavy machine. FIG. 2 is a hardware block diagram illustrating a hardware configuration of the heavy equipment periphery monitoring device according to the first embodiment. It is a functional block diagram explaining the functional structure of the periphery monitoring apparatus for heavy machines in Example 1. FIG. It is a side view explaining the coordinate system required when demonstrating Example 1 which is one Embodiment of this invention. It is a top view explaining the coordinate system required when explaining Example 1 which is one embodiment of the present invention. It is a figure which shows the example of the virtual image produced | generated in Example 1 and estimated to be observed from the virtual imaging part. It is a figure which shows the example which superimposed the icon on the virtual image produced | generated in Example 1 and estimated to be observed from the virtual imaging part. It is a figure which shows the structure of a coordinate conversion synthetic | combination table (LUT) corresponding to the case where the installation height of a virtual imaging part is a low position, and the turning angle of an arm is 0. It is a figure which shows the internal structure of a coordinate transformation synthetic | combination table (LUT). It is a figure which shows the example of a coordinate conversion synthetic | combination table (LUT) in case the installation height of a virtual imaging part is a middle position, and the turning angle of an arm is 0. It is a figure which shows the example of a coordinate conversion synthetic | combination table (LUT) in case the installation height of a virtual imaging part is a high position, and the turning angle of an arm is 0. In Example 1, it is a figure which shows the list of the coordinate transformation synthetic | combination table (LUT) prepared beforehand. It is a figure which shows the content of the icon table prepared in advance in Example 1. FIG. It is a figure which shows the virtual image with which the icon produced | generated in Example 1 was superimposed, and is an example of a virtual image in case the turning angle of an arm is 0 and an arm projection length is short. It is a figure which shows the virtual image with which the icon produced | generated in Example 1 was superimposed, and is an example of a virtual image in case the turning angle of an arm is a small angle in the right direction, and arm projection length is short. It is a figure which shows the virtual image on which the icon was superimposed produced | generated in Example 1, and is an example of a virtual image in case the turning angle of an arm is a middle angle in the right direction, and an arm projection length is short. It is a figure which shows the virtual image on which the icon was superimposed produced | generated in Example 1, and is an example of a virtual image in case the turning angle of an arm is a large angle in the right direction, and arm projection length is short. It is a figure which shows the virtual image with which the icon produced | generated in Example 1 was superimposed, and is an example of a virtual image in case the turning angle of an arm is 0 and an arm projection length is medium. It is a figure which shows the virtual image with which the icon was superimposed produced | generated in Example 1, and is an example of a virtual image in case the turning angle of an arm is 0 and arm projection length is long. 3 is a flowchart illustrating a flow of the entire process performed in the first embodiment. 12 is a flowchart showing a flow of determination processing of an arm turning direction and a turning amount performed in the flowchart shown in FIG. 11. It is a flowchart which shows the flow of the process which calculates the installation position of the virtual imaging part performed in the flowchart shown in FIG. It is a flowchart which shows the flow of the coordinate transformation synthesis process performed in the flowchart shown in FIG. 12 is a flowchart illustrating a flow of icon selection / superimposition processing performed in the flowchart illustrated in FIG. 11. It is a figure which shows the modification of the virtual image on which the icon was superimposed produced | generated in Example 1, and is an example of a virtual image in case the turning angle of an arm is 0 and arm projection length is short. It is a figure which shows the modification of the virtual image with which the icon was superimposed produced | generated in Example 1, and is an example of a virtual image in case the turning angle of an arm is a small angle in the right direction, and an arm projection length is short. It is a functional block diagram explaining the function structure of the periphery monitoring apparatus for heavy machines in Example 2 which is one Embodiment of this invention. It is a figure which shows an example of the virtual image produced | generated in Example 2 and displayed when a heavy machine advances. It is a figure which shows an example of the virtual image produced | generated in Example 2 and displayed when a heavy machine reverse | retreats. 10 is a flowchart illustrating the overall flow of processing performed in the second embodiment.

  Hereinafter, Example 1 which is one embodiment of the present invention will be described with reference to the drawings.

  In this embodiment, the present invention is applied to a heavy equipment periphery monitoring device that captures and displays the status of obstacles around heavy equipment.

(Explanation of schematic structure of crane truck)
First, the schematic structure of a crane truck, which is an example of a heavy machine, and the names of each part will be described with reference to FIG.

  The heavy equipment periphery monitoring device 10 of this embodiment is mounted on a construction machine such as a crane vehicle 12, for example. The crane vehicle 12 includes an upper swing body 20 and a lower traveling body 30. The upper-part turning body 20 includes a cabin 22 on which a driver 8 (not shown) is boarded, an arm 24 that performs work such as load transportation, and a hook 28 that is installed on the arm 24 via a wire 26 and lifts the load 29. Consists of.

  The upper turning body 20 can control the turning angle φ, the elevation angle ε, the arm length L, and the wire length (distance between the tip of the arm 24 and the hook 28) according to the operation of the driver 8. .

  The lower traveling body 30 includes a plurality of wheels and a power source that drives the wheels and the upper swing body 20. Then, a front camera 40a (imaging unit) that images the forward direction 42a of the crane vehicle 12 and a left camera 40b (imaging image) that captures the left direction 42b of the crane vehicle 12 toward the four directions around the lower traveling body 30. Part), a rear camera 40c (imaging part) for photographing the rear direction 42c of the crane truck 12, and a right side camera 40d (imaging part) for photographing the right side direction 42d of the crane truck 12. Hereinafter, the four cameras are collectively referred to as cameras (40a, 40b, 40c, 40d). Each camera (40a, 40b, 40c, 40d) is equipped with a wide-angle lens so that a wide range can be observed. Also, it is assumed that the installation positions (the height from the ground, the direction of the optical axis, and the positional relationship with other cameras) of each camera (40a, 40b, 40c, 40d) are known in advance.

(Description of System Configuration of Example 1)
Next, a hardware configuration of the heavy equipment periphery monitoring device 10 according to the first embodiment will be described with reference to FIG. The heavy equipment periphery monitoring apparatus 10 is configured to capture images taken by the front camera 40a, the left camera 40b, the rear camera 40c, the right camera 40d, and the cameras (40a, 40b, 40c, 40d) (imaging unit). An image processing calculation unit 50 that generates a single virtual image by transforming coordinates, a gyro sensor 70 that measures the turning angle φ of the arm 24 of the crane vehicle 12, and an arm length L of the crane vehicle 12 are measured. A laser distance sensor 72 that performs measurement, an inclination sensor 74 that measures an elevation angle ε of the arm 24 of the crane vehicle 12, and each sensor (gyro sensor 70, laser distance sensor 72, inclination sensor 74) and the image processing calculation unit 50. It consists of a CAN communication line 76 that transmits and receives information and a monitor 78 that displays the generated virtual image. In addition, since each sensor used is generally used for each use, description about the structure and measuring method of each sensor is abbreviate | omitted.

  The image processing calculation unit 50 further includes an AD converter (52a, 52b, 52c, 52d) that converts a composite signal (analog signal) output from each camera into a digitized image signal, and a digitized image. A decoder (54a, 54b, 54c, 54d) for converting the signal into an RGB signal or a YCbCr, H-Sync, V-Sync signal that is a luminance color difference signal, and each camera (40a, 40b, 40c, 40d) (imaging) A coordinate conversion calculation unit 56 for performing a coordinate conversion calculation on the image photographed in (1)), a FLASH ROM 58 in which an LUT (look-up table) used for the coordinate conversion calculation is stored, and a coordinate conversion calculation. SDRAM 60 as a video memory (frame buffer) for temporarily storing image signals at the time, and coordinates for coordinate conversion calculation The CPU 62 controls the conversion calculation unit 56, acquires the sensor information measured by the gyro sensor 70, the laser distance sensor 72, and the tilt sensor 74 described above via the CAN communication line 76, and the coordinate-converted RGB signal, Alternatively, it includes an encoder 64 that converts a YCbCr, H-Sync, V-Sync signal, which is a luminance color difference signal, into a composite signal (digital signal), and a DA converter 66 that converts the composite signal (digital signal) into an analog signal.

  Among the components of the image processing calculation unit 50, an AD converter (52a, 52b, 52c, 52d), a decoder (54a, 54b, 54c, 54d), a coordinate conversion calculation unit 56, an encoder 64, The DA converter 66 is configured by an ASIC 51 (Application Specific Integrated Circuit) which is an integrated circuit designed exclusively for the application of the present embodiment.

  Next, the functional block configuration of the heavy equipment periphery monitoring device 10 according to the first embodiment will be described with reference to FIG. The heavy equipment periphery monitoring apparatus 10 is configured to capture images taken by the front camera 40a, the left camera 40b, the rear camera 40c, the right camera 40d, and the cameras (40a, 40b, 40c, 40d) (imaging unit). A virtual image generation unit 100 that generates a single virtual image by performing coordinate transformation, a heavy machine posture measurement unit 90 that measures the posture and length of the arm 24 of the crane 12 (see FIG. 1), a virtual An icon table 86 storing a plurality of icons I imitating the crane vehicle 12 (see FIG. 1) to be superimposed on the image and an icon I selected from the plurality of icons I are superimposed on the virtual image. An icon superimposing display section 88, an overall control section 94 for controlling the movement of the entire apparatus, and a display section 96 for displaying the generated virtual image.

  The virtual image generation unit 100 is further photographed by a virtual imaging unit installation unit 80 that installs a virtual imaging unit at a position corresponding to the posture of the crane vehicle 12 and each camera (40a, 40b, 40c, 40d) (imaging unit). A coordinate conversion synthesis table 82 composed of an LUT (Look Up Table) that is used when each image is subjected to coordinate transformation and synthesized into one virtual image, and a coordinate transformation synthesis unit 84 that performs coordinate transformation processing and image synthesis processing. Consists of.

  The heavy equipment posture measuring unit 90 further includes a turning angle measuring unit 91 that measures the turning angle φ of the arm 24 of the crane vehicle 12, an arm length measuring unit 92 that measures the arm length L of the crane vehicle 12, and the crane vehicle 12. The arm elevation angle measuring unit 93 measures the elevation angle ε of the arm 24.

(Description of coordinate system used in Example 1)
Next, with reference to FIGS. 4A and 4B, a coordinate system necessary for describing the operation of the heavy equipment periphery monitoring device 10 according to the first embodiment will be described. FIG. 4A is a side view of the crane vehicle 12 equipped with the heavy equipment periphery monitoring device 10 according to the present embodiment as viewed from the side. FIG. 2B is a top view of the crane vehicle 12 mounted with the heavy equipment periphery monitoring device 10 according to the present embodiment as viewed from above.

  As shown in FIGS. 4A and 4B, the X axis is set in the left-right direction and the right-hand direction of the crane vehicle 12 with the intersection point of the vertical line passing the turning center point R of the upper swing body 20 of the crane vehicle 12 and the ground as the origin O. Then, the Y-axis is set in the front-rear direction and the forward direction of the crane vehicle 12, and the Z-axis is set in the up-down direction and the upward direction of the crane vehicle 12. The following description will be made using the coordinate system (X, Y, Z) set in this way.

The arm 24 of the crane vehicle 12 has a length L with the turning center R of the upper turning body 20 at the height h ₀ as a supporting end, an elevation angle ε (see FIG. 4A), and a turning angle φ. It is assumed that it faces the direction specified in (see FIG. 4B). The coordinates of the turning center point R are R (0, 0, h ₀ ).

  Here, the turning angle φ is positive when the tip C of the arm 24 is facing the right side with respect to the Y axis (φ> 0, right turning), and the tip C of the arm 24 is on the left side with respect to the Y axis. Negative (φ <0, left turn) when facing.

As shown in FIGS. 4A and 4B, the front camera 40a, the left side camera 40b, the rear camera 40c, and the right side camera 40d provided in the crane vehicle 12 are arranged obliquely downward, respectively, so It is assumed that a region including the ground within a distance K is being photographed. That is, as shown in FIG. 4B, each camera (40a, 40b, 40c, 40d) (imaging unit) captures the area of the imaging range 44a with the front camera 40a, and the left camera 40b captures the area of the imaging range 44b. The rear camera 40c is set to take a picture of the area of the imaging range 44c, and the right side camera 40d is set to take a picture of the area of the imaging range 44d. Incidentally, as shown in FIG. 4A, the distance from the origin O to the farthest point of the imaging range 44a, the respective K ₀ the distance from the origin O to the farthest point of the imaging range 44c.

  Images taken by the respective cameras (40a, 40b, 40c, 40d) (imaging units) are virtual cameras 40v (virtual imaging) virtually installed above the crane vehicle 12 by the action of the peripheral monitoring device 10 for heavy machinery. To an image (virtual image) that is predicted to be observed when looking down.

Virtual camera 40v (virtual imaging unit), as shown in FIG. 4A, is placed toward the right below the position of the height Z ₀ placed on the Z-axis. That is, the coordinates of the installation position V of the virtual camera 40v are V (0, 0, Z ₀ ). And the virtual camera 40v shall image the inside of the imaging | photography range of each camera (40a, 40b, 40c, 40d) (imaging part) installed in the crane vehicle 12. FIG.

  As shown in FIG. 4A, the installation position V of the virtual camera 40v (virtual imaging unit) passes through the point A that is the farthest point on the ground and the tip C of the arm 24 in the imaging range of the front camera 40a. It is assumed that the straight line intersects the Z axis. By providing such a rule, the installation position V of the virtual camera 40v can be uniquely determined.

That is, the installation height Z ₀ of the virtual camera 40v is calculated by (Equation 1).
Z ₀ = K ₀ * {(h ₀ + L sin ε) / (K ₀ −L cos ε)} (Formula 1)

  It is assumed that a pile Q that is an obstacle is present in front of the crane vehicle 12. This pile Q has a predetermined height from the ground, and in order to simplify the following description, the tip C of the arm 24 in the posture of the turning angle φ = 0 and the elevation angle ε is projected onto the ground from vertically above. It is assumed to exist at the position.

(Explanation of virtual image structure)
Next, an example of a virtual image predicted to be observed from the virtual camera 40v (virtual imaging unit, see FIG. 4A) will be described with reference to FIGS. 5A and 5B.

  FIG. 5A is a diagram showing an example in which images captured by the front camera 40a, the left camera 40b, the rear camera 40c, and the right camera 40d are coordinate-converted and combined into one virtual image Iv (x, y). It is.

That is, the image I _A (x, y) photographed by the front camera 40a is converted into a virtual image I _VA (x, y) predicted to be observed from the virtual camera 40v (virtual imaging unit), and the left side The image I _B (x, y) photographed by the side camera 40b is converted into a virtual image I _VB (x, y) predicted to be observed from the virtual camera 40v (virtual imaging unit), and the rear camera 40c The image I _C (x, y) photographed in step 1 is converted into a virtual image I _VC (x, y) predicted to be observed from the virtual camera 40v (virtual imaging unit), and photographed by the right-side camera 40d. The converted image I _D (x, y) is converted into a virtual image I _VD (x, y) predicted to be observed from the virtual camera 40v (virtual imaging unit). virtual image _{(I VA (x, y)} , I VB (x, y) I _{VC (x, y),} is obtained by combining I VD (x, y) a) a single picture.

Note that virtual images (I _VA (x, y), I _VB (x, y) obtained by observing images captured by the respective cameras (40a, 40b, 40c, 40d) (imaging units) from the virtual camera 40v (virtual imaging unit). ), I _VC (x, y), I _VD (x, y)), there are various methods, and any of them may be used. For example, it is assumed that a flat ground (a plane having a height of 0) is reflected in the images Ii (x, y) (i = A to D) captured by the cameras (40a, 40b, 40c, 40d). Then, by sequentially calculating which pixel (coordinate) the point of the ground is observed on the virtual image I _Vi (x, y) (i = A to D), the virtual image I _Vi (x , Y) can be generated.

In this way, the image I _A (x, y) captured by the front camera 40a is converted into the virtual image I _VA (x, y), and the image I _B (x, y) captured by the left-side camera 40b. ) Is converted into the virtual image I _VB (x, y), and the image I _C (x, y) captured by the rear camera 40c is converted into the virtual image I _VC (x, y), and the right-side camera 40d. The image I _D (x, y) photographed in is converted into a virtual image I _VD (x, y). The converted virtual image I _Vi (x, y) (i = A to D) is subjected to necessary rotation operations so that the directions of the virtual images are aligned, and one image is obtained as shown in FIG. 5A. To the virtual image Iv (x, y).

At this time, the pile Q in front of the crane vehicle 12 is converted into an image Q ′. The stake Q is converted into the image Q ′ when the virtual image I _VA (x, y) is generated, as described above, the image I _A (x, y) captured by the front camera 40a is all grounded. This is because the pile Q having a height from the ground is viewed from the virtual camera 40v (virtual imaging unit), so that the higher the position of the pile Q, the wider the position is observed. In other words, an object having a height from the ground is observed as a region having a spread in the left-right direction by an amount corresponding to the height from the ground.

  Further, when the virtual image Iv (x, y) is generated, the position where the crane vehicle 12 is present becomes a blind spot of each camera (40a, 40b, 40c, 40d) (imaging unit), and thus video information is lost. As shown in FIG. 5A, for example, a pixel value 0 is stored in the virtual image Iv (x, y), and the crane vehicle region Rv is set at a location where information is missing.

As described in FIG. 4A, the height _{Z 0} installation of the virtual camera 40v (virtual imaging unit), the tip of the arm 24 C (see FIG. 4A) is in the imaging range of the virtual camera 40v (virtual imaging unit) It is set to be located at the end. Therefore, the position corresponding to the tip C of the arm 24 in the crane vehicle region Rv is located at the upper end of the virtual image Iv (x, y).

  Thereafter, as shown in FIG. 5B, an icon I simulating the attitude of the crane vehicle 12 is superimposed on the crane vehicle region Rv as if looking down from above the crane vehicle 12 (see FIG. 1).

(Explanation of coordinate conversion synthesis table)
Next, the coordinate conversion synthesis table 82 (see FIG. 3) used when generating the virtual image Iv (x, y) will be described with reference to FIGS. 6A, 6B, 7, 8A, and 8B.

FIG. 6A shows a state in which the arm length L of the crane vehicle 12 is short and the virtual image Iv (x, y) when the turning angle φ of the arm 24 is substantially 0 (neutral position, −α <φ ≦ α). FIG. 9 is a diagram illustrating a structure of a coordinate transformation synthesis table T _A (x, y), which is an example of a coordinate transformation synthesis table 82 used for generating (). Hereinafter, for the sake of simplicity, the coordinate conversion synthesis table T _A (x, y) is simply referred to as a coordinate conversion synthesis table T _A.

Coordinate transformation Synthesis table T _A, as shown in FIG. 6A, to form a LUT (lookup table) in the form of two-dimensional frame memory. And it corresponds to each camera (40a, 40b, 40c, 40d) (imaging part) in a region corresponding to each virtual image I _Vi (x, y) (i = A to D) explained in FIG. 5A and FIG. 5B. And the coordinate values (left-right direction coordinate value (x address) and up-down direction coordinate value (y address)) of pixels photographed by each camera (40a, 40b, 40c, 40d) are stored. In the crane vehicle area Rv described above, a specific value indicating the crane truck area is stored.

Coordinate transformation Synthesis table T _A is specifically is composed of a frame memory bit width of 24 bits as shown in FIG 6B, for example. Then, the upper eight bits LUT number and the camera number i representing the coordinate transformation synthesis table T _A are stored respectively in the camera specified by the camera number i to middle 8 bits (i = to D) The x address value of the photographed image Ii (x, y) (i = A to D) is stored, and the lower eight bits are photographed by the camera specified by the camera number i (i = A to D) The y address of the image Ii (x, y) (i = A to D) is stored.

Right coordinate transformation synthesis table T _A having such a structure from the left, while sequentially scanning from top to bottom, and reads the information stored in the scanned pixel, camera number information read out is indicated Identified by camera number i (i = A to D) with reference to i (i = A to D) and address (x, y) indicating the pixel of the image taken by the camera identified by camera number i the pixel values of the camera of the address (x, y), by storing the corresponding position on the coordinate transformation synthesis table T _a, the coordinate conversion processing and image synthesizing processing can be carried out simultaneously.

In addition, it is necessary to distinguish the position corresponding to the crane vehicle area | region Rv mentioned above from the area | region where an actual camera image | video is stored, as shown to FIG. 6A. Therefore, in a region corresponding to the crane vehicle region Rv coordinate conversion synthetic table T _A is storing a particular value in a position to store the camera number i, for example, "0".

In addition, each camera (40a, 40b, 40c, 40d) (imaging unit) is equipped with a wide-angle lens so that a wide range can be imaged as much as possible, so that the captured image (I _A (x, y ), I _B (x, y), I _C (x, y), I _D (x, y)). Coordinate transformation Synthesis table T _A is also responsible for correcting the distortion.

That is, the images (I _A (x, y), I _B (x, y), I _C (x, y), I _D (x, the coordinate transformation rules for correcting y)) in the distorted there is no image can be stored in the coordinate conversion synthesis table T _a.

Specifically, the coordinate value (xq, yq) on the image Ii (x, y) including the distortion and the virtual image I _Vi (x, y with the distortion corrected by the transformation of the coordinates by the calibration. ) (I = A to D) A plurality of pairs of coordinate values (xr, yr) are acquired, a simultaneous equation is established assuming that these pixel pairs are connected by a coordinate transformation matrix, and the simultaneous equations are solved. Thus, a coordinate transformation matrix for correcting distortion can be determined. Since such a distortion correction method is known, a detailed description is omitted. The coordinate conversion synthesis table T _A includes such distortion correction, rules of the coordinate transformation and a plurality of coordinate synthesis are both taken into account for the virtual image conversion to the virtual image is stored.

  Next, variations of the coordinate conversion synthesis table 82 will be described with reference to FIGS. 7A and 7B.

  In this embodiment, a virtual image Iv (x, y) corresponding to the posture of the arm 24 of the crane vehicle 12 is generated. That is, it is necessary to prepare a plurality of coordinate conversion synthesis tables corresponding to the posture of the arm 24.

Figure 7A is a moderate arm length L is a diagram showing the structure of a rotation angle φ is 0 coordinate conversion synthesis table T _B which is an example of a coordinate conversion synthesis table 82 when a (neutral position). The coordinate transformation synthesis table T _B, when compared with the coordinate transformation synthesis table T _A of FIG. 6A, the coordinate conversion synthesis table for generating a virtual image of monitoring the periphery of the crane 12 (see FIG. 1) more extensive It has become.

  That is, in the case of FIG. 7A, since the virtual camera 40v (virtual imaging unit) is installed at a higher position than that in FIG. 6A, the vicinity of the crane vehicle 12 is monitored in a wider range. Therefore, compared with FIG. 6A, the crane vehicle region Rv becomes smaller. As the arm length L increases, the projection length of the arm 24 onto the ground (corresponding to Lcosε in FIG. 4A) becomes longer, so the length of the region corresponding to the arm 24 in the crane vehicle region Rv is longer. Become.

7B is the arm length L is long is a diagram showing the structure of a coordinate transformation synthetic table T _C which is an example of a coordinate conversion synthesis table 82 when the turning angle φ is zero (neutral position). The coordinate transformation synthesis table T _C is different from the coordinate conversion synthesis table T _B shown in FIG. 7A, the coordinate conversion synthesis table for generating a virtual image of monitoring the periphery of the crane 12 (see FIG. 1) more extensive It has become.

  That is, in the case of FIG. 7B, since the virtual camera 40v (virtual imaging unit) is installed at a higher position than that in FIG. 7A, the vicinity of the crane vehicle 12 is monitored in a wider range. Therefore, compared with FIG. 7A, the crane vehicle region Rv becomes smaller. As the arm length L increases, the projection length of the arm 24 onto the ground (corresponding to Lcosε in FIG. 4A) further increases, so that the length of the region corresponding to the arm 24 in the crane vehicle region Rv is further increased. become longer.

FIG. 8 is a diagram showing a list of coordinate conversion synthesis table 82 (LUT) prepared in advance. The vertical axis of FIG. 8 corresponds to the height Z ₀ installation of the monitoring range as a virtual camera 40v of the virtual camera 40v (virtual imaging unit), the coordinate conversion synthesis table 82 shown in FIG. 8, the virtual camera 40v high This is set in three stages. Then, as it goes down in the vertical axis direction in FIG. 8, the monitoring range is wider higher height Z ₀ installation of the virtual camera 40v.

  Further, the horizontal axis of FIG. 8 corresponds to the turning angle φ of the arm 24 of the crane vehicle 12, and indicates that the arm 24 rotates leftward as it goes to the left, and the turning angle φ increases. It shows that the arm 24 rotates in the right direction and the turning angle φ increases as it goes to the right.

  In the coordinate conversion synthesis table 82 shown in FIG. 8, the turning angle φ of the arm 24 of the crane vehicle 12 is set to the neutral position (−α <φ ≦ α) and the turning angle φ in the right direction is small (α <φ ≦ β). , Medium (β <φ ≦ γ), large (γ <φ ≦ θ), left turn angle φ is small (−α <φ ≦ −β), medium (−β <φ ≦ −γ), It is divided into a total of 7 stages of large (−γ <φ ≦ −θ). Note that α, β, γ, and θ are preset threshold values of the turning angle φ, and are positive constants having a relationship of α <β <γ <θ.

  That is, when the arm 24 turns left and right, a virtual image Iv (predicted to be taken when the virtual camera 40v (virtual imaging unit) is rotated around the optical axis by an amount corresponding to the turning angle φ). x, y) is generated.

In the first embodiment, a total of 21 coordinate conversion synthesis tables corresponding to the three stages of installation height Z _{0 of} the virtual camera 40v and the seven stages of turning angles φ shown in FIG. 8 are prepared. Yes. Each coordinate conversion synthesis table T _Xy (X = A to C, y = a to g) is, for example, a coordinate conversion synthesis table T _Ay (y = a to) at a position where the installation height Z ₀ of the virtual camera 40v is low. g), coordinate conversion synthesis table T _By (y = a to g) with the installation height Z ₀ at the middle position, coordinate conversion synthesis table T _Cy (y = a to g) with the installation height Z ₀ at the higher position. Further, the coordinate transformation synthesis tables T _Xa , T _Xb , T _Xc , T _Xd , respectively, according to the state in which the turning angle φ is large from the left turning angle to the right turning angle, respectively. T _Xe , T _Xf , T _Xg (X = A to C).

The coordinate transformation synthetic table T _A of Figure 6A described above, corresponding to the coordinate transformation synthesis table T _Ad in FIG. 8, the coordinate conversion synthetic table T _B in FIG. 7A, coordinate transformation synthesis table T _Bd 8 correspondingly, the coordinate transformation synthetic table T _C in Figure 7B corresponds to the coordinate transformation synthesis table T _Cd of FIG.

(Explanation of icons)
Next, variations of the icon I that simulates the posture of the crane vehicle 12 will be described with reference to FIG.

FIG. 9 shows the contents of the icon table 86. The icon I _Xy (X = A to C, y = a to g) is displayed on each coordinate conversion synthesis table T _Xy (X = A to C, y = a to g) prepared in the coordinate conversion synthesis table 82 described above. Are prepared in a one-to-one correspondence. That is, as shown in FIG. 9, each icon I _Xy has a shape when the crane vehicle 12 (see FIG. 1) is looked down from directly above, and the installation height Z ₀ of the virtual camera 40v _{and the} crane vehicle 12 It has a form and a size corresponding to the arm length L and the turning angle φ. These icons I may be generated using CG (Computer Graphic), or may be generated from a real image obtained by actually shooting the crane vehicle 12 from directly above.

Note that the icon I shown in FIG. 5B corresponds to the icon I _Ad in the icon table of FIG.

  (Explanation of virtual image examples)

  Next, an example of the virtual image Iv (x, y) generated in the crane vehicle 12 placed in the environment shown in FIGS. 4A and 4B will be described using FIGS. 10A to 10F.

FIG. 10A shows an example of a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is 0 (neutral position), the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. FIG. At this time, on the virtual image Iv (x, y), coordinate transformation synthesis is performed such that the installation height Z ₀ of the virtual camera 40v is low and the turning angle φ indicates the neutral position so that the arm 24 faces upward. and table _{T Ad} is selected, on the basis of the coordinate transformation synthesis table _{T Ad,} each camera (40a, 40b, 40c, 40d) virtual image captured by the imaging section, each image _I Vi (x, y) The coordinates are transformed into (i = A to D) and synthesized. Then, from among the icons table 86, low height Z ₀ installation of the virtual camera 40v, turning angle φ is superimposed is selected icons I _Ad indicating the neutral position. At this time, the pile Q photographed by the front camera 40a is displayed as an image Q ′ in the virtual image I _VA (x, y), and the driver 8 operates the arm 24 while paying attention to the image Q ′. To do.

FIG. 10B shows an example of the virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is “small” turning right, the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. FIG. At this time, on the virtual image Iv (x, y), the installation height Z ₀ of the virtual camera 40v is low and the turning angle φ indicates the small right turning state so that the arm 24 faces upward. A coordinate conversion synthesis table T _Ae is selected, and based on this coordinate conversion synthesis table T _Ae , images taken by the cameras (40a, 40b, 40c, 40d) (imaging units) are respectively virtual images I _Vi (x , Y) (i = A to D) is transformed into coordinates and synthesized. Then, from the icon table 86, an icon I _Ae indicating that the installation height Z ₀ of the virtual camera 40v is low and the turning angle φ is in the right turning “small” state is selected and superimposed. At this time, the pile Q photographed by the front camera 40a is displayed as an image Q ′ in the virtual image I _VA (x, y), and the driver 8 operates the arm 24 while paying attention to the image Q ′. To do.

FIG. 10C shows an example of a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is “middle” turning right, the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. FIG. At this time, on the virtual image Iv (x, y), the installation height Z ₀ of the virtual camera 40v is low and the turning angle φ indicates a state of turning right so that the arm 24 faces upward. A coordinate transformation synthesis table T _Af is selected, and based on this coordinate transformation synthesis table T _Af , images taken by each camera (40a, 40b, 40c, 40d) (imaging unit) are respectively virtual images I _Vi (x , Y) (i = A to D) is transformed into coordinates and synthesized. Then, from the icon table 86, an icon I _Af indicating that the installation height Z ₀ of the virtual camera 40v is low and the turning angle φ is in the “right” turning state is selected and superimposed. At this time, the pile Q photographed by the front camera 40a is displayed as an image Q ′ in the virtual image I _VA (x, y), and the driver 8 operates the arm 24 while paying attention to the image Q ′. To do.

FIG. 10D shows an example of a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is “large” turning right, the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. FIG. At this time, on the virtual image Iv (x, y), the installation height Z ₀ of the virtual camera 40v is low, and the turning angle φ is the right turning “large” state so that the arm 24 faces upward. It is selected coordinate transformation synthesis table _{T Ag} showing an, on the basis of the coordinate conversion synthesis table _{T Ag,} each camera (40a, 40b, 40c, 40d) virtual image image captured by the imaging section, each _{I Vi} The coordinates are transformed into (x, y) (i = A to D) and synthesized. Then, from among the icons table 86, low height Z ₀ installation of the virtual camera 40v, turning angle φ icon I _Ag showing a right turn "large" state is superimposed is selected. At this time, the pile Q photographed by the front camera 40a is displayed as an image Q ′ in the virtual image I _VA (x, y), and the driver 8 operates the arm 24 while paying attention to the image Q ′. To do.

FIG. 10E shows a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is 0 (neutral position), the arm length L is medium, and the installation height Z ₀ of the virtual camera 40v is medium. It is a figure which shows an example. At this time, on the virtual image Iv (x, y), coordinate conversion is performed such that the installation height Z ₀ of the virtual camera 40v is medium and the turning angle φ indicates the neutral position so that the arm 24 faces upward. An image captured by each camera (40a, 40b, 40c, 40d) (imaging unit) is selected based on the coordinate conversion synthesis table T _Bd when the synthesis table T _Bd is selected, and a virtual image I _Vi (x, y) is obtained. ) (I = A to D) is converted into coordinates and synthesized. Then, from among the icons table 86, medium height Z ₀ installation of the virtual camera 40v, turning angle φ is superimposed is selected icons I _Bd indicating the neutral position. At this time, since the installation height Z ₀ of the virtual camera 40v is increased as compared with the generation condition of the virtual image Iv (x, y) illustrated in FIG. 10A, the virtual image Iv (x, y) in which a wider range is monitored. Is displayed. And the size of the icon I becomes small according to the expansion of the display range. In the virtual image Iv (x, y) generated in this way, a pile Q photographed by the front camera 40a is displayed as an image Q ′, and the driver 8 pays attention to this image Q ′. Then, the arm 24 is operated.

FIG. 10F shows an example of a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is 0 (neutral position), the arm length L is long, and the installation height Z ₀ of the virtual camera 40v is high. FIG. At this time, on the virtual image Iv (x, y), coordinate transformation synthesis is performed such that the installation height Z ₀ of the virtual camera 40v is high and the turning angle φ indicates the neutral position so that the arm 24 faces upward. The table T _Cd is selected, and based on the coordinate conversion synthesis table T _Cd , images taken by the cameras (40a, 40b, 40c, 40d) (imaging units) are respectively virtual images I _Vi (x, y). The coordinates are transformed into (i = A to D) and synthesized. Then, from the icon table 86, an icon I _Cd having a high installation height Z ₀ of the virtual camera 40v and a turning angle φ indicating a neutral position is selected and superimposed. At this time, since the installation height Z ₀ of the virtual camera 40v is increased as compared with the generation condition of the virtual image Iv (x, y) illustrated in FIG. 10E, the virtual image Iv (x, y) in which a wider range is monitored. Is displayed. And the size of the icon I becomes small according to the expansion of the display range. In the virtual image Iv (x, y) generated in this way, a pile Q photographed by the front camera 40a is displayed as an image Q ′, and the driver 8 pays attention to this image Q ′. Then, the arm 24 is operated.

(Description of the operation of the first embodiment)
Next, the specific operation of the first embodiment will be described with reference to the flowcharts of FIGS.

  FIG. 11 is a flowchart illustrating an overall flow of processing performed in the first embodiment. In the following, description will be given in order.

  (Step S10) Photographing is performed with each camera (40a, 40b, 40c, 40d) (imaging unit), and the photographed image is input.

  (Step S12) The captured image signal is AD converted by an AD converter (52a, 52b, 52c, 52d) to be converted into a digitized image signal.

  (Step S14) The digitized image signal is decoded by a decoder (54a, 54b, 54c, 54d) and converted into an RGB signal or a YCbCr, H-Sync, V-Sync signal which is a luminance color difference signal. .

  (Step S <b> 16) The heavy equipment attitude measurement unit 90 measures the turning angle φ, the arm length L, and the elevation angle ε of the arm 24 of the crane vehicle 12.

  (Step S20) In the virtual imaging unit installation unit 80, the sign (turning direction) and the size (turning amount) of the turning angle φ of the arm 24 are determined. Details of the processing will be described later.

  (Step S50) In the virtual imaging unit installation unit 80, the installation position of the virtual camera 40v (virtual imaging unit) is calculated. The details of the specific processing performed here will be described later.

  (Step S <b> 70) In the virtual image generation unit 100, coordinate transformation synthesis processing is performed. The details of the specific processing performed here will be described later.

  (Step S90) In the icon superimposition display part 88, selection and superimposition of the icon I are performed. The details of the specific processing performed here will be described later.

  (Step S100) In the heavy equipment posture measurement unit 90, the turning angle φ, the arm length L, and the elevation angle ε of the arm 24 of the crane vehicle 12 are measured, and whether there is a change in the posture of the crane vehicle 12 measured in step S16. Confirm whether or not. When there is no change in the posture of the crane vehicle 12, the process proceeds to step S102, and when there is a change in the posture of the crane vehicle 12, the process returns to step S10.

  (Step S102) The encoder 64 encodes the virtual image Iv (x, y) superimposed on the RGB signal or the YCbCr, H-Sync, V-Sync signal which is a luminance color difference signal with the icon I superimposed. , Convert to composite signal (digital signal).

  (Step S104) The DA converter 66 converts the composite signal (digital signal) into an analog signal.

  (Step S106) The converted analog signal is displayed on the monitor 78 (display unit 96), and the process ends. After that, the process may be repeated by returning to step S10.

  Next, the flow of the turning direction and turning amount determination process will be described with reference to FIG. This process is performed in the virtual imaging unit installation unit 80.

  (Step S22) It is determined whether or not the turning angle φ of the arm 24 is substantially 0 (neutral position). Specifically, when the turning angle φ measured by the gyro sensor 70 is −α <φ ≦ α, it is determined that the turning angle φ is 0 (neutral position). When it is determined that φ = 0, the process proceeds to step S46, and otherwise, the process proceeds to step S24.

  (Step S24) It is determined whether or not the turning angle φ of the arm 24 is positive (right turning). When φ is positive, the process proceeds to step S26, and otherwise, the process proceeds to step S36.

  (Step S26) It is determined whether or not the turning angle φ of the arm 24 satisfies α <φ ≦ β. If the condition is satisfied, the process proceeds to step S34, and otherwise, the process proceeds to step S28.

  (Step S28) It is determined whether or not the turning angle φ of the arm 24 satisfies β <φ ≦ γ. If the condition is satisfied, the process proceeds to step S30. Otherwise, the process proceeds to step S32.

  (Step S30) It is determined that the turning direction of the arm 24 is the right direction and the turning amount is “medium”, and the process returns to the main routine (FIG. 11).

  (Step S32) It is determined that the turning direction of the arm 24 is the right direction and the turning amount is “large”, and the process returns to the main routine (FIG. 11).

  (Step S34) It is determined that the turning direction of the arm 24 is the right direction and the turning amount is “small”, and the process returns to the main routine (FIG. 11).

  (Step S36) It is determined whether or not the turning angle φ of the arm 24 satisfies −α <φ ≦ −β. If the condition is satisfied, the process proceeds to step S44, and otherwise, the process proceeds to step S38.

  (Step S38) It is determined whether or not the turning angle φ of the arm 24 satisfies −β <φ ≦ −γ. If the condition is satisfied, the process proceeds to step S40, and otherwise, the process proceeds to step S42.

  (Step S40) It is determined that the turning direction of the arm 24 is the left direction and the turning amount is “medium”, and the process returns to the main routine (FIG. 11).

  (Step S42) It is determined that the turning direction of the arm 24 is the left direction and the turning amount is “large”, and the process returns to the main routine (FIG. 11).

  (Step S44) It is determined that the turning direction of the arm 24 is the left direction and the turning amount is “small”, and the process returns to the main routine (FIG. 11).

  (Step S46) It is determined that the arm 24 is not turned, and the process returns to the main routine (FIG. 11).

  Next, the flow of the installation position calculation process of the virtual camera 40v (virtual imaging unit) will be described with reference to FIG. This process is performed in the virtual imaging unit installation unit 80.

(Step S52) Using the arm length L of the crane vehicle 12 and the elevation angle ε of the arm 24, the installation height Z ₀ of the virtual camera 40v (virtual imaging unit) is calculated by the above-described (Expression 1).

(Step S54) is the height Z ₀ installation calculated, it is determined whether the threshold Zα below a preset. When Z ₀ ≦ Zα, the process proceeds to step S62, and otherwise, the process proceeds to step S56.

(Step S56) It is determined whether or not the calculated installation height Z ₀ satisfies Zα <Z ₀ ≦ Zβ with respect to preset threshold values Zα and Zβ (Zα <Zβ). If the condition is satisfied, the process proceeds to step S58. Otherwise, the process proceeds to step S60.

(Step S58) The installation height Z ₀ of the virtual camera 40v (virtual imaging unit) is determined to be “medium” and the process returns to the main routine (FIG. 11).

(Step S60) The installation height Z ₀ of the virtual camera 40v (virtual imaging unit) is determined to be “high” and the process returns to the main routine (FIG. 11).

(Step S62) The installation height Z ₀ of the virtual camera 40v (virtual imaging unit) is determined to be “low”, and the process returns to the main routine (FIG. 11).

  Next, the flow of the coordinate transformation synthesis process will be described with reference to FIG. This process is performed in the coordinate transformation synthesis unit 84.

(Step S72) the coordinate transformation Synthesis table _{(LUT) T Xy (X =} A~C, y = a~g) from the coordinate transformation synthesis corresponding to the height _{Z 0} installation of the virtual camera 40v (virtual imaging unit) Select a table.

(Step S74) From the coordinate transformation synthesis table T _Xy corresponding to the installation height Z ₀ of the virtual camera 40v (virtual imaging unit) selected in Step S72, the coordinate transformation synthesis table corresponding to the turning angle φ of the arm 24. (LUT) Select one T _Xy .

(Step S76) A coordinate transformation synthesis process is executed using the selected coordinate transformation synthesis table T _Xy to generate a virtual image Iv (x, y), and the process returns to the main routine (FIG. 11).

  Next, the flow of icon selection and superimposition processing will be described with reference to FIG. This process is performed in the icon superposition display unit 88.

Among the (step S92) icon table 86, selects an icon I corresponding to the height _{Z 0} installation of the virtual camera 40v (virtual imaging unit).

(Step S94) from the icon corresponding to the height Z ₀ installation of selected virtual camera 40v (virtual imaging unit) at step S92, selects one icon I corresponding to the turning angle φ of the arm 24.

  (Step S96) The selected icon I is superimposed on the virtual image Iv (x, y), and the process returns to the main routine (FIG. 11).

  (Description of Modified Example of Example 1)

Note that the form of the virtual image Iv (x, y) is not limited to the form shown in FIGS. 10A to 10F. That is, as shown in FIGS. 16A and 16B, images taken by the cameras (40a, 40b, 40c, 40d) (imaging units) are respectively displayed as virtual images I _Vi (x, y) (i = A to D). After the conversion, the virtual image Iv (x, y) may be generated by arranging the virtual image I _Vi (x, y) (i = A to D) in a circle around the icon I. 16A shows a virtual image Iv (x, y) generated when the turning angle φ of the arm 24 is 0 (neutral position), the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. FIG. 16B is a diagram illustrating an example, and FIG. 16B is a virtual image Iv generated when the turning angle φ of the arm 24 is “small” turning right, the arm length L is short, and the installation height Z ₀ of the virtual camera 40v is low. It is a figure which shows an example of (x, y).

Thus, when the virtual image Iv (x, y) is updated according to the turning of the arm 24 by synthesizing the virtual image I _Vi (x, y) (i = A to D) into a circular shape, Since the shape of the display area of the image photographed by each camera (40a, 40b, 40c, 40d) (imaging unit) does not change, it is possible to make the video easier to see.

  Next, Example 2 which is another embodiment of the present invention will be described with reference to the drawings.

(Description of System Configuration of Example 2)
In the second embodiment, when the crane vehicle 12 is in a state where the crane 12 can run on the heavy equipment periphery monitoring device 10 described in the first embodiment, the situation around the crane vehicle 12 is photographed. Is a heavy equipment periphery monitoring device 11 to which a function of converting and displaying a virtual image in a form corresponding to the traveling direction of the crane vehicle 12 is displayed.

  The functional block configuration of the heavy equipment periphery monitoring device 11 shown in the present embodiment will be described with reference to FIG. The heavy equipment periphery monitoring device 11 has a functional block configuration shown in FIG. The configuration is the same as the functional block configuration of the heavy equipment periphery monitoring device 10 described in the first embodiment, but further includes a travelable direction detection unit 98 that detects the travelable direction of the crane vehicle 12.

  The travelable direction detection unit 98 determines that the crane vehicle 12 is in a travelable state by detecting that the arm 24 of the crane vehicle 12 is locked and does not move. Further, by detecting the shift position of the lower traveling body 30 (see FIG. 1) of the crane vehicle 12, the traveling direction (forward or backward) of the crane vehicle is detected.

(Description of the function of the second embodiment)
Next, functions of the heavy equipment peripheral monitoring device 11 will be described with reference to FIGS. 18A and 18B. 18A shows an example of the virtual image Iv (x, y) displayed on the display unit 96 shown in FIG. 17 when the crane vehicle 12 is in the environment shown in FIG. FIG.

  At this time, since the crane vehicle 12 is in a state in which the crane vehicle 12 can move forward, the image in the direction in which the crane vehicle 12 passes is displayed more widely in the virtual image Iv (x, y). The driver 8 refers to the virtual image Iv (x, y) and moves the crane vehicle 12 forward.

An icon _IAd is superimposed on the virtual image Iv (x, y). For this icon _IAd , the icon _IAd shown in FIG. 9 may be used, or an icon representing a running state may be prepared and the icon may be superimposed.

  For the function of generating such a virtual image Iv (x, y), a coordinate conversion synthesis table to be used when the crane vehicle 12 is in a forward-movable state is prepared in the coordinate conversion synthesis table 82 shown in FIG. In addition, when the travelable direction detection unit 98 detects that the crane vehicle 12 is in a state where the crane vehicle 12 can move forward, referring to the coordinate conversion synthesis table in which the forward direction of the crane vehicle 12 is displayed more widely, This is realized by generating a virtual image Iv (x, y) in the coordinate transformation / synthesis unit 84.

In order to generate a virtual image Iv (x, y) in which the forward direction of the crane vehicle 12 is displayed more widely, the virtual camera 40v (virtual imaging unit) is, for example, in the coordinate system described with reference to FIGS. 4A and 4B. , A position (0, Y) installed at a preset installation height Z ₁ (<Z ₀ ) (not shown) and moved by a predetermined amount Y ₁ (not shown) in the forward direction of the crane vehicle 12. ₁ , Z ₁ ) installed vertically downward.

  FIG. 18B shows an example of the virtual image Iv (x, y) displayed on the display unit 96 shown in FIG. 17 when the crane vehicle 12 is in the environment shown in FIG. 4A and is in a retractable state. FIG.

  At this time, since the crane vehicle 12 is in a retractable state, the image in the direction in which the crane vehicle 12 passes is displayed more widely in the virtual image Iv (x, y). The driver 8 refers to the virtual image Iv (x, y) and moves the crane vehicle 12 backward.

An icon _IAd is superimposed on the virtual image Iv (x, y). For this icon _IAd , the icon _IAd shown in FIG. 9 may be diverted, or a new icon representing a running state may be prepared and the icon may be superimposed.

  The function for generating such a virtual image Iv (x, y) is prepared in the coordinate conversion synthesis table 82 shown in FIG. 17 with a coordinate conversion synthesis table used when the crane vehicle 12 is in a retractable state. In addition, when the travelable direction detection unit 98 detects that the crane vehicle 12 is in a retractable state, refer to the coordinate conversion synthesis table in which the reverse direction of the crane vehicle 12 is displayed more widely. This is realized by generating a virtual image Iv (x, y) in the coordinate transformation / synthesis unit 84.

In order to generate a virtual image Iv (x, y) in which the backward direction of the crane vehicle 12 is displayed more widely, the virtual camera 40v (virtual imaging unit) is, for example, in the coordinate system described with reference to FIGS. 4A and 4B. , A position (0, −) that is installed at a preset installation height Z ₁ (<Z ₀ ) (not shown) and is moved by a predetermined amount Y ₁ (not shown) in the backward direction of the crane vehicle 12. Y ₁ , Z ₁ ) are installed vertically downward.

(Description of the operation of the second embodiment)
Next, the specific operation of the second embodiment will be described using the flowchart of FIG.

  (Step S120) Photographing is performed by each camera (40a, 40b, 40c, 40d) (imaging unit), and the photographed image is input.

  (Step S122) The captured image signal is AD-converted by an AD converter (52a, 52b, 52c, 52d) and converted into a digitized image signal.

  (Step S124) The digitized image signal is decoded by a decoder (54a, 54b, 54c, 54d) and converted into an RGB signal or a YCbCr, H-Sync, V-Sync signal which is a luminance color difference signal. .

  (Step S126) In the travelable direction detection unit 98, it is determined whether or not the crane vehicle 12 is in a state in which the crane vehicle 12 can move forward. When advance is possible, it progresses to step S130, and when that is not right, it progresses to step S128.

  (Step S128) In the travelable direction detection unit 98, it is determined whether or not the crane vehicle 12 is in a state in which it can move backward. When it is possible to move backward, the process proceeds to step S130, and otherwise, the process proceeds to step S136.

  (Step S130) In the virtual imaging unit installation unit 80, the installation position of the virtual camera 40v (virtual imaging unit) is calculated. Specifically, as described above, the virtual camera 40v (virtual imaging unit) is installed at a position corresponding to the traveling direction of the crane vehicle 12.

  (Step S132) In the virtual image generation unit 100, a coordinate transformation synthesis process is performed. The specific processing content is as described in the first embodiment.

  (Step S134) The icon superimposing display unit 88 selects and superimposes the icon I. Specifically, the icon I to be superimposed when the crane vehicle 12 is in the traveling state is selected and superimposed on the virtual image Iv (x, y).

  (Step S136) The heavy equipment attitude measurement unit 90 measures the turning angle φ, the arm length L, and the elevation angle ε of the arm 24 of the crane vehicle 12.

  (Step S138) The virtual imaging unit installation unit 80 determines the magnitude and direction of the turning angle φ of the arm 24. The specific processing content is as described in the first embodiment.

  (Step S140) In the virtual imaging unit installation unit 80, the installation position of the virtual camera 40v (virtual imaging unit) is calculated. The specific processing content is as described in the first embodiment.

  (Step S142) The virtual image generation unit 100 performs a coordinate transformation synthesis process. The specific processing content is as described in the first embodiment.

  (Step S144) The icon superimposition display unit 88 selects and superimposes the icon I. The specific processing content is as described in the first embodiment.

  (Step S146) In the heavy equipment posture measuring unit 90, the turning angle φ, the arm length L, and the elevation angle ε of the arm 24 of the crane vehicle 12 are measured, and whether there is a change in the posture of the crane vehicle 12 measured in step S136. Confirm whether or not. When there is no change in the posture of the crane vehicle 12, the process proceeds to step S148, and when there is a change in the posture of the crane vehicle 12, the process returns to step S120.

  (Step S148) The encoder 64 encodes the virtual image Iv (x, y) carried on the RGB signal or the YCbCr, H-Sync, V-Sync signal which is the luminance color difference signal with the icon I superimposed. , Convert to composite signal (digital signal).

  (Step S150) The DA converter 66 converts the composite signal (digital signal) into an analog signal.

  (Step S152) The converted analog signal is displayed on the monitor 78 (display unit 96), and the process ends. After that, the process may be repeated by returning to step S120.

  As described above, according to the heavy equipment periphery monitoring device 10 according to the first embodiment of the present invention configured as described above, a plurality of cameras attached around the lower traveling body 30 of the crane vehicle 12 (heavy equipment). (40a, 40b, 40c, 40d) A crane in which the virtual image generation unit 100 measures a plurality of images including the ground around the crane vehicle 12 taken by the imaging unit, and is measured by the heavy equipment attitude measurement unit 90. Observed from a virtual camera 40v (virtual imaging unit) installed at a position and direction corresponding to the turning angle φ of the arm 24 of the upper turning body 20 of the vehicle 12, the arm length L, and the elevation angle ε of the arm 24. The icon superimposed display unit 88 displays an icon I corresponding to the attitude of the crane vehicle 12 on the virtual image Iv (x, y). Superimposed on the crane Since the driver 8 of the vehicle 12 displays the information on the monitor 78 (display unit 96), the driver 8 of the crane vehicle 12 is in the vicinity of the crane vehicle 12 regardless of the change in the posture of the heavy machinery being worked on. The position and direction of the obstacle can be recognized immediately.

  Further, according to the heavy equipment periphery monitoring device 10 according to the first embodiment of the present invention configured as described above, the virtual camera 40v (virtual imaging unit) is at least from the position of the crane vehicle 12 (heavy equipment). Because the virtual line Iv (x, y) including the range up to the point where the extension line of the line connecting the installation position of 40v and the tip C of the arm 24 intersects the ground is installed at a position and direction where the virtual image Iv (x, y) can be generated. The range including the position immediately below the tip C of the arm 24 that requires special attention when working with the crane vehicle 12 can be imaged.

  Moreover, according to the heavy equipment periphery monitoring apparatus 10 according to the first embodiment of the present invention configured as described above, the virtual image generation unit 100 includes the arm length L of the arm 24, the elevation angle ε of the arm 24, and the upper part. It has a coordinate conversion synthesis table 82 corresponding to the installation position and installation direction of the virtual camera 40v (virtual imaging unit) corresponding to the turning angle φ of the arm 24 of the turning body 20, and was measured by the heavy equipment attitude measurement unit 90. Since the virtual image Iv (x, y) is generated based on the arm length L, the elevation angle ε of the arm 24, and the coordinate transformation synthesis table 82 corresponding to the turning angle φ of the arm 24, the generation of the virtual image is reduced. Can be done quickly in quantity.

  Further, according to the heavy equipment periphery monitoring device 10 according to the first embodiment of the present invention configured as described above, the virtual image generation unit 100 includes the virtual image Iv (x, x, Since y) is generated, a virtual image in the same direction as the direction in which the driver 8 of the crane vehicle 12 is looking can be generated and displayed. Accordingly, since the driver 8 can easily associate the virtual image Iv (x, y) with the actual situation, the driver 8 can immediately recognize the position and direction of the obstacle around the crane vehicle 12. .

  Further, according to the heavy equipment periphery monitoring apparatus 10 according to the first embodiment of the present invention configured as described above, the virtual image generation unit 100 uses the virtual camera 40v (virtual imaging unit) as the turning center of the upper swing body 20. Is taken downward on a vertical line passing through and is taken when rotated around the optical axis of the virtual camera 40v by an angle corresponding to the turning angle φ of the arm 24 measured by the heavy equipment attitude measurement unit 90. Therefore, the virtual image Iv (x, y) can be generated in a short time with a small amount of calculation.

  In addition, according to the heavy equipment periphery monitoring device 10 according to the first embodiment of the present invention configured as described above, the icon superimposed display unit 88 includes the icon table 86 in which a plurality of icons I are stored, and the heavy equipment To select an icon I corresponding to the attitude of the crane vehicle 12 (heavy equipment) measured by the attitude measuring unit 90 from the icon table 86 and to superimpose the selected icon I on the virtual image Iv (x, y). The icon I can be quickly superimposed with a small amount of calculation.

  Further, according to the heavy equipment periphery monitoring device 11 according to the second embodiment of the present invention configured as described above, the heavy equipment periphery monitoring device 11 includes a travelable direction detection unit 98 that detects the travelable direction of the crane vehicle 12 (heavy equipment), When the travelable direction detection unit 98 detects the travelable direction of the crane vehicle 12, the virtual image generation unit 100 can observe the travel direction of the crane vehicle 12 more widely with the virtual camera 40v (virtual imaging unit). Since it is installed in the position, height, and direction, the traveling direction of the crane vehicle 12 can be displayed more clearly.

In the first and second embodiments, the installation height Z ₀ of the virtual camera 40v (virtual imaging unit) is set in three stages according to the arm length L and the elevation angle ε of the arm 24, and the arm 24 is set. The turning angle φ of the virtual image Iv (x, y) is calculated by performing calculation for calculating the installation position and the installation direction of the virtual camera 40v each time. May be changed continuously.

In the first and second embodiments, when determining the installation height Z ₀ of the virtual camera 40v (virtual imaging unit), as shown in FIG. 4A, the point A that is the farthest point on the ground and the arm The position where the straight line passing through the tip C of 24 intersects the Z axis is set as the installation position V of the virtual camera 40v (virtual imaging unit), but the determination method of the installation position V of the virtual camera 40v (virtual imaging unit) is as follows. However, the present invention is not limited to this, and may be determined based on other criteria.

Further, in the first and second embodiments, the images Ii (x, y) (i = A to D) (i = A to D) photographed by the cameras (40a, 40b, 40c, 40d) (imaging unit) are all flat ground ( The virtual image I _Vi (x, y) (i = A to D) is generated on the assumption that a plane having a height of 0 is reflected, but this may be performed based on another rule. For example, in FIGS. 4A and 4B, a spherical surface in contact with the ground (origin O) immediately below the virtual camera 40v is set around the position of the virtual camera 40v (virtual imaging unit) installed directly below. By projecting an image Ii (x, y) (i = A to D) captured by the cameras (40a, 40b, 40c, 40d) onto this spherical surface, a virtual image I _Vi (x, y) (i = A to D) may be generated. By using such coordinate transformation, it is possible to obtain a virtual image I _Vi (x, y) (i = A to D) in which distortion of an object having a height from the ground is reduced.

  In the first and second embodiments, analog signals output from the cameras (40a, 40b, 40c, 40d) (imaging units) are converted into digital signals, and after necessary conversion / combination processing is performed, the analog signals are output again. The signal is converted into a signal and displayed on the monitor 78. However, the present invention is not limited to this structure, and a system configuration using a camera capable of digital output is adopted, and the digital signal output from the camera is subjected to image processing as it is. You may take the structure to do.

  Further, in the first and second embodiments, the result measured by the heavy equipment attitude measurement unit 90 using the CAN communication line 76 is communicated to the overall control unit 94, but this is not limited to the CAN communication line 76. Alternatively, another communication line such as a general serial communication line may be used.

  As mentioned above, although the Example of this invention was explained in full detail with drawing, since an Example is only an illustration of this invention, this invention is not limited only to the structure of an Example. Of course, changes in design and the like within a range not departing from the gist are included in the present invention.

10. Perimeter monitoring device 40a for heavy machinery Front camera (imaging unit)
40b Left side camera (imaging part)
40c Rear camera (imaging part)
40d Right side camera (imaging part)
76 CAN communication line 80 Virtual imaging unit installation unit 82 Coordinate transformation synthesis table 84 Coordinate transformation synthesis unit 86 Icon table 88 Icon superposition display unit 90 Heavy machine posture measurement unit 91 Turning angle measurement unit 92 Arm length measurement unit 93 Arm elevation angle measurement unit 94 Overall Control unit 96 Display unit 100 Virtual image generation unit

Claims (10)

A heavy machine comprising an arm that can change the length and elevation angle, and is composed of an upper swing body that can be swung right and left by a driver and a lower travel body that can travel and move around the lower travel body. An imaging unit that is mounted in a plurality and captures a plurality of images including the ground around the heavy machinery;
A plurality of images captured by the image capturing unit are converted and combined, respectively, and predicted to be observed from a virtual image capturing unit virtually installed at a predetermined position above the heavy machinery in a predetermined direction. A virtual image generation unit that generates one virtual image to be generated;
In the peripheral monitoring device for heavy equipment, which is installed on the upper swing body so that the driver can visually recognize the display unit on which the virtual image is displayed.
The heavy machine posture measuring unit that measures the length of the arm, the elevation angle of the arm, and the turning angle of the arm, respectively, and the heavy machine that is predicted to be observed from the virtual imaging unit on the virtual image posture simulating, a, and icons superimposed display unit that superimposes the size of the icon corresponding to the length of the arm, the virtual image generation unit, the heavy machinery to the heavy orientation measurement unit is measured A virtual image that is predicted to be observed from a virtual imaging unit installed in a direction according to the posture is generated at a position according to the posture, and the icon superimposed display unit corresponds to the posture of the heavy equipment an icon in the virtual image, and superposition so that the tip of the arm is positioned on the edge of the virtual image, a peripheral for heavy equipment virtual image which the icon is superimposed and displaying on the display unit Monitoring device.   The virtual imaging unit generates a virtual image including at least a range from a position of the heavy machine to a point where an extension line of a line connecting the installation position of the virtual imaging unit and the tip of the arm intersects the ground. The peripheral monitoring device for heavy machinery according to claim 1, wherein the peripheral monitoring device is installed in a possible position and direction. A heavy machine comprising an arm that can change the length and elevation angle, and is composed of an upper swing body that can be swung right and left by a driver and a lower travel body that can travel and move around the lower travel body. An imaging unit that is mounted in a plurality and captures a plurality of images including the ground around the heavy machinery;
A plurality of images captured by the image capturing unit are converted and combined, respectively, and predicted to be observed from a virtual image capturing unit virtually installed at a predetermined position above the heavy machinery in a predetermined direction. A virtual image generation unit that generates one virtual image to be generated;
In the peripheral monitoring device for heavy equipment, which is installed on the upper swing body so that the driver can visually recognize the display unit on which the virtual image is displayed.
The heavy machine posture measuring unit that measures the length of the arm, the elevation angle of the arm, and the turning angle of the arm, respectively, and the heavy machine that is predicted to be observed from the virtual imaging unit on the virtual image An icon superimposing display unit that superimposes an icon that simulates the posture of the virtual machine, and the virtual image generation unit is in a position corresponding to the posture of the heavy machine measured by the heavy machine posture measurement unit. A virtual image predicted to be observed from a virtual imaging unit installed in a direction is generated, and the icon superimposed display unit superimposes an icon corresponding to the posture of the heavy machine on the virtual image, and the icon There displays the virtual image superimposed on said display unit,
The virtual imaging unit generates a virtual image including at least a range from a position of the heavy machine to a point where an extension line of a line connecting the installation position of the virtual imaging unit and the tip of the arm intersects the ground. Perimeter monitoring device for heavy machinery, characterized in that it is installed in a possible position and direction . A travelable direction detection unit that detects a travelable direction of the heavy machine, and the virtual image generation unit detects the virtual imaging unit when the travelable direction detection unit detects the travelable direction of the heavy machine. The peripheral monitoring device for heavy machinery according to any one of claims 1 to 3 , wherein the heavy machinery is installed at a position, height, and direction in which the traveling direction of the heavy machinery can be observed more widely. A heavy machine comprising an arm that can change the length and elevation angle, and is composed of an upper swing body that can be swung right and left by a driver and a lower travel body that can travel and move around the lower travel body. An imaging unit that is mounted in a plurality and captures a plurality of images including the ground around the heavy machinery;
A plurality of images captured by the image capturing unit are converted and combined, respectively, and predicted to be observed from a virtual image capturing unit virtually installed at a predetermined position above the heavy machinery in a predetermined direction. A virtual image generation unit that generates one virtual image to be generated;
In the peripheral monitoring device for heavy equipment, which is installed on the upper swing body so that the driver can visually recognize the display unit on which the virtual image is displayed.
The heavy machine posture measuring unit that measures the length of the arm, the elevation angle of the arm, and the turning angle of the arm, respectively, and the heavy machine that is predicted to be observed from the virtual imaging unit on the virtual image An icon superimposing display unit that superimposes an icon that simulates the posture of the virtual machine, and the virtual image generation unit is in a position corresponding to the posture of the heavy machine measured by the heavy machine posture measurement unit. A virtual image predicted to be observed from a virtual imaging unit installed in a direction is generated, and the icon superimposed display unit superimposes an icon corresponding to the posture of the heavy machine on the virtual image, and the icon There displays the virtual image superimposed on said display unit,
A travelable direction detection unit that detects a travelable direction of the heavy machine, and the virtual image generation unit detects the virtual imaging unit when the travelable direction detection unit detects the travelable direction of the heavy machine. , heavy surroundings monitoring apparatus characterized that you set up the travelable direction of the heavy equipment to wider observable position and height and orientation. The virtual imaging unit generates a virtual image including at least a range from a position of the heavy machine to a point where an extension line of a line connecting the installation position of the virtual imaging unit and the tip of the arm intersects the ground. The peripheral monitoring device for heavy machinery according to claim 5, wherein the peripheral monitoring device is installed in a possible position and direction. The virtual image generation unit has a coordinate conversion synthesis table corresponding to the installation position and installation direction of the virtual imaging unit corresponding to the length of the arm, the elevation angle of the arm, and the turning angle of the arm, respectively. The virtual image is generated based on the coordinate transformation synthesis table corresponding to the length of the arm measured by the heavy equipment attitude measurement unit, the elevation angle of the arm, and the turning angle of the arm. The heavy equipment periphery monitoring apparatus according to any one of claims 1 to 6 . The heavy equipment periphery monitoring device according to any one of claims 1 to 7, wherein the virtual image generation unit generates a virtual image in which an extending direction of the arm is always located at an upper part. The virtual image generation unit installs the virtual imaging unit downward on a vertical line passing through the turning center of the upper revolving body, and according to the turning angle of the arm measured by the heavy equipment posture measuring unit. The heavy equipment periphery monitoring device according to claim 8 , wherein a virtual image predicted to be captured when the virtual imaging unit is rotated around the optical axis of the virtual imaging unit by an angle is generated. The icon superimposed display unit has an icon table in which a plurality of the icons are stored, and selects an icon corresponding to the posture of the heavy machinery measured by the heavy machinery posture measuring unit from the icon table, The peripheral monitoring device for heavy machinery according to any one of claims 1 to 9 , wherein the selected icon is superimposed on the virtual image.