JP2011151742A - Image display system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image display system in which a wide work range is displayed without a sense of incompatibility. <P>SOLUTION: The present invention relates to an image display system S, and the system is equipped with: a turn table 12 which is installed freely turnable relative to a car body 10; a boom 14 installed on the turn table 12 so as to freely rise and fall; a distal tip camera 21 mounted in the vicinity of a distal end of the boom 14; peripheral cameras 22-25 mounted on the car body 10 or the turn table 12; and a processing section 3 for displaying on a display section 6 an image of the distal end camera 21 and images of the peripheral cameras 22-25. The processing section 3 then synthesizes the image photographed by the distal end camera 21 and the images photographed by the circumferential cameras 22-25 and displays a resulting synthesized image on the display section 6. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image display system such as a crane or an aerial work platform.

  Conventionally, a system has been proposed in which a camera is mounted on a vehicle such as an automobile, and safety confirmation is supported using a captured image. In addition, a system has also been proposed in which a camera is mounted on a construction machine, a crane or the like to check the surrounding safety.

  For example, Patent Document 1 includes a camera system that includes a camera unit that captures the periphery of a revolving unit and an image processing device, and displays an arc indicating an outer peripheral position of a movable range by the revolving of the revolving unit on a display screen. It is disclosed.

  According to this configuration, the positional relationship between the turning body and the obstacle can be confirmed on the display screen, and the surrounding situation to be watched can be easily confirmed.

JP 2008-312004 A

  However, in the case of a large crane or the like, it was not possible to capture and display the entire work range with only a camera attached to a vehicle or a swivel.

  Therefore, an object of the present invention is to provide an image display system that can display a wide working range without a sense of incongruity.

  In order to achieve the above object, an image display system according to the present invention is attached to a swivel base that can be swung with respect to a vehicle body, a boom that can be swung up and down on the swivel base, and a front end of the boom. A front-end camera, a peripheral camera attached to the vehicle body or the swivel, and a processing unit that displays an image of the front-end camera and an image of the peripheral camera on a display unit, The processing unit synthesizes an image photographed by the front-end camera and an image photographed by the peripheral camera and displays them on the display unit.

  Moreover, the said process part can be set as the structure which converts the image image | photographed with the said front end camera, and the image image | photographed with the said peripheral camera into the bird's-eye view image from a reference position.

  Furthermore, when the processing unit combines the image captured by the front camera and the image captured by the peripheral camera, the processing unit includes: an image captured by the front camera; and an image captured by the peripheral camera. The overlapping portion may be configured to display with priority the image taken by the front-end camera.

  And the said process part can be set as the structure which correct | amends the image image | photographed with the said peripheral camera based on the feature point of the image image | photographed with the said front end camera.

  Moreover, the said process part can be set as the structure which overlaps and displays the line from which the load factor with respect to the rated load of the actual load which acts on the said boom becomes a predetermined load factor on the said bird's-eye view image.

  Further, the processing unit has a display area to be displayed on the display unit at a front side of the vehicle body up to a range obtained by adding a predetermined addition to a load factor of 100%, and on the side of the vehicle body, an outrigger maximum overhang width or tail Up to a range obtained by adding a predetermined additional amount to the swing width can be displayed.

  As described above, the image display system of the present invention is an image display system including the swivel base, the boom, the tip camera, the peripheral camera, and the processing unit, and the processing unit includes the image captured by the tip camera and the peripheral camera. The captured image is combined and displayed on the display unit.

  Therefore, since a wide work range can be displayed without a sense of incongruity, it is possible to work safely. In addition, the number of blind spots is reduced, and an object located in the sky can be displayed.

  In addition, the processing unit can grasp the positional relationship with the surrounding objects in a plane by converting the image taken with the front camera and the image taken with the peripheral camera into an overhead view image from the reference position, It becomes easy to grasp the distance to surrounding objects.

  Furthermore, when the processing unit combines the image captured by the front camera and the image captured by the peripheral camera, the overlapping portion of the image captured by the front camera and the image captured by the peripheral camera is By preferentially displaying images taken with the camera, a clearer display is possible and an image display system with less image processing load is provided.

  Then, the processing unit corrects the image captured by the peripheral camera based on the feature point of the image captured by the front-end camera, thereby converting the image to a bird's eye view more accurately while suppressing the burden of image conversion.

  In addition, the processing unit displays a line in which the load factor with respect to the rated load of the actual load acting on the boom is a predetermined load factor superimposed on the overhead view image, so that it is possible to prevent danger in advance by viewing the load factor line while working. Be able to detect and work safely.

  Further, the processing unit displays a display area to be displayed on a monitor as a display unit, up to a range obtained by adding a predetermined addition to a load factor of 100% in the front of the vehicle body, By displaying up to a range obtained by adding a predetermined additional amount to the tail swing width, it is possible to display all necessary areas for safety.

It is a block diagram explaining the structure of the image display system of this invention. It is a side view explaining the structure and camera position of a rough terrain crane. It is a top view explaining the structure and camera position of a rough terrain crane. It is explanatory drawing explaining the concept of a bird's-eye view conversion process. It is explanatory drawing explaining the display process of a load factor line. (A) is a case based on a vehicle, and (b) is a case based on a swivel base. It is explanatory drawing explaining the range of a display area. It is explanatory drawing explaining a mask area | region.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the mechanical configuration of the rough terrain crane 1 as a crane provided with the crane control device S of the present embodiment will be described with reference to FIG. In the following embodiments, the rough terrain crane 1 will be described as an example. However, the present invention is not limited to this, and the present invention is not limited to this, as long as it is a crane having a boom such as an all terrain crane, a truck crane, and a loading truck crane. Applicable.

  As shown in FIG. 2, the rough terrain crane 1 according to the present embodiment includes a carrier 10 serving as a main body portion of a vehicle having a traveling function, outriggers 11 provided at four corners of the carrier 10, and a carrier 10. A swivel base 12 is mounted so as to be horizontally swivelable, and a boom 14 is mounted on a bracket 13 erected on the swivel base 12.

  The outriggers 11 prevent the rough terrain crane 1 from overturning, and are provided on the left and right sides of the carrier 10 on the front side and the rear side, and are extended and grounded by hydraulic pressure.

  The swivel base 12 has a pinion gear to which the power of the turning motor is transmitted. The pinion gear meshes with a circular gear provided on the carrier 10 to rotate around the swivel axis.

  Further, the boom 14 is configured to be nested by a proximal boom 141, an intermediate boom 142, and a distal boom 143, and can be expanded and contracted by an expansion cylinder (not shown).

  Among these, the base end boom 141 is swingably attached to a support shaft installed horizontally on the bracket 13 at the base thereof, and can be raised and lowered up and down.

  Further, a hoisting cylinder 15 is stretched between the bracket 13 and the base end boom 141, and the entire boom 14 can be hoisted by extending and retracting the hoisting cylinder 15.

  Further, a wire 16 is wound around the sheave of the tip boom 143 and a hook 17 is suspended.

  As shown in FIGS. 2 and 3, the image display system S according to the present embodiment includes, in addition to the swivel base 12 and the boom 14, a tip camera 21 attached downward to the tip of the tip boom 143, A front camera 22, a rear camera 23, a left camera 24, a right camera 25 as peripheral cameras mounted sideways on the table 12, and a monitor serving as a display unit with an image of the front camera 21 and an image of the peripheral camera 6 is displayed.

  The front camera 21 is a digital movie camera with a standard lens that captures the lower side from the same high position as the front end of the front boom 143, and the captured image is transmitted to the processing unit 3.

  Further, since the front camera 21 is attached to the side surface of the front end of the front boom 143 so as to be swingable in the up-and-down direction, the front camera 21 always captures a vertically lower part.

  For this reason, the shooting range of the tip camera 21 changes depending on the undulation and height of the boom 14, and also changes depending on the turning position of the swivel base 12 on which the boom 14 is mounted.

  Similarly, the front camera 22, the rear camera 23, the left camera 24, and the right camera 25, which are peripheral cameras, shoot front, rear, left, and right images from a low position where the swivel base 12 is installed. This is a wide-angle lens digital movie camera, and the captured image is transmitted to the processing unit 3.

  Therefore, the front camera 22, rear camera 23, left camera 24, and right camera 25, which are peripheral cameras, do not change the shooting range depending on the undulation or height of the boom 14, and depends on the turning position of the turntable 12. The shooting range changes.

  The processing unit 3 is a microcomputer having a storage unit and a calculation unit. After correcting the distortion of the transmitted image and converting it to a bird's-eye view image, each of the bird's-eye view images is panorama synthesized and then displayed on the display unit. It is displayed on a certain monitor 6. Hereinafter, the processing contents will be described in the order of distortion correction processing, overhead conversion processing, panorama synthesis processing, correction processing based on feature points, load factor line display processing, and display area extraction processing.

(Distortion correction processing)
In this distortion correction processing, in order to correct distortion due to the lens, the processing unit 3 multiplies the coordinate value of the input pixel by a coefficient calculated based on the lens distortion coefficient, the aspect ratio, etc., and converts it to the coordinate value of the output pixel. To do.

(Overhead conversion process)
Further, in the overhead view conversion process for converting to a bird's eye view image, the processing unit 3 converts the viewpoint to a predetermined position vertically above the turning center that is the reference position, and converts the lens distortion coefficient or camera mounting angle into the coordinate value of the input pixel. Multiply by the coefficient calculated based on the above and convert it to the coordinate value of the output pixel.

  Specifically, as shown in Equations 1 and 2 and FIG. 4, a camera transformation position is measured to obtain a projection transformation matrix, or four or more feature points on the image before and after transformation are determined to obtain a projection transformation matrix. The coordinate value after conversion is calculated by multiplying the coordinate value before conversion by the projective conversion matrix. If this is described as a vector,

It becomes. Moreover, when describing a matrix,

It becomes. As described above, this projective transformation matrix can be obtained from the camera mounting position and the feature points (four or more points) on the images before and after the transformation.

  At this time, in order to obtain the mounting position of the front camera 21, the posture information of the boom 14 is required. Therefore, the processing unit 3 acquires the measurement values obtained by the undulation angle measurement unit 51, the length measurement unit 52, and the turning angle measurement unit 53 as posture measurement units from the control unit 4 and calculates the position of the front camera 21.

(Panorama composition processing)
Furthermore, in the panorama synthesis process, the processing unit 3 realizes a joint having no sense of incongruity by linearly interpolating the brightness of the corresponding coordinate values in order to connect adjacent captured images as one image.

  In this composition, when the image of the front camera 21 and the images of the front camera 22, rear camera 23, left camera 24, and right camera 25, which are peripheral cameras, have overlapping portions, the front camera 21 is used for the overlapping portions. Are preferentially displayed on the monitor 6 as the main image.

  That is, depending on the position of the front-end camera 21, a blind spot is generated around the vehicle only with the image of the front-end camera 21, so that the image of the peripheral camera is used to interpolate this blind spot and increase the resolution of the blurred portion.

  On the other hand, for example, when the tip camera 21 is far away from the vehicle with the boom 14 fully extended and undulating, or when the tip camera 21 is zoomed to obtain only a local image. The image of the peripheral camera is displayed on the monitor 6 as the main image.

(Correction processing based on feature points)
In the image display system S of the present embodiment, the processing unit 3 executes correction of the image of the peripheral camera, display correction of the three-dimensional object, and the like based on the feature points reflected in the image of the front camera 21 with less distortion. .

  Specifically, feature points are extracted from the image of the tip camera 21. As the feature point, for example, a corner of an object can be used. Next, a point corresponding to the feature point is searched from the other image, and a geometric transformation parameter between images is calculated from a set of coordinates of the associated feature point and converted (corrected). For the geometric transformation, affine transformation or projective transformation is used.

  The three-dimensional object is likely to have a discontinuous shift at the joint portion in the composite image of the peripheral camera. Therefore, the three-dimensional object is determined from the images of the front camera 21 and the peripheral camera, and the three-dimensional object of the image of the front camera 21 is displayed.

(Load factor line display processing)
Further, in the image display system S of the present embodiment, as shown in FIG. 5A, the processing unit 3 superimposes the actual load acting on the boom 14 on the overhead image created as described above. A line on which the load factor with respect to the rated load is a predetermined load factor is displayed on the monitor 6.

  Specifically, an arc-shaped line with a load factor of 100% where the actual load is equal to the rated load and an arc-shaped line with a load factor of 90% with a margin of 10% of the actual load with respect to the rated load are the monitors 6. Is displayed. Further, when the overhang width of the outrigger 11 is different on the left and right, a curve such as a combination of arcs having different diameters is displayed in the monitor 6.

  In FIG. 5A, the front direction of the carrier 10 is displayed so as to match the upper direction of the monitor 6 screen regardless of the turning position of the turntable 12, but as shown in FIG. 5B. Alternatively, the direction of the boom 14 that rotates together with the swivel base 12 may be displayed in the upper direction of the monitor 6 screen with the direction of the axis of the boom 14 as the forward direction.

(Cut out display area)
Finally, the display area of the image subjected to the distortion correction, the overhead view conversion, and the panorama synthesis is cut out in the processing unit 3 according to the shape of the monitor 6 as shown in FIG.

  This display area may be a range obtained by adding a predetermined additional amount to the load factor of 100% in front of the vehicle body, and a range obtained by adding a predetermined additional amount to the outrigger maximum overhang width or tail swing width on the side of the vehicle body. it can.

  In other words, the front of the carrier 10 and the front of the swivel base 12 include a vertex of a line with a load factor of 100%, and a line with a load factor (for example, 120%) with a predetermined additional portion added to make the whole easier to see. Include vertices.

  Further, on the side of the carrier 10 and on the rear side of the carrier 10, a predetermined additional portion (for example, 3 m) is added to a range obtained by adding a predetermined additional portion (for example, 3 m) to the maximum extension width of the outrigger 11 or the tail swing width of the swivel base 12. ) Is included.

  On the other hand, when the image range of the front-end camera 21 is narrower than the display area in front of the vehicle body, the processing unit 3 displays a black or gray mask on an area where no photographed image exists, as shown in FIG. The area is displayed on the monitor 6.

  Next, the operation of the image display system S of the present embodiment will be described.

  (1) As described above, the image display system S according to the present embodiment includes a swivel base 12 that can be swiveled with respect to the carrier 10 that is a vehicle body, a boom 14 that can be swung up and down on the swivel base 12, The front camera 21 attached in the vicinity of the tip of the boom 14, the front camera 22, the rear camera 23, the left camera 24 and the right camera 25 as peripheral cameras attached to the carrier 10 or the swivel base 12, and the images of the front camera 21 And a processing unit 3 that displays a peripheral camera image on a monitor 6 serving as a display unit. The processing unit 3 captures an image captured by the front-end camera 21 and a peripheral camera. It is characterized in that the image is synthesized and displayed on the monitor 6.

  Therefore, since a wide work range can be displayed without a sense of incongruity, it is possible to work safely. In addition, the number of blind spots is reduced, and an object located in the sky can be displayed.

  That is, in a large crane or the like, the work range extends not only around the carrier 10 but also around the suspended load at a position away from the carrier 10, so it is difficult to capture the entire work range using only the peripheral camera. is there.

  In this case, even if the peripheral camera is installed slightly upward and the vicinity of the suspended load is photographed, the image becomes blurred during the overhead view conversion process, resulting in an uncomfortable feeling.

  Therefore, by combining the image photographed by the front-end camera 21 and the image photographed by the peripheral camera and displaying them on the monitor 6, both the periphery of the carrier 10 and the periphery of the suspended load can be seen at a time. Work safety is improved.

  In particular, a large crane or the like has many blind spots around the carrier 10 that cannot be photographed by a peripheral camera. However, the blind spot can be reduced by using the tip camera 21 together.

  Furthermore, since the front-end camera 21 is attached in the vicinity of the front end of the boom 14, it is possible to take an image of an object in the sky that cannot be taken by a peripheral camera installed at a low position.

  Then, if the image of the front camera 21 and the image of the peripheral camera are combined and displayed in one monitor 6 in this way, the positional relationship with other objects can be grasped in advance. Safety is improved.

  (2) The processing unit 3 also uses an image captured by the front-end camera 21 and an image captured by the front camera 22, the rear camera 23, the left camera 24, and the right camera 25, which are peripheral cameras, from the reference position. By converting to an overhead view image, it is possible to grasp the positional relationship with surrounding objects in a planar manner, so that it is easy to grasp the distance to the surrounding objects.

  In addition, when the load factor line is superimposed on the composite image and displayed, if the composite image is a bird's-eye view image, the load factor line is displayed in an arc shape, so that the display is easy to understand.

  (3) Further, the processing unit 3 combines the image captured by the front camera 21 and the image captured by the front camera 22, the rear camera 23, the left camera 24, and the right camera 25 that are peripheral cameras. In this case, the overlapping portion of the image captured by the front camera 21 and the image captured by the peripheral camera can be displayed more clearly by displaying the image captured by the front camera 21 with priority. The image display system S has a low processing load.

  That is, when performing overhead conversion processing of an image, an image captured by a peripheral camera equipped with a wide-angle lens is unclear because the position changes greatly before and after conversion, but an image captured by the front-end camera 21 is much less. Since the position does not change, a clear image is obtained.

  Therefore, the clearer image captured by the front-end camera 21 is preferentially displayed, so that the entire composite image becomes clear. In addition, if the images of the peripheral cameras in the overlapped portion are not subjected to overhead conversion, the overall processing load is increased. Can be reduced.

  (4) Then, the processing unit 3 is photographed by the front camera 22, the rear camera 23, the left camera 24, and the right camera 25, which are peripheral cameras, based on the feature points of the image photographed by the front camera 21. By correcting the image, it is possible to convert the overhead image more accurately while suppressing the burden of image conversion.

  That is, since the front camera 22, the rear camera 23, the left camera 24, and the right camera 25 as the peripheral cameras are wide-angle lenses, image distortion is large and it is difficult to smoothly synthesize them.

  In this case, although the lens characteristics can be obtained and corrected in advance, correction in the height direction is particularly difficult, and a three-dimensional object extending between adjacent images cannot be accurately combined.

  Therefore, if the image of the peripheral camera is corrected based on the feature point of the image of the front camera 21 with a small image distortion, the image can be smoothly synthesized.

  (5) In addition, the processing unit 3 displays the line where the load factor with respect to the rated load of the actual load acting on the boom 14 is a predetermined load factor so as to be superimposed on the overhead image, so that the load factor line can be seen while working. Now you can detect danger in advance and work safely.

  (6) Furthermore, the processing unit 3 has a display area to be displayed on the monitor 6 as a display unit. By displaying up to a range obtained by adding a predetermined additional amount to the maximum overhang width or tail swing width, it is possible to display all necessary areas for safety.

  The embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes that do not depart from the gist of the present invention are not limited to the present invention. included.

  For example, in the above-described embodiment, the processing unit 3 and the control unit 4 are described as being configured separately. However, the present invention is not limited to this, and the processing unit 3 is configured integrally with the control unit 4. Each process may be executed by the control unit 4.

  Further, in the above-described embodiment, the case where the image of the front camera 21 and the image of the peripheral camera overlap with each other has been described as being automatically prioritized and displayed, but the present invention is not limited to this. Instead of this, an image switching means for manually switching a priority image may be provided.

  Furthermore, in the said Example, although the front-end camera 21 was demonstrated as what is installed in the side surface of the front-end | tip boom 143, it is not limited to this, In the case of an aerial work vehicle, it should be installed in the bucket side surface. Can do.

  In the above embodiment, the front camera 22, the rear camera 23, the left camera 24, and the right camera 25, which are peripheral cameras, have been described as being arranged on the turntable 12. However, the present invention is not limited to this. The peripheral camera may be disposed on the carrier 10.

  In the above embodiment, the case where the front camera 21 is installed at the front end of the front boom 143 has been described. However, the present invention is not limited to this, and when the jib is attached, the front camera is placed near the front end of the jib. May be installed.

  Furthermore, in the above-described embodiment, the case where the load factor line is displayed superimposed on the synthesized overhead image has been described, but the present invention is not limited to this, and the work radius may be displayed instead of the load factor, The target position and the lifting performance of the suspended load can be displayed superimposed on the overhead image.

  The image display system S of the present invention can also be used to monitor the intrusion around the vehicle and work area, monitor the collision between the boom 14 and the swivel base 12 and surrounding objects, and check the outrigger 11 and the jib overhang area. Available.

S image display system 1 rough terrain crane 10 carrier 12 swivel base 14 boom 21 tip camera 22 front camera (peripheral camera)
23 Rear camera (peripheral camera)
24 Left camera (peripheral camera)
25 Right camera (peripheral camera)
3 Processing Unit 4 Control Unit 6 Monitor (Display Unit)

Claims (6)

A swivel that can be swiveled with respect to the vehicle body, a boom that can be swung up and down on the swivel, a tip camera that is attached in the vicinity of the tip of the boom, and a peripheral camera that is attached to the car body or the swivel And an image display system comprising: a processing unit that displays an image of the front-end camera and an image of the peripheral camera on a display unit,
The image processing system, wherein the processing unit synthesizes an image photographed by the front-end camera and an image photographed by the peripheral camera and displays them on the display unit.   The image display system according to claim 1, wherein the processing unit converts an image photographed by the front-end camera and an image photographed by the peripheral camera into an overhead view image from a reference position.   When the processing unit combines the image captured by the front camera and the image captured by the peripheral camera, an overlapping portion of the image captured by the front camera and the image captured by the peripheral camera The image display system according to claim 1, wherein an image captured by the tip camera is preferentially displayed.   The said process part correct | amends the image image | photographed with the said peripheral camera based on the feature point of the image image | photographed with the said front-end | tip camera. Image display system.   5. The display unit according to claim 2, wherein the processing unit displays a line in which a load factor with respect to a rated load of an actual load acting on the boom is a predetermined load factor superimposed on the overhead image. The image display system according to item. The processing unit is a display area to be displayed on the display unit.
On the front side of the vehicle body, display up to a range obtained by adding a predetermined additional amount to 100% of the load factor, and on the side of the vehicle body, display up to a range obtained by adding a predetermined additional amount to the outrigger maximum overhang width or tail swing width. The image display system according to any one of claims 1 to 5, wherein: