JP2020147962A - Ambient display system and work machines equipped with the same - Google Patents

Abstract

To provide an ambient display system capable of stably obtaining a certain level of accuracy or higher in a determination related to calibration and a work machine equipped with the system.SOLUTION: A controller 12 that generates a composite image 18 that gives a bird's-eye view of a vehicle body and its surroundings from above the vehicle body from images captured by a plurality of cameras 11a to 11d that capture the surroundings of the vehicle body of a hydraulic excavator 1, in a plurality of images captured by a plurality of cameras 11a to 11d, identifies the positions of calibration markers 20 having a predetermined shape for use in the calibration process, respectively, calculates the coordinates of the feature points of the calibration marker 20 in the vehicle body coordinate system for each of the plurality of images, calculates the coordinate difference of the feature points of the calibration marker 20 in the plurality of images, determines the necessity of the calibration process based on the coordinate difference, and outputs the determination result to a display device 17.SELECTED DRAWING: Figure 6

Description

The present invention relates to a surrounding display system and a work machine including the same.

In recent years, the digitization and computerization of work machines have progressed, and many electronic devices have been installed. For example, in a work machine such as a hydraulic excavator, the surroundings of the vehicle body are photographed by a plurality of cameras, and the image is displayed on the display unit in the cab so that the worker in the cab can view the image while sitting in the driver's seat. There are things that can be confirmed and the situation around the car body can be known. In particular, in a work machine such as a hydraulic excavator, the visibility of the worker in the cab with respect to the surroundings of the vehicle body is likely to be restricted due to structural reasons, so that the operator in the cab can see the situation around the vehicle body from the image on the display. Technology that can be accurately grasped is important.

As a conventional technique for assisting the field of view of a worker in a cab in such a work machine, for example, in Patent Document 1, a plurality of cameras mounted on a vehicle body photograph the surroundings of the vehicle body, and the captured images are combined. A technology is disclosed in which a bird's-eye view of the vehicle body and its surroundings is generated from the sky above the vehicle body (hereinafter referred to as a bird's-eye view image for convenience) and displayed on a display unit in the cab.

Japanese Unexamined Patent Publication No. 2014-125856

By the way, when the camera is newly installed or reattached, or when the camera is found to be misaligned, the camera mounting position is such that the bird's-eye view image synthesized from multiple images does not lap or shift. It is necessary to perform work (that is, calibration) to correct the error and the mounting angle error. As a calibration method, for example, after installing a marker (marker) in a predetermined range of the shooting range of each camera, each camera (mark) uses the marker in the image shot by each camera as a reference point. There is a method of calculating a correction value for correcting laps and deviations between (each image), and there is a method of correcting laps and deviations by applying the correction value to a bird's-eye view image generation process.

On the other hand, when an operator, a serviceman, or the like determines the necessity of calibration or the pass / fail of the calibration result, the accuracy of the determination differs depending on the person, so the accuracy of the determination related to the calibration is not always constant. Therefore, it is possible that the required accuracy cannot be obtained.

The present invention has been made in view of the above, and an object of the present invention is to provide an ambient display system capable of stably obtaining a certain level of accuracy or higher in a determination related to calibration, and a work machine equipped with the system.

The present application includes a plurality of means for solving the above-mentioned problems. For example, a plurality of cameras for photographing the surroundings of a vehicle body and a composite image synthesized from images photographed by the plurality of cameras are displayed. In a work machine including a display device to be displayed and a controller that generates the composite image based on the composition conditions and outputs the composite image to the display device, the controller calibrates a plurality of images captured by the plurality of cameras. The positions of the calibration markers having a predetermined shape to be used for the processing are specified, the coordinates of the feature points of the calibration markers in the vehicle body coordinate system are calculated for each of the plurality of images, and the features of the calibration markers are calculated. It is assumed that the coordinate difference of the points in the plurality of images is calculated, the necessity of the calibration process is determined based on the coordinate difference, and the determination result is output to the display device.

According to the present invention, it is possible to stably obtain a certain level of accuracy or higher in the determination related to calibration.

It is a side view which shows typically the structure of the hydraulic excavator which is an example of the work machine which concerns on one Embodiment of this invention. It is a top view which shows typically the structure of the hydraulic excavator which is an example of the work machine which concerns on one Embodiment of this invention. It is a figure which shows typically the surrounding display system of a work machine. It is a figure which shows an example of the relationship between the photographing range of a camera, and the calibration marker arranged around a vehicle body. It is a functional block diagram which shows by extracting the processing function which concerns on the calibration determination of a controller. It is a flowchart which shows the processing content of the calibration determination. It is a figure which shows an example of the difference of the coordinates of the calibration marker. It is a figure which shows the display example in the display apparatus of the determination result of the calibration determination, and is the figure which shows the display example when it is determined that it has passed. It is a figure which shows the display example in the display apparatus of the determination result of the calibration determination, and is the figure which shows the display example when it is determined that it has failed. It is a figure which magnifies and exemplifies a part of the bird's-eye view image displayed on the display device in a state where the calibration marker is arranged around the vehicle body. It is a figure which magnifies and exemplifies a part of the bird's-eye view image displayed on the display device in a state where the calibration marker is arranged around the vehicle body.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a hydraulic excavator will be described as an example of a work machine, but the present invention can also be applied to a work machine such as a dump truck.

1 and 2 are views schematically showing a configuration of a hydraulic excavator which is an example of a work machine according to the present embodiment, and FIG. 1 is a side view and FIG. 2 is a top view.

In FIG. 1, the hydraulic excavator 1 rotatably supports an upper swing body 3, an articulated front work machine 4 supported on the front side of the upper swing body 3 so as to be able to move up and down, and an upper swing body 3. It is composed of a self-propelled crawler type lower traveling body 2. The upper swing body 3 and the lower traveling body 2 constitute the vehicle body of the hydraulic excavator 1.

The upper swivel body 3 has a swivel frame 5 forming a support structure, and a base end portion of the front working machine 4 is rotatably supported on the front side of the swivel frame 5. Further, a counterweight 10 for balancing the weight with the front working machine 4 is attached to the rear side of the swivel frame 5. A cab 6 for an operator operating the hydraulic excavator 1 to board is arranged on the front left side of the swivel frame 5.

In the cab 6, there is a seat on which the operator sits, an operation device for driving the front working machine 4, turning the upper swing body 3, running the lower traveling body 2, and the like (none of which are shown). The display device 17 is arranged at a position that is easy for the operator seated in the seat to see and does not interfere with the external view of the operator.

Further, on the front side of the counter weight 10, a building cover 7 for accommodating on-board equipment (none of which is shown) such as an engine, a hydraulic pump, and a heat exchange device is provided inside. The building cover 7 is roughly composed of left and right side plates 7A located on both the left and right sides of the swivel frame 5 and extending in the front-rear direction, and a horizontally extending top plate 7B connecting the upper ends of the side plates 7A. The rear side of the building cover 7 is closed by the counter weight 10. On the upper surface plate 7B of the building cover 7, an engine cover 8 is provided which covers the mounted devices housed in the building cover 7 from above.

The front upper part of the cab 6 on the front side of the upper turning body 3, the left and right upper parts of the upper turning body 3 (that is, the left and right ends of the upper surface plate 7B of the building cover 7), near the center in the front-rear direction, and the counter weight. Cameras 11a to 11d arranged so that the surroundings of the vehicle body including the vicinity of the vehicle body are the shooting range are installed at the left and right center positions of 10.

In FIGS. 1 and 2, a state in which the calibration marker 20 is arranged behind the cab 6 is illustrated. The calibration marker 20 is used for the calibration process of the surrounding display system in the present embodiment, and for example, a predetermined shape drawn on a sheet-shaped member (hereinafter, the shape includes a shape and a size). It is a figure having a proofreading). That is, when the calibration marker 20 is photographed by cameras 11a to 11d having known performance, the calibration marker 20 is based on the shape of the calibration marker 20 and the predetermined shape of the calibration marker 20 in the image. And the cameras 11a to 11d can be obtained. This will be described in detail later.

FIG. 3 is a diagram schematically showing a peripheral display system of a work machine according to the present embodiment.

In FIG. 3, the surrounding display system is a display such as a plurality of (for example, four in the present embodiment) cameras 11a to 11d and a monitor (liquid crystal monitor) for displaying images taken by the plurality of cameras 11a to 11d. It includes a device 17, a controller 12 that controls the overall operation of the surrounding display system, and a personal computer 13 that cooperates with the controller 12 to perform various controls.

The display device 17 may include images taken by the cameras 11a to 11d, images whose viewpoints are converted so that these images are taken vertically above (that is, upward viewpoints), or images taken by the cameras 11a to 11a. A screen 15 that displays a bird's-eye view image 18 (a composite image obtained by simulating a bird's-eye view of the vehicle body and its surroundings from above the vehicle body) generated by synthesizing the images 18a to 18d shot at 11d. Is provided.

In the example shown in FIG. 3, a case where the bird's-eye view image 18 is displayed on the screen 15 of the display device 17 is shown. In the bird's-eye view image 18, for example, the own vehicle icon 16 that roughly shows the shape and size of the own vehicle in the bird's-eye view image 18 is arranged in the center, and the cameras 11a, 11c, 11b, and 11d are used vertically and horizontally on the own vehicle icon 16. It is formed by converting the captured image into images 18a, 18c, 18b, and 18d from the upper viewpoint so that the vehicle is the center, and arranging them.

The present invention is applied to the work machine configured as described above, and the controller 12 of the peripheral display system of the work machine according to the present embodiment is a bird's-eye view of displaying on the display device 17 from the images of the cameras 11a to 11d. In addition to the image generation function that generates images and the calibration function that adjusts the composition conditions used for synthesizing the bird's-eye view image, the calibration judgment function that determines the necessity of adjusting the composition conditions by the calibration function and the pass / fail of the adjustment result is provided. Have.

The image generation function is a function for generating a bird's-eye view image or the like. For example, images taken by cameras 11a to 11d are acquired and synthesized based on information on mounting positions and mounting angles of a plurality of cameras 11a to 11d with respect to the vehicle body. This is a function of specifying a composition condition for generating an image, synthesizing a composite image (overhead image 18) from the images taken by the cameras 11a to 11d based on the specified composition condition, and outputting the composite image (overhead image 18) to the display device 17.

FIG. 4 is a diagram showing an example of the relationship between the shooting range of the camera and the calibration markers arranged around the vehicle body. FIG. 4 illustrates a case where the calibration marker 20 is arranged in an overlapping range 111cd of the shooting range 111c of the rear camera 11c and the shooting range 111d of the camera 11d on the left side.

The calibration function is a function for adjusting the composition conditions used for synthesizing the bird's-eye view image. For example, as shown in FIG. 4, the image obtained by shooting the calibration marker 20 in the overlapping range 111cd of the adjacent cameras 11a to 11d is captured. Each of them is acquired, the positional relationship between the adjacent cameras 11a to 11d is specified from the positional relationship between each of the cameras 11a to 11d and the calibration marker 20, and the video composition position is specified as a composition condition based on the result.

FIG. 5 is a functional block diagram showing the processing functions related to the calibration determination of the controller. Further, FIG. 6 is a flowchart showing the processing contents of the calibration determination.

In FIG. 5, the controller 12 has image acquisition unit 121, image marker position identification unit 122, marker coordinate calculation unit 123, marker coordinate difference calculation unit 124, calibration determination unit 125, and a calibration determination unit 125 as processing function units related to calibration determination. , Has a marker shape storage unit 126.

In FIG. 6, when the controller 12 is instructed to start the calibration determination by the operation of the personal computer 13 or another operating device, the image acquisition unit 121 first acquires the images from the cameras 11a to 11d (step S100). ). In step S100, as shown in FIG. 4, an image is acquired in a state where the calibration marker 20 is arranged in the overlapping shooting range of the adjacent cameras 11a to 11d (for example, the range 111cd for the cameras 11c and 11d). To do. The method of acquiring the image in step S100 is not limited, but for example, the calibration markers 20 are set in each of the overlapping shooting ranges (there are four in the present embodiment) of the cameras 11a to 11d. When performing in the arranged state, when performing the image individually while moving the calibration marker 20 to the overlapping shooting range, or when the calibration marker 20 is arranged at a fixed position, the upper swivel body 3 is swiveled. It is conceivable that an image is acquired when the calibration marker 20 is included in the overlapping shooting ranges after driving.

Subsequently, in the image marker position specifying unit 122, for each of the images acquired in step S100, the vehicle body coordinate system of the calibration marker 20 in the image is based on the shape of the calibration marker stored in the marker shape storage unit 126. The position (coordinates) in the above is specified (step S110). In the vehicle body coordinate system, for example, as shown in FIGS. 1 and 2, the z-axis whose upper side is positive along the turning axis of the upper swivel body 3 is positive in the front perpendicular to the z-axis. It is assumed that the x-axis is set to be perpendicular to the z-axis and the y-axis with the left side as positive.

The marker shape storage unit 126 stores the shape (shape and size) of the calibration marker 20 used for the calibration process. The shape of the calibration marker 20 stored in the marker shape storage unit 126 is preset by a setting function of the marker shape setting unit 130 of the personal computer 13. The setting function of the marker shape setting unit 130 is, for example, to set the calibration marker 20 to be selectively used by the operator for the calibration process from the candidates of a plurality of calibration markers. The controller 12 may have a setting function of the marker shape setting unit 130.

The image marker position specifying unit 122 first specifies the position (coordinates) of the calibration marker 20 in the image in the camera coordinate system, and the installation information (attachment direction in the vehicle body coordinate system) of each camera 11a to 11d stored in advance. The position (coordinates) of the calibration marker 20 in the vehicle body coordinate system is calculated based on (or angle, etc.). The method of specifying the position of the calibration marker 20 is not particularly limited, and it is sufficient that the position of the calibration marker 20 in the image can be specified by using, for example, a method such as pattern matching.

Subsequently, the marker coordinate difference calculation unit 124 calculates the difference between the positions (coordinates) of the calibration markers 20 captured in the overlapping imaging ranges of the adjacent cameras 11a to 11d between the cameras 11a to 11d (step). S120).

FIG. 7 is a diagram showing an example of the difference in the coordinates of the calibration marker.

In FIG. 7, the position of the calibration marker 20c in the vehicle body coordinate system in the image captured in the imaging range 111c of the camera 11c and the calibration marker 20d in the vehicle body coordinate system in the image captured in the imaging range 111d of the camera 11d. The position is illustrated. The calculation of the coordinate difference in the marker coordinate difference calculation unit 124 is performed by subtracting the smaller one from the larger one on each coordinate axis for the coordinates of the installation positions of the cameras 11a to 11d in each of the x coordinate and the y coordinate (). Difference calculation conditions). That is, for example, when considering the difference between the feature point of the calibration marker 20c, for example, the feature point C (x1, y1) and the feature point of the calibration marker 20d, for example, the feature point D (x2, y2), the difference (X2-x1, y2-y1) is calculated. The same applies to the calculation of the coordinate difference of the calibration marker 20 between the images acquired by the other cameras 11a to 11d.

Subsequently, the calibration determination unit 125 determines whether or not the calibration process is necessary by determining whether or not the coordinate difference of each axis is equal to or greater than a predetermined threshold value based on the coordinate difference calculated in step S120. (Step S130), if the determination result is YES, that is, if it is determined that the result is passed, a message indicating that the result is passed is output to the display device 17 as the determination result (step S140), and the process is terminated. .. Further, when the determination result in step S130 is NO, that is, when it is determined that the calibration is rejected, the calibration determination unit 125 outputs a message indicating the failure to the display device 17 as the determination result. (Step S150), the process is terminated.

8 and 9 are diagrams showing a display example in the display device of the determination result of the calibration determination, FIG. 8 is a display example when it is determined to be acceptable, and FIG. 9 is determined to be unacceptable. It is a figure which shows each display example at the time of. When the calibration determination unit 125 determines that the calibration determination has passed, as shown in FIG. 8, the calibration determination unit 125 outputs a message indicating that the calibration has passed to the display device 17 as a determination result and displays the result. When it is determined that the result is acceptable, as shown in FIG. 9, a message indicating the failure and a message prompting the execution of recalibration are output to the display device 17 as the determination result and displayed. ..

The method of calibration determination in the calibration determination unit 125 is not limited to the method of determining whether or not the coordinate difference of each axis is equal to or greater than a predetermined threshold value as shown in step S130.

Another example of the calibration determination method will be described below.

10 and 11 are enlarged views of a part of the bird's-eye view image displayed on the display device in a state where the calibration marker is arranged around the vehicle body.

In the bird's-eye view image as shown in FIGS. 10 and 11, the boundary line (for example, the boundary line 18bc, 18cd, etc.) with the images of the adjacent cameras 11a to 11d is referred to as a mask line, and the mask line 18bc, 18cd is referred to. The vertical direction is defined as the lap direction, and the direction along the mask lines 18bc and 18cd is defined as the deviation direction.

In the state shown in FIG. 10, the entire calibration marker 20 is reflected in either the image 18c or the image 18d at the mask line 18cd portion, and in the bird's-eye view image 18, any of the images taken by the adjacent cameras 11c and 11d. It can be said that there is no range (in other words, a blind spot) that does not appear. On the other hand, the state shown in FIG. 11 is a case where a part of the calibration marker 20 is not reflected in either the image 18c or the image 18d at the part of the mask line 18cd, and the cameras 11c and 11d adjacent to each other in the bird's-eye view image 18 It can be said that there is a range (in other words, a blind spot) that does not appear in any of the captured images. Then, in the present invention, it is determined that the state in which there is at least no blind spot in the bird's-eye view image 18 has passed the calibration determination.

Specifically, in the calibration determination by the calibration determination unit 125, the coordinates in the direction (lap direction) perpendicular to the boundaries (mask lines 18bc, 18cd) of the boundary portions of a plurality of images in the composite image (overhead image 18). The difference (that is, the distance between the feature points in the lap direction and satisfying the above-mentioned difference calculation condition) is compared with a predetermined threshold value, and if it is equal to or more than the threshold value, it passes, and if it is smaller than the threshold value, it passes. Determines to be unacceptable. In this case, by setting the threshold value to 0 (zero), the calibration judgment is passed when there is no blind spot at the boundary lines 18bc and 18cd. The threshold value does not have to be 0 (zero), and may be a positive value within a range in consideration of ease of monitoring the surroundings of the vehicle body in the composite image 18 by the operator, for example.

Further, in the present embodiment, the case where the lower limit is set as the threshold value has been described as an example, but by setting the threshold value within the range that defines the upper limit and the lower limit, the bird's-eye view image that is determined to be necessary or not in the calibration determination The accuracy can be adjusted arbitrarily, and the accuracy above a certain level can be stably obtained.

The effects of the present embodiment configured as described above will be described.

In the prior art, when calibrating the mounting position and mounting angle of the camera, if an operator, a serviceman, or the like determines the necessity of calibration or the pass / fail of the calibration result, the determination accuracy is improved. Since it differs from person to person, the accuracy of the determination related to calibration is not always constant, and it is conceivable that the required accuracy cannot be obtained.

On the other hand, in the present embodiment, the plurality of cameras 11a to 11d for photographing the surroundings of the vehicle body of the hydraulic excavator 1 and the plurality of cameras 11a so as to give a bird's-eye view of the vehicle body and the surrounding conditions from above the vehicle body. In an ambient display system including a display device 17 that displays a composite image 18 synthesized from the images captured by ~ 11d, and a controller 12 that generates the composite image 18 based on the composition conditions and outputs the composite image 18 to the display device 17. , The controller 12 specifies the position of the calibration marker 20 having a predetermined shape to be used for the calibration process in the plurality of images taken from 11a to 11d by the plurality of cameras, and the feature points of the calibration marker 20. The coordinates in the vehicle body coordinate system are calculated for each of the plurality of images, the coordinate difference of the feature points of the calibration marker 20 in the plurality of images is calculated, and the necessity of the calibration process is determined based on the coordinate difference. Is configured to be output to the display device 17, so that it is possible to stably obtain a certain level of accuracy or higher in the determination related to calibration.

Next, the features of each of the above embodiments will be described.

(1) In the above embodiment, a plurality of cameras 11a to 11d for photographing the surroundings of the vehicle body (for example, the upper swing body 3 and the lower traveling body 2) of the work machine (for example, the hydraulic excavator 1), and the vehicle body. A display device 17 that displays a composite image 18 synthesized from images taken by the plurality of cameras and the composite image are generated based on the composition conditions so as to show a bird's-eye view of the vehicle body and the surrounding conditions from the sky. In an ambient display system including a controller 12 that outputs to the display device, the controller 20 is a calibration marker 20 having a predetermined shape for use in calibration processing in a plurality of images captured by the plurality of cameras. Each of the positions is specified, the coordinates of the feature points of the calibration marker in the vehicle body coordinate system are calculated for each of the plurality of images, and the coordinate difference of the feature points of the calibration marker in the plurality of images is calculated. The necessity of the calibration process is determined based on the coordinate difference, and the determination result is output to the display device.

As a result, it is possible to stably obtain a certain level of accuracy or higher in the determination related to calibration.

(2) Further, in the above embodiment, in the surrounding display system of (1), when the controller 12 has a coordinate difference between the plurality of images of the feature points of the calibration marker 20 or more than a predetermined threshold value. In addition, it is determined that the calibration process has passed.

(3) Further, in the above embodiment, in the surrounding display system of (1), the controller has a predetermined threshold value in which the coordinate difference in the direction perpendicular to the boundary of the boundary portion of the plurality of images in the composite image is predetermined. In the above cases, it is determined that the calibration process has passed.

(4) Further, in the above-described embodiment, the vehicle body on which the plurality of cameras are installed and the surrounding display system of (1) are provided.

<Additional notes>
The present invention is not limited to the above-described embodiment, and includes various modifications and combinations within a range that does not deviate from the gist thereof. Further, the present invention is not limited to the one including all the configurations described in the above-described embodiment, and includes the one in which a part of the configurations is deleted. Further, each of the above configurations, functions and the like may be realized by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function.

1 ... hydraulic excavator, 2 ... lower traveling body, 3 ... upper swivel body, 4 ... front work machine, 5 ... swivel frame, 6 ... cab, 7 ... building cover, 7A ... side plate, 7B ... top plate, 8 ... engine Cover, 10 ... counterweight, 11a to 11d ... camera, 12 ... controller, 13 ... personal computer, 15 ... screen, 16 ... own car icon, 17 ... display device, 18 ... composite image (overhead image), 18a to 18d ... image , 18bc, 18cd ... Boundary line (mask line), 20, 20c, 20d ... Calibration marker, 111c, 111d ... Shooting range, 111dc ... Range, 121 ... Video acquisition unit, 122 ... Video marker position identification unit, 123 ... Marker Coordinate calculation unit, 124 ... Marker coordinate difference calculation unit, 125 ... Calibration judgment unit, 126 ... Marker shape storage unit, 130 ... Marker shape setting unit

Claims (4)

Multiple cameras that take pictures of the surroundings of the body of the work machine,
A display device that displays a composite image synthesized from images taken by the plurality of cameras so as to give a bird's-eye view of the vehicle body and its surroundings from above the vehicle body.
In an ambient display system including a controller that generates the composite video based on the composite conditions and outputs it to the display device.
The controller specifies the position of a calibration marker having a predetermined shape to be used for the calibration process in a plurality of images captured by the plurality of cameras, and the feature points of the calibration marker are in the vehicle body coordinate system. The coordinates are calculated for each of the plurality of images, the coordinate difference between the feature points of the calibration marker in the plurality of images is calculated, and the necessity of the calibration process is determined based on the coordinate difference, and the determination result is obtained. A peripheral display system characterized by outputting the above to the display device. In the surrounding display system according to claim 1,
The controller is an ambient display system, characterized in that, when the coordinate difference between the plurality of images of the feature points of the calibration marker is equal to or greater than a predetermined threshold value, it is determined that the calibration process has passed. In the surrounding display system according to claim 1,
The controller is characterized in that it determines that the calibration process has passed when the coordinate difference in the direction perpendicular to the boundary of the boundary portion of the plurality of images in the composite image is equal to or greater than a predetermined threshold value. Surrounding display system. The vehicle body on which the multiple cameras are installed and
A work machine including the surrounding display system according to claim 1.

Images