JP2020142903A - Image processing apparatus and control program - Google Patents

Abstract

To provide a driver with information useful for operation of a forklift.SOLUTION: An image processing apparatus 20 comprises a display unit 27 attached to a position where the display unit 27 is visually recognizable from a driver seat of a forklift, photographing units 21a, 21b photographing the front of a car body of the forklift, an object detecting unit 31 detecting an object from a picture obtained by photographing of the photographing units, a speed acquiring unit 23 acquiring movement speed information of the car body, a fork state acquiring unit 24 acquiring state information regarding a posture and a position in a height direction, of the fork supported in a freely ascendable and descendable manner at the front side of the car body, a cargo detecting unit 22 detecting presence or absence of a cargo on the fork or a size of the cargo, and an image processing unit 32 determining working condition of the forklift according to state information of the fork acquired from the fork state acquiring unit 24 and detection results of the cargo detecting unit 22, and generating a display image according to the determined working conditions and causing the display unit 27 to display the generated image.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus and a control program.

The forklift moves by placing the luggage on the pallet on the fork. For example, in a seat-type forklift in which the driver faces the direction of travel and sits in the driver's seat, a blind spot is created in the front when the fork is loaded higher than the driver's line of sight. When traveling, the driver moves while moving the forklift backward. However, at the time of cargo handling, it is necessary to move forward, and the driver has no choice but to lean out to the side and visually recognize.

In addition, there is a demand for accurate and easy positioning of the fork or the pallet insertion port on the fork even in the loading / unloading work on a high-altitude shelf where it is difficult for the driver to see the front of the fork. In the image processing device for a forklift disclosed in Patent Document 1 to solve such a problem, a camera for photographing the front is provided on the fork, and the photographed image is reduced to represent a fork or a part of a palette on the fork. The image is superimposed and displayed on the display.

JP-A-2007-84162

In the image processing apparatus disclosed in Patent Document 1, the output of the pallet sensor on the fork is used to determine whether the work is a loading operation or a loading operation, and a reduced image is displayed according to the determination result.

However, in cargo handling work, not only the loading and unloading work of loading and unloading luggage on the fork of the forklift, but also the empty forklift is moved to the luggage, and the forklift is moved after loading the luggage. Work is also included. In addition, the information required by the driver differs depending on the size of the luggage and the surrounding environment such as the place where the luggage is placed.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a driver with information useful for operating a forklift.

The above object of the present invention is achieved by the following means.

(1) A display unit mounted at a position visible from the driver's seat of the forklift
A photographing unit attached to the forklift and photographing the front of the vehicle body of the forklift,
An object detection unit that detects an object from the image obtained by shooting the imaging unit,
A speed acquisition unit that acquires the moving speed information of the vehicle body, and
A fork state acquisition unit that acquires state information regarding the posture of the fork that is vertically supported on the front side of the vehicle body and the position in the height direction, and
A luggage detection unit that detects the presence or absence of luggage on the fork or the size of the luggage,
The forklift according to the object detected by the object detection unit, the movement speed information acquired from the speed acquisition unit, the state information of the fork acquired from the fork state acquisition unit, and the detection result of the luggage detection unit. An image processing unit that determines the work status of the above, generates a display image according to the determined work status, and displays the display image on the display unit.
An image processing device comprising.

(2) The image processing unit determines whether or not the object is a warning target based on the type of the object detected by the object detection unit.
The image processing device according to (1) above, wherein the image processing unit includes information about an object to be warned in the display image.

(3) The work situation includes a running state, a loading state, and a loading state.
The image processing unit
When the speed of the moving speed information exceeds the predetermined value, or when the speed is equal to or less than the predetermined value and the inclination of the fork with respect to the ground is equal to or more than the predetermined value in the state information, the traveling state is determined.
If there is no load on the fork when it is not in the running state, it is determined that the load is in the loading state.
The image processing device according to the above (1) or (2), wherein if there is a load on the fork when the vehicle is not in the running state, the load is determined to be in the loading state.

(4) In the running state,
The image processing unit determines the height of the luggage and the height of the fork when the type of the object to be warned includes a luggage and detects the luggage.
The image processing apparatus according to (3) above, wherein the display image includes an image according to a determination result.

(5) When the cargo is in the pick-up state,
The image processing unit determines the height of the fork and determines the height of the fork.
The image processing apparatus according to (3) or (4) above, wherein the display image includes an image according to the determination result.

(6) In the case of the loading state,
The image processing unit determines the presence or absence of an obstacle in front of the vehicle body, and if there is an obstacle, the position of the obstacle and the width of the luggage.
The image processing apparatus according to any one of (3) to (5) above, wherein the display image includes an image according to a determination result.

(7) The image processing apparatus according to any one of (1) to (6) above, wherein the photographing unit includes a camera attached to the tip end portion of one of the forks.

(8) The display unit is any of the above (1) to (7), which is a head-up display that projects a virtual image onto a combiner attached to the front side of the vehicle body, or a transmissive liquid crystal display. The image processing device described.

(9) An image processing device used for a forklift, which includes a display unit attached to a position visible from the driver's seat of the forklift and a photographing unit attached to the forklift to photograph the front of the vehicle body. A control program that runs on the controlling computer
In step (a) of acquiring an image by the photographing unit,
In step (b) of detecting an object from the image acquired in step (a),
In step (c) of acquiring the moving speed information of the vehicle body,
Step (d) of acquiring state information regarding the posture of the fork supported so as to be able to move up and down on the front side of the vehicle body and the position in the height direction, and
In step (e) of detecting the presence or absence of luggage on the fork or the size of the luggage,
According to the object detected in the step (b), the moving speed information obtained in the steps (c) and (d), the state information of the fork, and the detection result of the step (e). A step (f) of determining the work status of the forklift, generating a display image according to the determined work status, and displaying the display on the display unit.
A control program for executing processing including.

(10) The work situation includes a running state, a loading state, and a loading state.
In the step (f), when the speed of the moving speed information exceeds a predetermined value, or when the speed is equal to or less than the predetermined value and the inclination of the fork with respect to the ground is equal to or more than the predetermined value in the state information. Judged as running condition,
If there is a load on the fork when it is not in the running state, it is determined that the load is in the loading state.
The control program according to (9) above, wherein when there is no load on the fork when the vehicle is not in the traveling state, it is determined that the load is placed.

According to the present invention, the image processing device is obtained by photographing a display unit attached to a position visible from the driver's seat of the forklift, a photographing unit attached to the forklift and photographing the front of the vehicle body of the forklift, and the photographing unit. A state relating to an object detection unit that detects an object, a speed acquisition unit that acquires movement speed information of the vehicle body, a fork that is supported so as to be able to move up and down on the front side of the vehicle body, and a position in the height direction from the captured image. The fork state acquisition unit that acquires information, the presence / absence of luggage on the fork, or the luggage detection unit that detects the size of the luggage, the object detected by the object detection unit, the moving speed information acquired from the speed acquisition unit, and the fork state. Image processing that determines the work status of the forklift according to the fork status information acquired from the acquisition unit and the detection result of the luggage detection unit, generates a display image according to the determined work status, and displays it on the display unit. It has a part. As a result, it is possible to provide the driver with information useful for operating the forklift by displaying an image according to the work situation.

It is a side view which shows the appearance of a forklift. It is a schematic diagram which shows the state which the 1st and 2nd cameras are attached to the tip | tip portion of one fork. It is an enlarged view of a fork. It is a block diagram which shows the hardware structure of the image processing apparatus which concerns on this embodiment, and the functional structure of the whole processing part. It is a flowchart which shows the display process which an image processing apparatus executes. It is a subroutine flowchart which shows the work situation determination processing (S20). It is a subroutine flowchart which shows the display image generation processing (S30). It is a subroutine flowchart which shows the display image generation processing (S30). It is a subroutine flowchart which shows the display image generation processing (S30). It is a schematic diagram which shows the forklift which has no luggage in a running state. This is an example of a display image of display type 1 corresponding to the situation of FIG. 8A. It is a schematic diagram which shows the forklift which carries the load in a running state. It is an example of the display image of the display type 2 corresponding to the situation of FIG. 9A. This is an example of display images of display type 3 and display type 4 to be displayed in the picked-up state. It is an example of the display image of the display type 3 and the display type 4 to be displayed in the loading state.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted. The dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios. Further, in the drawing, the vertical direction is the Z direction, the traveling direction of the forklift is the X direction, and the direction orthogonal to these is the Y direction.

(forklift)
FIG. 1 is a side view showing the appearance of a forklift. An image processing device 20 is attached to the forklift 10. The forklift 10 has a vehicle body 11, a driver's seat 12, a mast 13, a finger bar 14, a pair of forks 15 and 16, and a head guard 17. A pallet 91 and a load 92 on the pallet 91 are loaded on the forks 15 and 16. A mast 13 that can be expanded and contracted in the vertical direction is provided in front of the vehicle body 11. The forks 15 and 16 are supported by the finger bar 14, and are attached to the mast 13 via the finger bar 14 so as to be vertically movable. The forks 15 and 16 are positioned in the vertical direction by moving the finger bar 14 up and down along the mast 13 via a chain (not shown) attached to the mast 13 and a wheel. Further, the inclination angle (tilt) of the forks 15 and 16 with respect to the ground (running surface), the opening angle between the forks 15 and 16, and the interval are within a predetermined range by the hydraulic cylinder (not shown) in the finger bar 14. It can be changed.

(Camera 21a, 21b)
FIG. 2 is a schematic view showing a state in which the first and second cameras 21a and 21b are attached to the tip of one fork as a fork camera. FIG. 3 is an enlarged plan view of the fork.

As shown in FIG. 2, of the two forklifts 15 and 16 of the forklift 10, two cameras 21a and 21b for stereo viewing (also referred to as compound eyes) are attached to the vehicle body 11 at the tip of one fork 15. It is installed so that the front of the camera is the shooting area. The cameras 21a and 21b function as a photographing unit. Further, in the width direction (Y direction) of the fork 15, both cameras 21a and 21b are separated by a predetermined interval (baseline length). In the example shown in the figure, two cameras are attached to the fork 15 on the left side, but the present invention is not limited to this, and the fork 16 on the right side may be attached. The cameras 21a and 21b and the overall processing unit 26 described later are connected by a cable (not shown) or wirelessly, and a video signal is transmitted to the overall processing unit 26. The first and second cameras 21a and 21b each include an image pickup element having sensitivity in a visible light region such as CCD or CMOS and an optical system such as a lens, and photograph the front of the forklift 10 to generate an image. In the following, when the two cameras 21a and 21b are not distinguished, they are collectively referred to simply as "camera 21".

As an example of the size of the fork 15, the thickness is 10 mm, the width is 100 mm, the tapered portion is R (radius) 60 mm, and the tapered portion is from the tip s1 to 40 mm in the X direction and from the side surface to 40 mm in the Y direction. .. As another application example, when a fork 15 having a tapered surface provided not only on the side surface but also on the lower surface side is used, the cameras 21a and 21b may be arranged on the tapered portion.

Cylindrical holes are provided on both the left and right sides of the tapered portion of the fork 15, and the cameras 21a and 21b are embedded in the holes, respectively. It is preferable to arrange the cameras 21a and 21b so that the front surface of the lens slightly protrudes from the outer peripheral surface of the tapered portion in that a wide angle of view can be secured, but due to a collision of the fork 15 with the floor surface or the like during use. From the viewpoint of breakage, it is more preferable to arrange it inside the opening surface of the columnar hole.

When viewing an object in stereo (stereoscopic view) for distance measurement, it is necessary to overlap the imaging areas of the two cameras 21a and 21b. In order to secure a wide field of view (angle of view) and allow more regions to overlap, it is preferable to provide cameras 21a and 21b on the tapered portion on the side surface or the lower surface. As shown in FIG. 3, the two cameras 21a and 21b arranged in the tapered portion on the side surface side can secure a wide angle of view toward the front side of the fork 15. By cooperating with the two cameras 21a and 21b arranged in this way and the overall processing unit 26, the distance to the object is detected and the AF point cloud data (also referred to as "three-dimensional distance map") is generated. ..

(Configuration of image processing device 20)
Next, the configuration of the image processing device 20 will be described with reference to FIG. 1 and FIG. FIG. 4 is a block diagram showing a hardware configuration of the image processing device 20 and a functional configuration of the overall processing unit 26.

As shown in FIGS. 1 and 4, in addition to the fork camera 21 described above, the image processing device 20 includes a luggage sensor 22, a vehicle body speed sensor 23, a fork state sensor 24, a side camera 25, an overall processing unit 26, and an overall processing unit 26. A display 27 is provided.

The overall processing unit 26 includes a CPU (Central Processing Unit) and a memory, and the CPU executes a control program stored in the memory to perform various controls on the entire image processing apparatus 20. Each function of the overall processing unit 26 will be described later.

(Luggage sensor 22)
The luggage sensor 22 functions as a luggage detection unit. The luggage sensor 22 is arranged on the finger bar 14 and determines whether or not the luggage 92 (including the pallet 91) is placed on the forks 15 and 16. The luggage sensor 22 is, for example, an optical sensor provided with a light emitting unit and a light receiving unit, and detects the presence or absence of luggage by the reflected light of the light radiated to the luggage.

(Vehicle speed sensor 23)
The vehicle body speed sensor 23 functions as a speed acquisition unit. The vehicle body speed sensor 23 includes an acceleration sensor and a gyro sensor, and outputs the detected movement speed information of the vehicle body 11. The forklift 10 and the acquisition unit 261 (described later) of the image processing device 20 may be electrically connected, and the acquisition unit 261 may function as a speed acquisition unit. The acquisition unit 261 acquires the moving speed information of the vehicle body 11 from the forklift 10. As a result, the vehicle body speed sensor 23 can be omitted.

(Fork state sensor 24)
The fork state sensor 24 functions as a fork state acquisition unit. The fork state sensor 24 is arranged, for example, on the finger bar 14. The fork state information detected by the fork state sensor 24 includes information on the posture of the fork 15 (and fork 16), that is, the inclination angle (tilt) with respect to the ground, and the height of the fork 15 with respect to the mast 13, that is, with respect to the ground. Contains height position information. These sensors are composed of, for example, an actuator and an optical element. Further, a sensor for detecting the opening angle and the distance between the forks 15 and 16 may be included. In addition, these sensors may be composed of an acceleration sensor and a gyro sensor, and may be shared with the vehicle body speed sensor 23. Angular velocity information and turning angular velocity information can be acquired by an acceleration sensor or a gyro sensor, and the position of the fork 15 relative to the mast 13, that is, the vehicle body 11, or the surroundings can be grasped. Further, as another example, the forklift 10 and the acquisition unit 261 of the image processing device 20 may be electrically connected as in the above-described modification, and the acquisition unit 261 may function as a fork state acquisition unit. The acquisition unit 261 acquires posture information and height position information of the fork 15 from the forklift 10.

(Side camera 25)
The side camera 25 has the same configuration as the camera 21, and photographs the front of the forklift 10 to generate an image. This side camera 25 also functions as a photographing unit. The side camera 25 is installed next to the backrest of the driver's seat 12 (see FIG. 1), and photographs the front of the vehicle body 11. The side camera 25 is preferably arranged so as to photograph the front at a position as far as possible from the driver's seat so that the side camera 25 can photograph a position that cannot be recognized from the driver's line of sight. The side camera 25 is attached to the driver's seat 12 or the head guard 17.

(Overall processing unit 26)
The overall processing unit 26 is mainly composed of a CPU, a memory, and an input / output I / F (interface), and processes the entire image processing apparatus 20. As shown in FIG. 4, the overall processing unit 26 functions as an acquisition unit 261, a control unit 262, and an image output unit 263.

(Acquisition unit 261)
The acquisition unit 261 is an input / output I / F, and is connected to the above-mentioned electric parts such as a camera and a sensor to acquire signals such as a video signal, luggage presence / absence information, movement speed information, fork height, and inclination information. To do. The acquired signal is sent to the control unit 262.

(Control unit 262)
The control unit 262 functions as an object detection unit 31 and an image processing unit 32.

(Object detection unit 31)
The object detection unit 31 extracts common corresponding points from the extracted feature points of a set of images, and calculates the distance to each feature point from the corresponding points using conversion parameters. For example, in a pair of cameras 21a and 21b arranged on the left and right, the distance value of each pixel is calculated according to the amount of deviation of the left and right pixel values of the same corresponding point using the baseline length (distance measurement), and three-dimensional. Generate a distance map.

Further, the object detection unit 31 detects an object in front of the forklift 10 and detects the position of the detected object in the three-dimensional space by using the generated distance map. Objects to be detected include luggage 92, pallets 91, people (workers), and other objects such as shelves and obstacles on aisles. The detection of this object may be performed, for example, by clustering the pixels according to the similarity of the distance values of the pixels. Further, clustering may be performed by combining the similarity of the distance values with the similarity of the color of the pixels. Calculate the size of each cluster determined from clustering. For example, the vertical dimension, the horizontal dimension, the total area, and the like are calculated. The "size" referred to here is an actual size, and is different from the apparent size (angle of view, that is, the spread of pixels), and a block of pixel groups is determined according to the distance to the object. For example, the object detection unit 31 determines whether or not the calculated size is equal to or larger than a predetermined size threshold value for identifying the object to be analyzed to be extracted. The size threshold can be arbitrarily set depending on the measurement location, the behavior analysis target, and the like. If the behavior is analyzed by tracking the passing workers, the minimum value of the size of a normal person may be used as the size threshold value for clustering. Objects below the size threshold are excluded. Further, if the environment in which the forklift 10 travels is limited in a specific warehouse or the like, a size threshold value corresponding to an object existing in that environment may be applied. Further, the object detection unit 31 creates a three-dimensional distance map including information on the size and position of each object of a predetermined size or larger in front of the generated forklift 10 (vehicle body 11) in the internal memory of the image processing device 20 or a server. Etc. are stored in the external memory.

Further, as the type of the object, machine learning may be used or proportion discrimination (aspect ratio) may be used for discriminating between a human being and another object. In machine learning, a computer repeats learning using the obtained three-dimensional distance map data and correct label data. Especially in the work space, it is important to discriminate the worker in order to prevent a serious accident, and it is expected that the discriminating accuracy will be improved by using these.

The object detection unit 31 also functions as a luggage detection unit that detects the size of the luggage by cooperating with the luggage sensor 22. Immediately before loading the luggage 92 on the pallet 91 on the forks 15 and 16, the size of the entire luggage 92 including the pallet 91 is detected and stored in the internal memory. As a result, the size of the luggage placed on the forks 15 and 16 can be grasped. This size includes the height and width (length in the Y direction) of the luggage 92 (including the pallet 91).

Further, the distance measurement may be performed by one camera 21a instead of the configuration in which the distance measurement is performed by the images of the pair of cameras 21a and 21b. For example, by arranging a floodlight at the position of the camera 21b and irradiating the floodlight with a pattern light composed of a plurality of grid-like lines or dots at a predetermined timing, the distance is measured and the focus point cloud group data is generated. You may try to do it.

(Image processing unit 32)
The image processing unit 32 includes a work status determination unit 321, a warning determination unit 322, and an image generation unit 323. The image processing unit 32 determines the working status of the forklift 10 and generates an image according to the determination. In addition, warning processing is performed according to the type of the object in front detected by the object detection unit 31 and the relative position (distance) to the object.

(Work status determination unit 321)
The work status determination unit 321 determines the work status of the forklift 10 according to the object detected by the object detection unit 31, the moving speed information acquired via the acquisition unit 261, the fork state information, and the detection result of the luggage detection unit. judge. The work status to be judged includes the "running state" of moving to the destination for transporting the luggage, the "picking condition" of placing the luggage on the fork at the destination, and the "loading state" of placing the luggage on the shelf of the destination. "Placed state" is included.

(Warning judgment unit 322)
The warning determination unit 322 determines whether or not to issue a warning to the object or person detected by the object detection unit 31. For example, if a person exists within a predetermined distance in the moving direction of the vehicle body 11, the driver is warned. Although the warning is given through the display 27 (for example, the display image 271 in FIG. 8B described later), the warning may be given by a warning light or a speaker (neither shown) provided on the forklift 10.

(Image generation unit 323)
The image generation unit 323 generates a display image to be displayed on the display 27 based on the processing results of the object detection unit 31, the work status determination unit 321 and the warning determination unit 322.

(Image output unit 263)
The image output unit 263 outputs the display image generated by the image generation unit 323 to the display 27 in real time and displays it to the driver.

(Display 27)
The display 27 functions as a display unit. The display 27 is attached at a position visible from the driver's seat 12 of the forklift 10. As the display 27, for example, a HUD (head-up display) can be applied. When the HUD is applied, the display 27 comprises a combiner that is a semitransparent concave or plane mirror and projects a virtual image onto the combiner. The virtual image is a display image generated by the above-mentioned overall processing unit 26. This combiner is arranged on the front side of the driver's seat 12, and the driver sitting in the driver's seat 12 can see the real image (forks 15, 16, luggage 92, etc.) ahead of the combiner through the combiner, and the combiner reflects. You can recognize virtual images at the same time. By using the HUD, even if the combiner is placed in front of the driver's seat, the combiner becomes transparent when no image is projected, so that the forward visibility is not obstructed.

Further, as another example of the display 27, a transmissive liquid crystal display may be applied. The transmissive liquid crystal display does not obstruct the forward view as much as the HUD.

(Image display processing)
Hereinafter, the display processing executed by the image processing apparatus 20 will be described with reference to FIGS. 5 to 11. FIG. 5 is a flowchart showing the display process, and FIGS. 6 and 7A to 7C are subroutine flowcharts showing a part of the process of FIG.

(Step S10)
Through the acquisition unit 261, the control unit 262 receives video signals and luggage from the above-mentioned camera 21, luggage sensor 22, vehicle body speed sensor 23, fork state sensor 24, and various cameras and sensors of the side camera 25. Acquire signals such as presence / absence information, moving speed information, fork height, and inclination information.

(Step S20)
The control unit 262 determines the work status from the signal acquired in step S10. FIG. 6 is a subroutine flowchart showing the work status determination process in step S20.

(Step S210)
The work status determination unit 321 of the image processing unit 32 determines whether or not the current vehicle body speed exceeds a predetermined value from the movement speed information acquired from the vehicle body speed sensor 23. As the predetermined value, for example, any value in the range of 2 to 10 km / h can be applied. For example, assuming that the predetermined value is 3 km / h, if the vehicle body 11 is moving at a speed exceeding the predetermined value (YES), the process proceeds to step S220, and if it is equal to or less than the predetermined value (NO), the process is performed in step S230. Proceed to.

(Step S220)
The work status determination unit 321 determines the work status as a "running state". As described above, the processing of the subroutine flowchart is completed, and the process returns to the main processing of FIG. 5 (end (return)).

(Step S230)
The work status determination unit 321 further determines the inclination of the forks 15 and 16 based on the state information of the forks 15 and 16 acquired in step S10. If there is an inclination angle equal to or more than a predetermined value, that is, if the inclination angle is equal to or more than a predetermined value with respect to the horizontal plane and the tip side of the forks 15 and 16 is inclined to be higher than the root side (YES), the process is performed in step S220. Proceed to. In this case as well, the determination result is the "running state". On the other hand, if it is not tilted (NO), the process proceeds to step S240.

(Step S240)
The work status determination unit 321 determines whether or not the luggage 92 is on the forks 15 and 16 based on the signal from the luggage sensor 22 acquired in step S10. If there is no baggage 92 (NO), the process proceeds to step S250, and if there is baggage 92 (YES), the process proceeds to step S260.

(Steps S250, S260)
The work status determination unit 321 determines the work status as the “loading state” in step S250, and determines the work status as the “loading state” in step S260. As described above, the processing of the subroutine flowchart is completed, and the process returns to the main processing of FIG. 5 (end (return)).

(Step S30)
In step S30 of FIG. 5, a display image corresponding to the work situation determined in step S20 (FIG. 6) is generated. 7A to 7C are subroutine flowcharts showing the display image generation process of step S30.

(Running condition)
(Step S310)
The image processing unit 32 proceeds to the next step S320 if the work status determined in step S20 is the "running state", and proceeds to the process if the work status is the "pick-up state" in step S410 (FIG. 7B). ) (Indicated by the circled number 40). Further, if the work status is "loading state", the image processing unit 32 proceeds to the process in step S510 (FIG. 7C) (indicated by the circled number 50).

(Step S320)
The warning determination unit 322 determines whether or not an object to be warned is detected at a predetermined position from the object detected by the object detection unit 31. As the object to be warned, for example, when there is a person in front of the forklift 10 in the moving direction, this person is detected as the object to be warned because it may come into contact with the object in the future. In addition, when an obstacle such as a wall, a pillar, or an abandoned object on the road that may come into contact with the forklift 10 is detected, this is detected as an object to be warned. If the object to be warned is not detected (NO), the image processing unit 32 ends the process and returns to the main process of FIG. In this case, the image generation unit 323 does not generate a display image.

(Step S330)
The image processing unit 32 determines whether or not the luggage 92 is on the forks 15 and 16. This determination can be made by the signal of the luggage sensor 22. If the luggage 92 is on the forks 15 and 16 (YES), the process proceeds to step S340, and if there is no luggage 92 (NO), the process proceeds to step S350.

(Step S340)
When there is a load 92 on the fork, the image processing unit 32 further determines the height of the load (the height of the top of the load, hereinafter referred to as “lift height” or “load lift height”). Judge whether it is less than or equal to the predetermined value. This lift height is the height from the ground on which the forklift 10 travels, and a predetermined value is set based on the line-of-sight height of the driver sitting in the driver's seat 12. That is, if it is not more than a predetermined value, in the horizontal direction, the driver can see ahead of the luggage 92 without obstructing the front view even if there is the luggage 92. The predetermined value may be adjusted within a predetermined range according to the seat height of the driver's seat 12 or the seat height of the driver.

Here, the image processing unit 32 calculates the load lifting height by adding the height of the load 92 loaded on the forks 15 and 16 to the height of the forks 15 and 16 from the ground. The height of the luggage 92 is calculated by the object detection unit 31 from the three-dimensional distance map generated by the image of the camera 21 and stored in the internal memory immediately before loading on the forks 15 and 16. .. The lift height of the luggage 92 is more preferably calculated by taking into account the thickness of the pallet insertion port in the height direction. Further, the lift height of the luggage 92 is the total height when a plurality of luggage 92s are stacked as shown in FIG.

If the lift height of the luggage 92 calculated from the luggage height and the fork height is equal to or less than a predetermined value (YES), the image processing unit 32 proceeds to step S350, and if it exceeds the predetermined value (NO), the processing is performed. To step S360.

(Steps S350, S360)
The image generation unit 323 generates a display image of display type 1 in step S350, and generates a display image of display type 2 in step S360. As described above, the processing of the subroutine flowchart is completed, and the process returns to the main processing of FIG. 5 (end (return)).

(Step S40)
Here, the control unit 262 sends the display image of the display type 1 or the display type 2 as shown below generated in step S30 to the image output unit 263 and displays it on the display 27.

(Display image of display type 1)
FIG. 8A is a schematic view showing a forklift 10 without luggage in a traveling state. In FIG. 8A, in front of the forklift 10, there is a shelf 96 on which a worker (pedestrian) 95 and a luggage 92 are placed. The shelf 96 is a two-level shelf composed of a metal frame, and is arranged on the front side of the wire mesh 97. FIG. 8B is an example of the display image 271 of the display type 1 displayed on the display 27, which corresponds to the situation of FIG. 8A. In step S350, there is no luggage on the forks 15 and 16, or even if there is luggage, the front view is not obstructed. Therefore, the driver in the driver's seat 12 can sufficiently visually confirm the front, and detailed warning is unnecessary. Therefore, the display image of the display type 1 is limited to a simple warning display. In the example of the display image 271 of the display type 1 shown in FIG. 8B, a person as a warning object is detected. Then, an additional image 51 of an annotation for warning to a person is added. The display image 271 is blank except for the additional image 51, which is omitted in the figure, but the driver can recognize the real image beyond the display 27 (combiner).

(Display image of display type 2)
FIG. 9A is a schematic view showing a forklift 10 having a traveling load. Compared to FIG. 8A, the overall height of the load 92 on the loaded fork 15 is higher, obstructing the driver's view. FIG. 9B is an example of a display image 272 of the display type 2 displayed on the display 27, which corresponds to the situation of FIG. 9A. In step S360, as shown in FIG. 9A, the front view is obstructed by luggage on the forks 15 and 16, and it is difficult for the driver in the driver's seat 12 to visually confirm. Therefore, in the display image 272 displayed on the display 27, the image of the camera 21 is displayed as it is, and the additional image 52 of the annotation of the rectangular frame corresponding to the person is displayed as the information for the object to be warned. Display on top of the position. Further, in the display image 272, an additional image 53 indicating the traveling direction of the forks 15 and 16 is superimposed and displayed in the center. Further, on the right side of the display image 272, an additional image 54 showing the inclination angles of the forks 15 and 16 is superimposed and displayed.

(Collecting state)
(Step S410)
Next, with reference to FIG. 7B, the processing in the “pick-up state” will be described. Step S410 is a process following step S310 of FIG. 7A. In this process, the image processing unit 32 determines whether or not the heights of the forks 15 and 16 are equal to or less than a predetermined value. This determination can be made by the signal of the fork state sensor 24. This predetermined value is set to a height at which the driver can visually confirm the tips of the luggage 92 (pallet 91) and the forks 15 and 16. For example, this predetermined value is the height of the line of sight, or a height slightly higher than this. If the height is equal to or less than a predetermined value (YES), the image processing unit 32 proceeds to step S420, and if it exceeds the predetermined value (NO), the image processing unit 32 proceeds to step S430.

(Steps S420, S430)
The image generation unit 323 generates a display image of display type 3 in step S420, and generates a display image of display type 4 in step S430. As described above, the processing of the subroutine flowchart is completed, and the process returns to the main processing of FIG. 5 (end (return)). After that, the image processing unit 32 causes the display 27 to display the display image generated by the process of step S40.

(Display image of display type 3)
FIG. 10 is an example of the display image 273 of the display type 3 and the display image 274 of the display type 4 displayed in the picked-up state. In step S420, the heights of the forks 15 and 16 are not so high, and the driver can sufficiently visually confirm the positions of the tips of the forks 15 and 16 and the luggage 92. Therefore, it is not necessary to display auxiliary information on the display 27, particularly in the loading operation. In step S420, the image generation unit 323 does not create a display image, or generates a blank display image as a display image of display type 3. In this case, the driver visually performs the loading work through the combiner.

(Display image of display type 4)
In step S430, since the heights of the forks 15 and 16 are high, it is difficult for the driver to visually confirm the positions of the tips of the forks 15 and 16 and the luggage 92. The image generation unit 323 generates an image useful for loading as a display image of the display type 4. As shown in FIG. 10, in the display image 274 of the display type 4 displayed on the display 27, the image of the camera 21 attached to the tip of the fork 15 is displayed as it is, and a cross or the like indicating the tip of the fork 15 is added. The image 55 is superimposed and displayed. Further, on the right side of the display image 274, an additional image 54 showing the inclination angles of the forks 15 and 16 is superimposed and displayed. The image processing unit 32 recognizes the insertion port of the pallet 91 by the matching process with the brightness difference from the surroundings and the pallet shape information registered in advance based on the image obtained by the camera 21, and the recognized insertion port and the insertion port. , Information indicating the positional relationship with the fork tip (for example, a distance value) may be superimposed on the display image 274 as an additional image.

(Loading state)
(Step S510)
Next, with reference to FIG. 7C, the processing in the “loading state” will be described. Step S510 is a process following step S310 of FIG. 7A. In this process, the image processing unit 32 determines whether or not there is a loading obstacle. A loading obstacle is an object that becomes an obstacle when loading. In front of the forklift 10, the object detection unit 31 detects the frame of the shelf at the loading location, other luggage in the shelf, walls, and the like, and determines these objects as obstacles. Based on the detection result of the object detection unit 31, the image processing unit 32 proceeds to the process in step S520 if there is an obstacle (YES), and proceeds to the process in step S530 if there is no obstacle (NO). The object detection unit 31 may detect the size and shape of the space for placing the luggage in front of the forks 15 and 16.

(Step S520)
The image processing unit 32 determines the position of the obstacle and the width of the load 92 loaded on the forks 15 and 16. Specifically, it is determined from the position information of the obstacle whether or not the difference distance between the width of the loading space (space) and the width of the luggage 92, that is, whether the gap is equal to or less than a predetermined value. If it is equal to or less than the predetermined value (YES), the process proceeds to step S540, and if there is a margin exceeding the predetermined value (NO), the process proceeds to step S530.

(Steps S530, S540)
The image generation unit 323 generates a display image of display type 5 in step S530, and generates a display image of display type 6 in step S540. As described above, the processing of the subroutine flowchart is completed, and the process returns to the main processing of FIG. 5 (end (return)). After that, the image processing unit 32 causes the display 27 to display the display image generated by the process of step S40.

(Display image of display type 5)
FIG. 11 is an example of the display image 275 of the display type 5 and the display image 276 of the display type 6 displayed in the loading state. In step S530, there is no obstacle around the loading space, or there is a margin (gap) to the extent that the work is not hindered, and there is no risk of collision with the obstacle during the loading work. , There is no need to display auxiliary information in the loading operation. In step S530, the image generation unit 323 does not create a display image, or generates a blank display image as a display image of display type 5. In this case, the driver visually performs the loading work through the combiner.

(Display image of display type 6)
In step S540, the difference distance (gap) is equal to or less than a predetermined value, and the luggage 92 may collide with an obstacle. Therefore, the driver is required to perform careful work. The image generation unit 323 generates an image useful for loading as a display image of the display type 6. As shown in FIG. 11, in the display image 276 of the display type 6 displayed on the display 27, the image acquired from the side camera 25 is displayed as it is as auxiliary information for loading. In the display image 276, the additional image 57 is further superimposed. The additional image 57 is an image of a rectangular frame corresponding to the positions of the lower end portion and the right end portion (side camera 25 side) of the luggage 92 as information indicating the position of the luggage 92. Further, on the right side of the display image 276, an additional image 58 showing the inclination angles of the forks 15 and 16 is superimposed and displayed.

As described above, the image processing device of the present embodiment has the object detected by the object detection unit, the moving speed information acquired from the speed acquisition unit, the fork state information acquired from the fork state acquisition unit, and the detection result of the luggage detection unit. It is provided with an image processing unit that determines the work status of the forklift truck, generates a display image according to the determined work status, and displays the display image on the display unit. As a result, it is possible to provide information useful for the operation of the driver's forklift by displaying an image according to the work situation.

The configuration of the image processing device 20 for a forklift described above is not limited to the above configuration, and various modifications are made within the scope of the claims, as the main configuration has been described in explaining the features of the above-described embodiment. can do. Further, the configuration provided in a general image processing device is not excluded.

For example, in the above-described embodiment, a transparent combiner arranged in front of the driver's seat 12 or a transmissive liquid crystal display is used, and a display unit capable of visually confirming the front of the vehicle body 11 without obstructing the driver's field of vision. Was used. However, the present invention is not limited to this, and a normal liquid crystal display may be applied as the display unit. In this case, it is preferable to arrange the liquid crystal display diagonally above the driver's seat 12 at a position that does not obstruct the front view, for example, a head guard 17. By using a normal liquid crystal display, it is possible to display a high-resolution image even at a low cost.

Further, the display image of each display type is merely an example, and other display images may be applied. For example, the image of the side camera 25 may be displayed as the display image of the display type 2. The image of the camera 21 may be displayed as the display image of the display type 6. Further, in the loading state, as the display images of the display types 5 and 6, the lift height of the luggage 92 on the forks 15 and 16 is determined as in step S340 of the running state, and when the lift height exceeds a predetermined value. May generate and display a display image for assisting the loading operation. Even in the loading state and when the lift height of the luggage 92 obstructs the front view, the work can be easily performed.

Further, the determination of the inclination angles of the forks 15 and 16 in step S230 of FIG. 6 is effective when there is a function of automatically tilting the fork when the forklift 10 is running, and when there is no such function, step S230 The processing of may be omitted.

The means and methods for performing various processes in the image processing apparatus according to the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versaille Disc) -ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. Further, the above program may be provided as a single application software, or may be incorporated into the software of the device as a function of the image processing device.

10 Forklift 11 Body 12 Driver's seat 13 Mast 14 Finger bar 15, 16 Fork 20 Image processing device 21a, 21b Camera (fork camera)
22 Luggage sensor 23 Body speed sensor 24 Fork state sensor 25 Side camera 26 Overall processing unit 261 Acquisition unit 262 Control unit 31 Object detection unit 32 Image processing unit 321 Work status judgment unit 322 Warning judgment unit 323 Image generation unit 263 Image output unit 27 Display 271-276 Display image 91 Pallet 92 Luggage

Claims (10)

A display mounted in a position that can be seen from the driver's seat of the forklift,
A photographing unit attached to the forklift and photographing the front of the vehicle body of the forklift,
An object detection unit that detects an object from the image obtained by shooting the imaging unit,
A speed acquisition unit that acquires the moving speed information of the vehicle body, and
A fork state acquisition unit that acquires state information regarding the posture of the fork that is vertically supported on the front side of the vehicle body and the position in the height direction, and
A luggage detection unit that detects the presence or absence of luggage on the fork or the size of the luggage,
The forklift according to the object detected by the object detection unit, the movement speed information acquired from the speed acquisition unit, the state information of the fork acquired from the fork state acquisition unit, and the detection result of the luggage detection unit. An image processing unit that determines the work status of the above, generates a display image according to the determined work status, and displays the display image on the display unit.
An image processing device comprising. The image processing unit determines whether or not the object is a warning target based on the type of the object detected by the object detection unit.
The image processing device according to claim 1, wherein the image processing unit includes information about an object to be warned in the display image. The work conditions include a running state, a unloading state, and a unloading state.
The image processing unit
When the speed of the moving speed information exceeds the predetermined value, or when the speed is equal to or less than the predetermined value and the inclination of the fork with respect to the ground is equal to or more than the predetermined value in the state information, the traveling state is determined.
If there is no load on the fork when it is not in the running state, it is determined that the load is in the loading state.
The image processing device according to claim 1 or 2, wherein if there is a load on the fork when the vehicle is not in the traveling state, it is determined that the load is in the loading state. In the above-mentioned running state,
The image processing unit determines the height of the luggage and the height of the fork when the type of the object to be warned includes a luggage and detects the luggage.
The image processing apparatus according to claim 3, wherein the display image includes an image according to the determination result. In the case of the loading state,
The image processing unit determines the height of the fork and determines the height of the fork.
The image processing device according to claim 3 or 4, wherein the display image includes an image according to the determination result. In the case of the loading state,
The image processing unit determines the presence or absence of an obstacle in front of the vehicle body, and if there is an obstacle, the position of the obstacle and the width of the luggage.
The image processing apparatus according to any one of claims 3 to 5, wherein the displayed image includes an image according to the determination result. The image processing apparatus according to any one of claims 1 to 6, wherein the photographing unit includes a camera attached to a tip end portion of the fork. The image processing apparatus according to any one of claims 1 to 7, wherein the display unit is a head-up display that projects a virtual image onto a combiner attached to the front side of the vehicle body, or a transmissive liquid crystal display. .. A computer that controls an image processing device used for a forklift, which includes a display unit attached to a position visible from the driver's seat of the forklift and an imaging unit attached to the forklift to photograph the front of the vehicle body. It is a control program executed by
In step (a) of acquiring an image by the photographing unit,
In step (b) of detecting an object from the image acquired in step (a),
In step (c) of acquiring the moving speed information of the vehicle body,
Step (d) of acquiring state information regarding the posture of the fork supported so as to be able to move up and down on the front side of the vehicle body and the position in the height direction, and
In step (e) of detecting the presence or absence of luggage on the fork or the size of the luggage,
According to the object detected in the step (b), the moving speed information obtained in the steps (c) and (d), the state information of the fork, and the detection result of the step (e). A step (f) of determining the work status of the forklift, generating a display image according to the determined work status, and displaying the display on the display unit.
A control program for executing processing including. The work conditions include a running state, a unloading state, and a unloading state.
In the step (f), when the speed of the moving speed information exceeds a predetermined value, or when the speed is equal to or less than the predetermined value and the inclination of the fork with respect to the ground is equal to or more than the predetermined value in the state information. Judged as running condition,
If there is a load on the fork when it is not in the running state, it is determined that the load is in the loading state.
The control program according to claim 9, wherein when there is no load on the fork when the vehicle is not in the traveling state, it is determined that the load is stored.