WO2019163378A1 - Forklift image processing device and control program - Google Patents

Abstract

The purpose of the invention is to allow a forklift to check in front while in a loaded state and to measure the distance to an object stably. An image processing device 20 is provided with, on the distal end of one fork 15, cameras 21, 22 to photograph in front of the fork 15, and a detection sensor for detecting the distance to an object in front. Based on the detection information from the detection sensor, the image processing device processes the images acquired by the cameras 21, 22 and displays same on a display 25.

Description

Image processing apparatus and control program for forklift

The present invention relates to an image processing apparatus for a forklift mounted on a forklift and a control program.

The forklift moves with the load on the pallet on the fork. For example, in a seat-type forklift in which a driver faces a traveling direction and sits on a driver's cab, a blind spot is formed forward when a load is loaded on the front fork higher than the driver's line of sight. When traveling, the driver moves while moving the forklift backward. However, at the time of cargo handling, it is necessary to move forward, and the driver has to lean out to the side and see it.

Also, there is a demand for the driver to see the front side of the pallet in front of the fork or in the state where the fork is inserted even in the loading and unloading work at the high shelf where it is difficult to see the front of the fork from the driver. With respect to such a problem, the forklift disclosed in Patent Document 1 is provided with a camera for photographing the front of the fork, and the photographed image is displayed on the display.

Also, there is a technique for obtaining a forward view by providing a camera on each of the left and right forks of the forklift, calculating a distance to an object in front by stereo vision, and displaying the calculation result (Patent Document 2).

JP 2003-246597 A JP 2013-86959 A

¡When two forks of a forklift are attached to a finger bar, generally there is intentionally rattling between them. For this reason, the two forks may move apart from each other due to vibration during traveling, or may move differently from the attached finger bar and the mast portion that supports them.

In Patent Document 1, one camera is provided on the fork. In this case, although it is possible to visually recognize the front, the distance from the object in front is not known only by looking at the image.

In this regard, with the technique disclosed in Patent Document 2, it is possible to measure the distance by stereo viewing by attaching a camera to each of the two forks. However, as described above, the distance between the two forks is not stable, and the distance between the two cameras (base line length) is not stable, and the two forks move apart due to the loose mounting of the forks. . In many cases, the parallelism of the two forks cannot be ensured. For this reason, it is impossible or very difficult to maintain accuracy in stereo vision calculation.

Furthermore, in the technique disclosed in Patent Document 2, since the camera is arranged on the base side of each fork, the front field of view becomes very narrow in the loaded state, so that the front field of view is sufficiently reduced. It cannot be secured.

In the technique disclosed in Patent Document 1, a camera is provided at the tip of the fork. For this reason, when loaded on a fork, the camera at the tip of the fork is generally close to the ground, and the image from the camera has an angle of view looking up from below. Even if is displayed, the driver has a problem that it is difficult to grasp the surrounding situation.

The present invention has been made in view of the above-described circumstances, and a first object of the present invention is to provide a forklift that can confirm the front in a loaded state and can stably measure the distance to an object. An image processing apparatus is provided.

The second purpose is to provide a safe working environment while easily confirming the front situation even in a loaded state in a forklift.

The above object of the present invention is achieved by the following means.

(1) An image processing apparatus used for a forklift,
A camera that is provided at a front end portion of one fork among a plurality of forks supported to be movable up and down on the front side of the forklift, and shoots the front of the forklift;
A detection sensor for detecting a distance to an object in front of the forklift, provided at the tip portion of the fork provided with the camera;
Based on detection information of the detection sensor, a processing unit that processes the video acquired by the camera;
A display for displaying a processed image processed by the processing unit;
An image processing apparatus comprising:

(2) The first imaging device and the second imaging device are provided on one fork so that at least a part of each imaging region overlaps,
At least one of the first and second imaging elements functions as a part of the camera, and both the first and second imaging elements function as the detection sensor,
The processing unit detects a distance to an object in front of the fork based on images acquired from both the first and second imaging elements, and processes the image based on the detected distance. The image processing apparatus according to (1) above.

(3) A projector that emits light toward the front of the forklift, or that emits a two-dimensional pattern light toward the front of the forklift,
With
The projector and the processing unit also function as the detection sensor, and the processing unit is in front of the forklift based on an image from the camera that has captured the light emitted from the projector or the pattern light. The image processing apparatus according to (1) or (2), wherein a distance to an object is detected and the video is processed based on the detected distance.

(4) The image processing apparatus according to (1), wherein the detection sensor is a distance measurement sensor that detects a distance to an object in front of the forklift and acquires a plurality of distance measurement point group data.

(5) The tapered portion of the tip of the fork, the width gradually decreases toward the tip when viewed from above, and / or the bottom surface is inclined when viewed from the side so that the thickness gradually decreases toward the tip. The image processing apparatus according to any one of (1) to (4), wherein the camera and the detection sensor are provided in a tapered portion.

(6) The front of the forklift is provided as an imaging region so that the first imaging device and the second imaging device share at least a part of each imaging region with one fork.
At least one of the first and second imaging elements functions as a part of the camera, and both the first and second imaging elements function as the detection sensor,
The processing unit detects a distance to an object in front of the forklift based on images acquired from both the first and second imaging elements,
The tapered portion at the tip of the fork has a width that gradually decreases toward the tip when viewed from above and a thickness that gradually decreases toward the tip when the bottom surface is inclined when viewed from the side. The image processing apparatus according to (1), wherein the first and second imaging elements are arranged on both sides.

(7) Furthermore, a storage unit is provided,
The processing unit generates a distance map that is a distance measuring point group indicating a position or a shape of an object in a work space where the forklift operates based on an image from the camera and detection information of the detection sensor. The image processing apparatus according to any one of (1) to (6), which is stored in the storage unit.

(8) The distance map in which the processing unit accumulates a part of the distance data to the object detected based on the detection sensor or detection information of the detection sensor and an image from the camera in the storage unit. The image processing apparatus according to (7), wherein correction is performed by

(9) Furthermore, a position detection sensor for acquiring the position state of the fork is included,
The image processing apparatus according to any one of (1) to (8), wherein the processing unit acquires the position state of the fork provided with the camera by the position detection sensor.

(10) As the processed image, the processing unit
The image processing device according to any one of (1) to (9), wherein an image obtained by adding additional information corresponding to a distance to a front object to the image acquired by the camera is displayed on the display. .

(11) As the processed video, the processing unit
The image processing apparatus according to any one of (1) to (10), wherein an image obtained by performing viewpoint conversion on an image acquired by the camera is displayed on the display.

(12) The image processing apparatus according to any one of (1) to (11), wherein the camera performs exposure using a central portion of a shooting angle of view.

(13) The image processing apparatus according to any one of (1) to (12), wherein the display is a head-up display that projects a virtual image on a combiner attached to the forklift.

(14) The combiner is disposed at a position where the front side of the forklift can be seen through,
The image processing apparatus according to (13), wherein the head-up display has a virtual image projection distance set in a range of 50 cm to 20 m.

(15) Any of (1) to (14) above, wherein the camera and the detection sensor provided at a tip portion of the fork are attached to a main body of the fork via an impact relaxation member. An image processing apparatus according to 1.

(16) At least a part of the electronic components constituting the camera and the detection sensor are connected to the main body of the fork via a heat conductive member made of a flexible and high heat conductive material. The image processing apparatus according to (15).

(17) An image processing apparatus used for a forklift,
A camera for photographing the front of the forklift;
A distance sensor to measure the distance to an object in front of the forklift, and a distance sensor data indicating a distance value distribution, and a detection sensor;
A processing unit that performs processing for adding an image of a distance value based on the acquired distance measuring point group data to the video acquired by the camera;
A display for displaying the processed video processed by the processing unit;
An image processing apparatus comprising:

(18) The image processing device according to (17), wherein the camera includes an image sensor having sensitivity in a visible light region.

(19) The image processing apparatus according to (17) or (18), wherein the camera is installed on a fork supported to be movable up and down on a front side of the forklift so as to photograph the front.

(20) The image processing apparatus according to (19), wherein the camera performs exposure using a central portion of a shooting angle of view.

(21) The processing unit according to any one of (17) to (20), wherein the processing unit further performs viewpoint conversion processing on the video based on the distance measurement point group data. Image processing apparatus.

(22) Furthermore, a position detection sensor for acquiring posture information of the camera is provided,
The image processing apparatus according to (21), wherein the processing unit performs the viewpoint conversion process using the posture information acquired from the position detection sensor.

(23) Furthermore, a storage unit is provided,
The image processing apparatus according to (21) or (22), wherein the processing unit creates a three-dimensional distance map using distance measurement point group data acquired by the detection sensor and stores the three-dimensional distance map in the storage unit.

(24) The three-dimensional distance map stored in the storage unit includes drawing data relating to a building or facility in which the forklift is used, distance measuring point group data obtained from sensors installed in the building, and other vehicles. The image processing apparatus according to (23), wherein the position information of the package and / or the position information of the package acquired from the physical distribution information system used in the building are reflected.

(25) The image processing device according to (24), wherein the three-dimensional distance map includes position information of a floor surface, a wall surface, a window, or a lighting device related to the building or equipment.

(26) The viewpoint conversion process includes a viewpoint conversion process in which the viewpoint position of the driver sitting on the cab of the forklift is a virtual viewpoint position, and a viewpoint conversion process in which a position higher than the driver's viewpoint position is a virtual viewpoint position. Alternatively, the image processing device according to any one of (21) to (25) above, wherein the viewpoint conversion processing uses a position away from the forklift as a virtual viewpoint position.

(27) The viewpoint conversion process in which the driver's viewpoint position is a virtual viewpoint position is a viewpoint conversion process by keystone correction according to an angle or height of the camera with respect to the ground, or the distance measurement point group data, Alternatively, the image processing device according to (26), which is viewpoint conversion processing using a three-dimensional distance map stored in a storage unit.

(28) The viewpoint conversion process in which a position higher than the driver's viewpoint position is a virtual viewpoint position is a viewpoint conversion process using the distance measurement point group data or a three-dimensional distance map stored in a storage unit. The image processing apparatus according to (26) above.

(29) In the viewpoint conversion process in which a position higher than the viewpoint position of the driver is a virtual viewpoint position, or in the viewpoint conversion process in which a position away from the forklift is a virtual viewpoint position, regarding the blind spot area of the camera, In the angle of view of the camera, the texture of the surface of the object above the object where the blind spot area is formed and at a distance farther than the object is arranged in the blind spot area (26) or ( 28) The image processing apparatus according to 28).

(30) In the viewpoint conversion process in which a position higher than the viewpoint position of the driver is a virtual viewpoint position, or in the viewpoint conversion process in which a position away from the forklift is a virtual viewpoint position, regarding the blind spot area of the camera, (26) or (28), wherein the object outline information in the three-dimensional distance map stored in the storage unit is used to superimpose the object outline existing in the blind spot area on the blind spot area. Image processing apparatus.

(31) The image processing device according to any one of (21) to (30), wherein the viewpoint conversion processing changes presence or absence of the viewpoint conversion processing or intensity according to a distance to the object.

(32) The display is a transparent screen or a head-up display attached to the forklift so that the front of the forklift can be seen through.
The processing unit recognizes an object in front of the forklift, generates an additional image corresponding to the distance and / or direction to each of the recognized objects, and superimposes the generated additional image on each of the objects. The image processing apparatus according to any one of (17) to (20), wherein the image is displayed on the transparent screen or the head-up display.

(33) The processing unit recognizes an object in front of the forklift and, as the processing process, adds an additional image corresponding to the recognized type of the object, the distance to the object, or the position to the video. The image processing device according to any one of (17) to (32), which is generated and added to the video.

(34) When the processing unit recognizes a pallet as the object, the processing unit determines an inclination with respect to the pallet based on a shape of the insertion port of the pallet, and the additional image corresponding to the determined inclination amount of the horizontal plane of the pallet The image processing device according to (33), wherein

(35) In the additional image generated by the processing unit, the content information of the package acquired from the physical distribution information system used in the building where the forklift is used, the empty shelf information indicating the availability of the shelf, and the procedure for handling the cargo are included. The image processing device according to any one of (32) to (34), wherein at least one of the cargo handling procedure information to be displayed is included.

(36) The processing unit generates an overhead view image of an upper viewpoint according to the distance to the object, adds the generated overhead image and displays the generated image on the display, any of (17) to (35) above An image processing apparatus according to claim 1.

(37) The processing unit recognizes an object in front of the forklift and issues a warning when the distance from the forklift or the fork tip of the forklift becomes a predetermined value or less, or the display of the display The image processing device according to any one of (17) to (36), wherein the display is switched to a proximity screen.

(38) The processing unit recognizes an object in front of the forklift and outputs information on the shortest distance from the fork tip of the object recognized in the image of the distance value. 37) The image processing device according to any one of

(39) An image processing apparatus used for a forklift, a distance measuring point group data indicating a distance value distribution by measuring a distance between a camera for photographing the front of the forklift and an object in front of the forklift A control program that is executed by a computer that controls an image processing apparatus including a detection sensor for acquiring
Acquiring an image by the camera (a);
A step (b) of obtaining ranging point cloud data with the detection sensor;
A step (c) of performing processing for adding an image of a distance value based on the acquired distance measuring point group data to the video acquired by the camera;
Displaying the processed image on a display (d);
A control program for causing the computer to execute a process including:

(40) The control program according to (39), wherein in the step (c), as the processing, a viewpoint conversion process is further performed on the video based on the distance measurement point group data.

(41) The process further includes:
Recognizing an object in front of the forklift (e),
In the step (c), as the processing, an additional image corresponding to the recognized object type, the distance to the object, or the position and position is generated in the video and added to the video (39) Or the control program as described in said (40).

According to the first invention, a camera for photographing the front of the fork and a detection sensor for detecting the distance to the object in front are provided at the tip of one fork. Based on this, the video acquired by the camera is processed and displayed on the display. By doing in this way, the driver can confirm the front by the screen displayed on the display and can stably measure the distance to the object even when it is difficult to see the front by loading on the fork.

Further, according to the second invention, a camera for photographing the front of the forklift and a detection for measuring the distance to the object in front of the forklift and obtaining distance measurement point group data indicating the distribution of distance values A sensor, and a processing for adding an image of a distance value based on the acquired distance measurement point group data to the video acquired by the camera, and displaying the processed video on a display. By doing in this way, the driver can easily check the front situation by the screen displayed on the display even when it is difficult to see the front by loading on the fork, and can provide a safe working environment. .

It is a side view which shows the external appearance of a forklift. 1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to a first embodiment and a functional configuration of a processing unit. It is a schematic diagram which shows the state which attached the 1st, 2nd camera to the front-end | tip part of one fork. It is an enlarged view of a fork. FIG. 5 is a cross-sectional view taken along the line A-A ′ of FIG. It is an enlarged view of the fork of another example. It is a schematic diagram explaining the angle of view of the horizontal direction of a 1st, 2nd camera. It is a schematic diagram explaining the angle of view of the perpendicular direction of a 1st, 2nd camera. It is a schematic diagram explaining the attachment position and angle of view of a camera. It is an example of the display screen displayed on the display. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on 2nd Embodiment, and the functional structure of a process part. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on an example in 3rd Embodiment. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on a 1st modification, and the function structure of a process part. It is a schematic diagram which shows the state which attached the 1st camera and the 2nd camera to the front-end | tip part of one sheath. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on a 3rd modification, and the function structure of a process part. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on a 4th modification, the function structure of a process part, and the structure of HUD. It is a schematic diagram which shows the structure of HUD. It is a schematic diagram which shows the state which attached the 1st camera and the 2nd camera to the front-end | tip part of the one sheath fork in a 5th modification. It is a flowchart which shows the display process which the image processing apparatus which concerns on 4th Embodiment performs. It is a modification of the display screen which added the bird's-eye view image. It is a figure explaining the process to the blind spot area | region which arises by viewpoint conversion. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on a 6th modification, and the function structure of a process part. It is an example of the screen for proximity | contact displayed on the display in a 7th modification. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on an 8th modification, the functional structure of a process part, and the structure of HUD. It is an example of the virtual image regarding the package content information displayed on HUD in the 8th modification, empty shelf information, and cargo handling procedure information.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. The dimensional ratios in the drawings are exaggerated for convenience of explanation, and may differ from actual ratios. In the drawings, the vertical direction is the Z direction, the forklift traveling direction is the X direction, and the direction perpendicular to these is the Y direction.

(forklift)
FIG. 1 is a side view showing an appearance of a forklift. The forklift 10 includes a main body 11, a cab 12, a mast 13, a finger bar 14, a pair of forks 15 and 16, and a head guard 17. A pallet 91 and a luggage 92 on the pallet 91 are loaded on the forks 15 and 16. A mast 13 that can be expanded and contracted in the vertical direction is provided in front of the main body 11, and the forks 15 and 16 are supported by the finger bar 14 and are attached to the mast 13 through the finger bar 14 so as to be vertically movable. It has been. As the finger bar 14 moves up and down along the mast 13 via a chain (not shown) attached to the mast 13 and a wheel, the positions of the forks 15 and 16 are controlled in the vertical direction. Further, the inclination angle (tilt) of the forks 15 and 16 with respect to the ground (traveling surface) can be changed within a predetermined range by a hydraulic cylinder (not shown) connected to the mast 13. Further, the opening angle and interval between the forks 15 and 16 may be changed within a predetermined range by a hydraulic cylinder (not shown) in the finger bar 14. The forks 15 and 16 are generally made of hard metal.

(Image processing device)
FIG. 2 is a block diagram illustrating a hardware configuration of the image processing apparatus according to the first embodiment and a functional configuration of the processing unit. FIG. 3 is a schematic diagram showing a state in which the first camera and the second camera are attached to the tip portion of one fork 15.

The image processing apparatus 20 includes a first camera 21, a second camera 22, a processing unit 23, a storage unit 24, and a display 25, and these components are mounted on the forklift 10.

As shown in FIG. 2, each of the first and second cameras 21 and 22 includes an imaging device 200 (first and second imaging devices) having sensitivity in a visible light region such as a CCD or a CMOS, and an optical device such as a lens. A system is provided, and the front of the forklift 10 is photographed to obtain an image (video). At least a part of the imaging areas of the first and second cameras 21 and 22 overlap. In the first embodiment, the imaging devices 200 of both the first and second cameras 21 and 22 detect the distance to the object in cooperation with the processing unit 23 and generate distance measurement point group data. It also functions as a detection sensor.

(Cameras 21, 22)
As shown in FIG. 3, in the first embodiment, of the two forks 15, 16 of the forklift 10, two units for stereoscopic viewing (also referred to as compound eyes) at the tip of one fork 15. The cameras 21 and 22 are attached so that the front of the main body 11 is an imaging region. Further, in the width direction (Y direction) of the fork 15, both the cameras 21 and 22 are separated by a predetermined distance (base line length). In the example shown in the figure, two cameras are attached to the left fork 15, but the present invention is not limited to this, and may be attached to the right fork 16. The cameras 21 and 22 and the processing unit 23 are connected by a cable (not shown) or wirelessly, and a video signal is transmitted to the processing unit 23.

Next, the mounting positions of the cameras 21 and 22 on the fork 15 will be described with reference to FIGS. 4 (a) to 4 (c) and FIG. 4 (a) to 4 (c) are enlarged views of the front end side (also referred to as “claw” or “blade”) of the fork 15, and FIG. 5 is an AA view of FIG. 4 (a). 5 is a cross-sectional view, and in FIG. 5, a contour line on the upper surface s <b> 2 in FIG.

The cameras 21 and 22 are preferably arranged on the side surface or the lower surface in the vicinity of the boundary between the linear portion on the side surface of the fork 15 and the tip protrusion (tip s1 described later) so that a wider angle of view can be obtained. More specifically, the cameras 21 and 22 are preferably arranged in a tapered portion s51 described below.

4A is a side view of the fork 15 to which two cameras 21 and 22 are attached, FIG. 4B is a plan view, and FIG. 4C is a view from the front end side of the fork 15. It is a front view.

The fork 15 has a tip s1, an upper surface s2, a lower surface s3, a side surface s4, and a tapered portion s51 of the tip portion. The tip s1 is a plane extending in the YZ plane. Here, the “tip portion” includes not only the tip s1 but also its peripheral portion. For example, a range of 20 to several centimeters from the tip s1 in the X direction is included (see FIG. 9 described later). Further, the peripheral portion includes a tapered portion s51. The tapered portion s51 has a width that gradually decreases toward the tip s1 in the top view as shown in FIG. 4B, and the bottom surface s3 inclines in the side view as shown in FIG. 4A. And the tapered surface gradually decreases in thickness toward the tip s1. The tip s1 may be formed as a curved surface instead of a flat surface.

A cylindrical hole is provided on each of the left and right sides of the tapered portion s51 of the fork 15, and the cameras 21 and 22 are embedded in the holes, respectively. It is preferable that the cameras 21 and 22 be arranged so that the front surface of the lens slightly protrudes from the outer peripheral surface of the tapered portion s51 in terms of securing a wide angle of view. From the viewpoint of breakage due to, it is more preferable to dispose inside the opening surface of the cylindrical hole.

When the object is viewed in stereo (stereoscopic view) for distance measurement, it is necessary to overlap the shooting areas of the two cameras 21 and 22. In order to secure a wide field of view (angle of view) and allow more regions to overlap, it is preferable to provide the cameras 21 and 22 on the tapered portion s51 of the side surface s4 or the lower surface s3. As shown in FIG. 5, the two cameras 21 and 22 arranged in the tapered portion s <b> 51 can ensure a wide angle of view toward the front side of the fork 15.

6 (a) to 6 (c) are enlarged views of a fork 15 according to another example. The fork 15 shown in FIGS. 6A to 6C has a thickness of 10 mm, a width of 100 mm, and a tapered portion s51 whose width gradually decreases toward the tip s1 when viewed from above. The tapered portion s51 has an R (radius) of 60 mm, and the tapered portion s51 extends from the tip s1 to 40 mm in the X direction and from the side surface s4 to 40 mm in the Y direction. The first and second cameras 21 and 22 are provided in the tapered portion s51.

In the height direction (Z direction), the cameras 21 and 22 are preferably separated from the upper surface s2 and the lower surface s3 by 2 mm or more, respectively. As described above, if the thickness of the fork 15 is 10 mm, it is preferable that the cameras 21 and 22 are both arranged in a size and position so as to be within a range of 2 to 8 mm from the lower surface. In general, in the loading operation, the fork 15 may intentionally come into contact with a floor surface or a load and collide with it. For this reason, this arrangement prevents the cameras 21 and 22 from directly colliding with the floor or luggage. In addition, the front surfaces of the lenses of the cameras 21 and 22 are preferably arranged on the inner side of the front surface, that is, the front surface s1 and the surface of the tapered portion s51 in the top view. With such an arrangement, when the fork collides (intentionally) with another object from the front, the cameras 21 and 22 can be prevented from directly colliding with another object.

(Angle of view)
Hereinafter, the angle of view will be described with reference to FIGS. FIG. 7 is a schematic diagram for explaining the angle of view in the horizontal direction. In FIG. 7, the angle of view will be described using the fork 15 having the shape shown in FIGS. 6 (a) to 6 (c) as an example, but the shape of the shape shown in FIGS. 4 (a) to 4 (c) and FIG. The present invention can be applied to any fork shape such as the fork 15 (the same applies to FIG. 8). The angle of view of the first and second cameras 21 and 22 is ideally 180 degrees in the horizontal direction with the front at the center. However, it is difficult to arrange the cameras 21 and 22 at the tip s1 in consideration of an impact caused by a collision with another object. The minimum value of the horizontal angle of view is 44 degrees or more so that an object with a width of 2 m can be photographed (stereoscopic view) 5 m ahead of the fork 15 (the half angle of view to the inside of both cameras is 22 degrees) The above is preferably ensured. By arranging the cameras 21 and 22 in the tapered portion s51, it is possible to secure a field angle equal to or greater than the minimum value. The reason for the width of 2 m is that in the case of a large forklift, the distance between the two forks 15 and 16 is about 2 m, so that the area (an additional image 402 described later) on which the fork 15 hits can be viewed in stereo at a minimum. It is.

FIG. 8 is a schematic diagram for explaining the angle of view in the vertical direction. In the vertical direction, ideally, it is preferable that a 5 m high rack located 3 m ahead of the forklift can be photographed. In other words, when the fork 15 is located at the side of the ground, the vertical angle of view is 120 degrees (60 degrees with a half angle of view above the horizontal) so that a range of up to 5 m in height 3 m can be photographed. It is preferable.

As in the horizontal direction, the minimum angle of view in the vertical direction is at least 44 degrees so that an object with a height of 2 m can be photographed 5 m ahead of the fork 15 (half angle of view to the inside of both cameras). Of 22 degrees or more) is preferably ensured. 5m forward and 2m high is based on the fact that the overall height of a small forklift that is widely used indoors is 2m. This is because it is possible to confirm that the top of the head does not contact any object in front.

FIG. 9 shows a state in which the fork 15 is inserted from the insertion port of the pallet 91 and the pallet 91 is placed on the fork 15. In the figure, it is assumed that the position at which the fork 15 is inserted is shifted by 50% from the center (indicated by a broken line) of the insertion port on one side of the pallet 91 to one side. In order to ensure a half angle of view of 30 degrees on the horizontal plane (XY plane) even at such a position shifted by 50%, the tip positions of the cameras 21 and 22 should be arranged within 14 cm from the end of the pallet. preferable.

If the standard pallet length is 110 cm and the standard fork length is 122 cm, the amount of protrusion from the end of the pallet to the tip s1 of the fork 15 is 12 cm. Therefore, it is preferable to arrange the cameras 21 and 22 so that the entire surface of the lens is located in the range from the leading edge (tip s1) of the fork 15 to 26 cm (14 + 12 cm) in the X direction.

Thus, in the present embodiment, by arranging the two cameras 21 and 22 on one hard (rigid) fork 15, the relative positions of both cameras are always constant. As a result, the base line length and parallelism between the two cameras 21 and 22 can always be kept constant. When distance measurement is performed using images from the two cameras 21 and 22 described later, this is performed stably with high accuracy. be able to.

Further, by disposing the cameras 21 and 22 on the tapered portion s51 at the tip portion of the fork 15, the angle of view can be increased compared to the case where the cameras 21 and 22 are disposed on the flat lower surface s3 or the side surface s4, and a wider range can be obtained. Can shoot.

Again, the processing unit 23 and the like will be described with reference to FIG. The processing unit 23 includes a CPU (Central Processing Unit) and a memory, and performs various controls of the entire image processing apparatus 20 by the CPU executing a control program stored in the memory. Each function performed by the processing unit 23 will be described later.

The storage unit 24 is a hard disk or a semiconductor memory, and stores a large amount of data. The storage unit 24 stores a three-dimensional distance map, which will be described later, and accumulates or updates the distance map sent from the processing unit 23. The storage unit 24 may be mounted on the forklift 10, but all or part of the storage unit 24 may be provided in an external file server. Providing a part of the storage unit 24 in an external device is particularly useful when using a distance measurement map from an external distance measurement sensor 80 (see FIGS. 8 and 22) described later. Data transmission / reception with an external file server is performed via a LAN by a wireless communication unit included in the image processing apparatus 20.

Also, the three-dimensional distance map may reflect drawing data relating to a building or facility such as a warehouse or a factory in which the forklift 10 is used, that is, a work space where the forklift travels. The drawing data includes, for example, position information on the floor surface, wall surface, window, and lighting device. Further, the three-dimensional distance map may include position information of other vehicles such as forklifts that travel in the building. Further, the location information of the luggage 92 used in the building and acquired from an external logistics system (see FIG. 24 described later) that manages the logistics of the inside luggage may be included.

For example, this logistics system has a server and is connected to the image processing apparatus 20 via a network, for example. The server stores package position information in the building, package content information, empty shelf information indicating shelf availability, cargo handling procedure information indicating procedures for cargo handling, and the like. For example, an IC tag is attached to each load 92 or a pallet 91 on which the load 92 is placed, and the physical distribution system can grasp position information of each load 92 using the IC tag. Note that the position information of other vehicles (including forklifts) operating in the building may be grasped and acquired by a signal from an external distance measuring sensor 80, or the image processing apparatus 20 may acquire other vehicles. It may be obtained directly by performing P2P (peer-to-peer) communication with a communication unit mounted on the PC.

The display 25 is attached to a frame that supports the head guard 17 in front of the driver as shown in FIG. 1, and displays a processed image generated and processed by the processing unit 23 as described below. The processed image is, for example, an image obtained by performing viewpoint conversion processing of images acquired by the cameras 21 and 22 and / or processing processing for adding an image of a distance value. The processing for adding the image of the distance value includes a process of superimposing an additional image corresponding to the type of the recognized object or the distance and position to the object on the video. Accordingly, the driver can check the status of the front of the luggage on the display screen of the display 25 even when the forward visibility is deteriorated due to the luggage loaded on the forks 15 and 16. The display 25 is a liquid crystal display, for example. The display 25 may be a HUD (head-up display) or a head-mounted display worn by the driver. A display for HUD includes a combiner which is a concave mirror or a plane mirror having semi-transparency, and projects a virtual image on the combiner. As the virtual image, there is an additional image generated by the processing unit 23 described later. A driver sitting on the driver's cab 12 can visually recognize the real image ahead through the combiner and simultaneously recognize the virtual image reflected by the combiner. By using the HUD, even if the combiner is arranged on the front side of the driver's seat, the combiner becomes transparent when the image is not projected, so that the forward view is not hindered.

(Processing unit 23)
The processing unit 23 includes an image acquisition unit 301, a preprocessing unit 302, a feature point extraction unit 303, a distance map generation unit 304, an object position determination unit 305, an additional image generation unit 306, an association unit 307, a viewpoint conversion unit 308, an image It functions as a composition unit 309 and an image output unit 310. These functions are performed by the processing unit 23 executing a program stored in the internal memory, but some of these functions may be performed by a built-in dedicated hardware circuit. In the embodiment described below, the processing unit 23 generates a distance map that can grasp the position and distance of the object. However, the present invention is not limited to this, and only the distance measurement to the object may be performed. .

(Image acquisition unit 301)
The image acquisition unit 301 performs control such as synchronizing the two cameras 21 and 22 by applying a timing trigger, and acquires images (videos) photographed at a predetermined frame rate by these imaging elements 200. Here, regarding the control of the cameras 21 and 22, the image acquisition unit 301 may perform exposure using the central portion at the shooting angle of view. This is because the image from the camera is bright only at the center and the periphery is dark until the fork is inserted into the insertion port of the pallet 91 and the fork tip passes through the insertion port. In other words, in order to take a picture of the space that has passed through the opposite side of the insertion port from the side where the fork is inserted, exposure is performed using the central portion of the angle of view. As a result, an image from the central portion can be appropriately captured without being overexposed.

(Pre-processing unit 302)
The preprocessing unit 302 adjusts the brightness and contrast of a set of images acquired from the two cameras 21 and 22 via the image acquisition unit 301. A known adjustment process can be applied to these adjustments. Further, post-processing such as binarization processing is further performed on the adjusted image, and the processed image is supplied to the feature point extraction unit 303. On the other hand, the preprocessing unit 302 supplies the color image that has been subjected to the preprocessing in the previous stage to the viewpoint conversion unit 308. It should be noted that stitch processing for pasting a set of images in a positional relationship according to the base line length may be performed, and the processed image may be supplied to the viewpoint conversion unit 308. May be supplied to the viewpoint conversion unit 308.

(Feature point extraction unit 303)
The feature point extraction unit 303 extracts feature points corresponding to the shape and contour of an object that is an index of association from each set of images. Note that color, contrast, edge, and / or frame information may be used as the association index.

(Distance map generator 304)
The distance map generation unit 304 extracts common corresponding points from the extracted feature points of a set of images, and calculates distances from the corresponding points to the feature points using the conversion parameters. For example, in the pair of cameras 21 and 22 arranged on the left and right, the distance value of each pixel is calculated according to the shift amount of the left and right pixel values of the same corresponding point using the baseline length (ranging).

Further, the distance map generation unit 304 may perform SLAM (Simultaneous localization and mapping) processing. As software for executing SLAM processing, there is SDK software for ZED cameras. In addition, as an open source for creating SLAM, there are RGB-D SLAMV2, etc., and these may be used. By performing the SLAM processing, the moving position of the own vehicle, that is, the forklift 10 in the three-dimensional distance map (hereinafter simply referred to as “distance map”) of the forklift 10 can be grasped in real time. In the SLAM process, a distance map in the work space where the forklift 10 stored in the storage unit 24 is used may be used. As a result, it is possible to grasp the situation in an area outside the imaging area (view angle range) of the cameras 21 and 22. In addition, a set of images taken by the cameras 21 and 22 is different in the obtained images due to a phenomenon such as halo, ghost, flare, light glare, reflection of an external light source (sunlight, etc.), etc. In the case where distance measurement cannot be performed in some pixel areas, the distance sensor 80 is generated by past driving of an external distance measuring sensor 80 (see FIG. 8 and FIG. 22 described later) or the forklift 10 described later and stored in the storage unit 24. You may correct | amend using the accumulated distance map. This situation in which distance measurement cannot be performed occurs, for example, when external light is illuminated from a factory or warehouse window. As this correction, for example, there is a process of replacing an area in which a distance value could not be detected in the imaging area with data at a corresponding position in the past distance map.

(Object position determination unit 305)
The object position determination unit 305 determines the position of the object in front of the forklift 10 in the three-dimensional space. This object determination may be performed, for example, by clustering pixels according to the similarity of the distance values of each pixel. Further, the similarity of distance values may be combined with the similarity of pixel colors for clustering. The size of each cluster determined by clustering is calculated. For example, the vertical dimension, horizontal dimension, total area, etc. are calculated. The “size” here is an actual size, and unlike the apparent size (field angle, that is, the spread of pixels), the cluster of pixels is determined according to the distance to the object. For example, the object position determination unit 305 determines whether or not the calculated size is equal to or smaller than a predetermined size threshold value for specifying the analysis target object to be extracted. The size threshold can be arbitrarily set depending on the measurement location, the behavior analysis target, and the like. If the behavior is analyzed by tracking a worker who passes, the minimum value of the size of a normal person may be set as a size threshold for clustering. Further, if the environment in which the forklift travels is limited in a specific warehouse or the like, a size threshold corresponding to an object existing in the environment may be applied. The object position determination unit 305 accumulates the generated three-dimensional distance map ahead of the forklift in the storage unit 24. This distance map includes information on the size and position of each object having a predetermined size or larger. As described above, only the distance measurement to the object may be performed to reduce the processing load.

In addition, as for the type of the object, it is possible to use machine learning or proportion discrimination (aspect ratio) for discrimination between a human and another object. In machine learning, learning is repeated using data of a three-dimensional distance map obtained by a computer. Particularly in the work space, it is important to discriminate the worker in order to prevent a serious accident, and the use of these is expected to improve the discrimination accuracy.

(Additional image generation unit 306)
The additional image generation unit 306 adds the distance according to the distance to the object ahead of the forklift 10 determined by the object position determination unit 305, more specifically, the distance to the object ahead of which the forks 15 and 16 are extended. An image (also referred to as an annotation image) is generated. Examples of the additional image include a distance ladder indicating a distance and a numerical value display (see FIG. 10 described later). In addition, the additional image may be a mark or text for alerting the driver when there is a rectangular frame whose color or form has been changed according to the type and distance of the object, or an object requiring attention. Good. Further, the additional image may be information on an inclination angle, an opening angle, an inclination angle, or a height from the ground on the horizontal plane or the XY plane in front of the fork with respect to the pallet.

Furthermore, it is also possible to acquire package position information, package content information, empty shelf information indicating shelf availability, cargo handling procedure information indicating a procedure for cargo handling, etc. from the logistics system and generate these information as additional images. Good (see FIG. 25 described later).

As an additional image, when the luggage is further lifted, the distance in the height direction between the forks 15 and 16 and the truck bed, the distance between the mast 13 and the ceiling of the building or facility, the upper end on the front side of the luggage 92, the ceiling, An additional image corresponding to the distance in the height direction to the canopy of the truck or the upper stage of the shelf may be generated. Of these, the additional image corresponding to the relative positional relationship with the tips of the forks 15 and 16 is the posture information of the cameras 21 and 22 on the forks 15, that is, the direction (opening angle), height, and the like of the forks 15. The angle is changed according to the change of the tilt angle (tilt) and the interval between the forks 15 and 16. The direction, height, inclination angle, and distance between the forks of these forks 15 may be detected from the captured images of the cameras 21 and 22 and detected by various sensors attached to the main body 11 of the forklift 10. Also good.

Further, an additional image corresponding to the inclination amount of the pallet 91 may be generated. Specifically, the object position determination unit 305 recognizes the shape of the insertion port of the pallet 91 ahead, and inclines according to the comparison with the shape and size of the insertion port registered in the storage unit 24 in advance. Determine the amount. Then, the additional image generation unit 306 generates an additional image corresponding to the tilt amount. The tilt amount is, for example, a tilt angle (tilt) with respect to a horizontal plane that occurs when the upper surface of the load 92 is not horizontal when the pallet 91 and the load 92 are stacked two or more levels as shown in FIG. Further, this inclination angle may be a relative angle with respect to the fork 15. Further, it may be a relative inclination angle (yaw angle) between the virtual extension line of the fork 15 and the pallet 91 in the horizontal plane (XY plane). Further, when these inclination amounts are equal to or larger than a predetermined value, a warning may be given.

(Association unit 307)
The association unit 307 associates the position of each object in the two-dimensional image captured by the cameras 21 and 22 with the position of each object in the distance map.

(Viewpoint conversion unit 308)
The viewpoint conversion unit 308 is provided for each angle and direction when viewed from a viewpoint position (virtual viewpoint position) designated in advance according to the viewpoint position corresponding to the eye height of the driver sitting on the cab 12. The viewpoint conversion of the image is performed by performing coordinate conversion of the interval and position of the pixel points. The driver can set the viewpoint position (virtual viewpoint position) using an input device (not shown) such as a keyboard, a pointing device, or a touch sensor of the image processing apparatus 20. At this time, display processing is performed for a blind spot area that cannot be captured from the camera. For example, the blind spot area (data NULL area) is replaced with a pixel in the storage unit 24 corresponding to the area. The blind spot area occurs when the position of the camera is different from the viewpoint position of the driver. Further, this viewpoint conversion is performed according to the distance to the object in the image using a conversion by trapezoidal correction of the image according to the tilt angle or height of the camera 21 or 22 with respect to the floor surface and a three-dimensional distance map. The case where the conversion is performed is included. For example, when the distance map is not sufficiently generated over the photographing region, the trapezoidal correction may be performed.

Further, the viewpoint conversion unit 308 determines an inclination angle or height as posture information of the cameras 21 and 22 themselves provided on the ground or the fork 15 with respect to the main body 11 from the images acquired from the cameras 21 and 22, and the determination May be reflected in the viewpoint conversion. For example, when the fork 15 is tilted upward toward the tip or moved upward, the viewpoint conversion is performed so as to cancel the tilt angle and the upward movement amount. The inclination angle or height of the fork 15 with respect to the ground surface of the fork 15 or the main body 11 may be detected by various sensors (see modifications described later) attached to the forklift 10. Further, the posture information of the fork 15 and the cameras 21 and 22 may be detected by various sensors attached to the forklift 10 (see a modified example of FIG. 22 described later).

(Modification of viewpoint conversion)
As the virtual viewpoint position, instead of the viewpoint conversion process in which the driver's viewpoint position is the virtual viewpoint position, a viewpoint conversion process in which a higher position is set as the virtual viewpoint position, or a position away from the main body of the forklift 10 is used. A viewpoint conversion process using the virtual viewpoint position may be performed. For example, a viewpoint conversion process in which the height position of the forks 15 and 16 or a height position obtained by adding a distance corresponding to the height of the pallet 91 and the luggage 92 above the fork 15 or the virtual viewpoint position or the forklift 10 Is a viewpoint conversion process for setting a virtual viewpoint position (overhead view) from above or a virtual position viewpoint (third-person viewpoint position) from the back to the front. Thereby, an overhead image can be obtained. From the bird's-eye view image, for example, when the luggage 92 is lowered to a shelf higher than the driver's eyes, the front of the forklift 10 can be easily confirmed. The change of the virtual viewpoint position and the setting of the position can be appropriately set from an input device provided in the forklift 10.

In addition, in the viewpoint conversion processing in which these high positions or positions away from the main body are used as the virtual viewpoint position, in addition to the distance map generated by photographing with the first and second cameras 21 and 22, an external distance measuring sensor It is preferable to use the distance map acquired by 80 and accumulated in the storage unit 24. The distance measuring sensor 80 is, for example, a laser rider (Laser Lidar (Light Detection And Ranging)), and is provided on the ceiling of a building or facility where the forklift 10 is used as shown in FIG.

Further, in these viewpoint conversion processes, the presence / absence of viewpoint conversion or the viewpoint conversion amount (intensity) may be changed according to the distance from the forklift 10 to the object. Specifically, for an object having a distance value greater than or equal to a predetermined value, the viewpoint conversion process may be turned off, the viewpoint conversion amount (viewpoint movement distance) may be reduced, or the object may be processed by two-dimensional trapezoidal correction. In addition, the two-dimensional video used for the viewpoint conversion may be obtained by extracting an image from only an overlapping area of the two cameras 21 and 22 and performing viewpoint conversion processing based on the image. Alternatively, the stitch processing for pasting may be performed based on the positional relationship, and the processed image may be used. In that case, viewpoint conversion processing may be performed on a central overlapping area using a three-dimensional distance map, and viewpoint conversion processing may be performed on pixels in a non-overlapping area by two-dimensional trapezoidal correction.

In addition, when the viewpoint conversion process is performed, if the blind spot area increases, the viewpoint conversion amount may be reduced. For example, when a viewpoint conversion process for upward viewing is performed on a photographed image, if the ratio of the number of pixels serving as a blind spot area is greater than or equal to a predetermined value with respect to the total number of displayed images, the virtual viewpoint position is set. Limit to lower than the set position or switch to two-dimensional trapezoidal correction.

(Image composition unit 309)
The image composition unit 309 superimposes the additional image generated by the additional image generation unit 306 on the image generated by the viewpoint conversion unit 308 at a display position corresponding to the position of the object detected by the object position determination unit 305. Generate a composite image.

When a HUD or a transparent screen (transparent screen) is used as the display 25, the display position and content of the additional image are displayed in different directions and positions so as to be superimposed on the object (real image) that the driver is looking at. Calculate and generate. Further, when the HUD is configured to change the virtual image distance, the additional image may be generated with a virtual image distance corresponding to the position and direction of the object.

(Image output unit 310)
The image output unit 310 outputs the combined image generated by the image combining unit 309, that is, the processed image (video) to the display 25 in real time and displays it to the driver. In addition, the image output unit 310 outputs the processed image generated by the image composition unit 309, that is, the image (video) after the process of superimposing the viewpoint conversion process and / or the additional image, to the display 25 in real time. Display to the driver.

FIG. 10 is an example of a screen 250 displayed on the display 25. In front of the forklift 10, there are a truck 95 as an object, a pallet 91 and a luggage 92 thereon, and an operator 96, and the camera 21 in a situation where the forklift 10 is brought close to the loading platform of the truck 95, The image of the front of the forklift 10 taken by 22 is displayed. As shown in the figure, on the screen 250, additional images 401 to 406 are superimposed.

The additional image 401 is an illustration image (animation image) corresponding to the forks 15 and 16. The additional images 402 and 403 are images showing a line in which the forks 15 and 16 are extended forward and the vicinity of the contact position. Accordingly, the driver can recognize the position where the forks 15 and 16 are hit (inserted). The additional images 404 and 405 are images indicating the distance to the object in front of the forklift 10. The additional images 404 and 405 are also referred to as a distance ladder together with the additional image 403. The additional image 406 shows the distance of the forks 15 and 16 to the upper surface of the loading platform of the truck 95 in the height direction. The additional image 407 is a mark that alerts the driver when a person (worker 96) approaches the front of the forklift 10. When a person approaches within a predetermined range and makes an approach prediction, a warning sound is emitted from a speaker attached to the side of the display 25 as a notification process. At this time, the additional image 407 may be changed in color or blinked to indicate a warning.

The additional images 401 to 406 correspond to the change in the distance and the inclination angle if the distance between the pair of forks 15 and 16 and the inclination angle of the forks 15 and 16 with respect to the ground can be changed. The shape, size, and orientation may be changed. For example, when the fork 15 is tilted upward, the additional images 403 to 406 are changed according to the tilt angle. The interval distance and the inclination angle may be obtained by the processing unit 23 based on images acquired from the cameras 21 and 22, or may be detected by various sensors attached to the forklift 10 (see modifications described later). Also good.

As described above, in the present embodiment, a pair of cameras 21 and 22 provided on one fork 15 is used, and the image is acquired in front of the forklift 10 based on the images acquired by the imaging elements of the cameras 21 and 22. In addition to detecting the distance to the object, processing is performed by superimposing an additional image on the acquired video or changing the viewpoint, and the processed video is displayed on the display 25. By providing a pair of cameras 21 and 22 on one fork 15, stable ranging can be achieved. Further, by displaying the processed image, the driver can confirm the front on the screen displayed on the display 25 even when the front is difficult to see due to loading on the forks 15 and 16. Also, by adding an additional image related to the distance to the object, work support, safety warning, etc. can be performed, so that the forklift can be operated more safely.

(Second Embodiment)
FIG. 11 is a block diagram illustrating a hardware configuration of an image processing apparatus according to the second embodiment and a functional configuration of a processing unit. In the image processing apparatus according to the above-described first embodiment (FIG. 2 and the like), the front of the forklift 10 is photographed and the distance is measured using the two cameras 21 and 22. On the other hand, in the second embodiment, forward shooting and distance measurement are performed using one camera 21 and a distance measuring sensor 22b. In the second embodiment, the distance measuring sensor 22b functions as a “detection sensor” for detecting the distance.

The distance measuring sensor 22b is, for example, an area distance measuring device such as a laser lidar (Light Lidar (Light Detection And Ranging)), a TOF (Time Of Flight) scanner, or a sonar. Visible light, infrared light, sound waves, and the like are emitted from the output unit by the area range finder, and the distance to the object ahead is measured by the time difference until the energy reflected from the object reaches the input unit. The distance to the object in front of the forklift 10 is measured by the area range finder, the distance measurement point group data of a plurality of points is measured, and the distance measurement point group data indicating the distribution of distance values is acquired.

For example, when a laser lidar is used as the distance measuring sensor 22b, the emitted pulsed laser is irradiated while scanning the front measurement space, and the reflected light is detected. Then, distance information at each irradiation position is obtained according to the time difference between the emission timing and the light reception timing, and distance measurement point group data in the measurement space is acquired.

The acquired ranging point group data is sequentially sent to the distance map generation unit 304 and used for various processes. The distance measurement point group data is stored in the storage unit 24 as a distance map.

Further, as in the first embodiment, the distance measuring sensor 22b and the camera 21 are disposed at the tip of one fork 15. The measurement space of the distance measuring sensor 22b and the shooting area of the camera 21 are arranged so that part or all of them overlap. In addition, it is preferable that the distance measuring sensor 22b and the camera 21 are arranged in the tapered portion s51 at the tip portion of the fork 15 as in the first embodiment (see FIGS. 4 to 9).

As described above, also in the second embodiment using the distance measuring sensor and the camera, the same effect as in the first embodiment can be obtained. Also for the second embodiment, the position detection sensor 26 may be provided, or the distance map acquired by the external distance measuring sensor 80 and stored in the storage unit 24 may be used.

(Third embodiment)
FIG. 12 is a block diagram illustrating a hardware configuration of the image processing apparatus 20 according to the third embodiment. In the third embodiment, one camera 21 and a projector 22c are provided. In the present embodiment, the projector 22c and the processing unit 23 described below function as a detection sensor for detecting the distance to the object in front of the forklift by cooperating with the camera 21.

In response to the control signal of the processing unit 23, the projector 22c projects pulsed pattern light toward the front of the forklift. The pattern light is, for example, a plurality of vertical and horizontal line lights or lattice pattern light composed of dot light at predetermined vertical and horizontal intervals. Further, a random dot pattern may be projected as the pattern light. This pattern light irradiation region overlaps part or all of the photographing region of the camera 21. In addition, instead of the pattern light, one point of irradiation light may be configured to sequentially scan the imaging region of the camera 21.

Similarly to the first embodiment, the projector 22c and the camera 21 are arranged at the tip portion of one fork 15. In the width direction (Y direction) of the fork 15, the projector 22c and the camera 21 are separated by a predetermined distance (base line). (Long) is separated and attached. In addition, it is preferable that the projector 22c and the camera 21 are disposed in the tapered portion s51 at the tip portion of the fork 15 as in the first embodiment (see FIGS. 4 to 6).

When the processing unit 23 (distance map generation unit 304) irradiates pattern light at a predetermined timing, the captured camera 21 acquires the interval or position of each dot light or line light constituting the pattern light. Detect by image. Using the conversion parameter based on the position of the detected pattern light and the base line length, the distance in the plurality of pixels of the acquired image is calculated.

Then, based on the obtained distance value, the distance to the object ahead of the forklift is detected, and the viewpoint is converted, the additional image is superimposed, or the viewpoint is converted to the acquired video. The processed image is displayed on the display 25.

As described above, also in the third embodiment, by using the projector 22c and the camera 21 (imaging device), as in the first embodiment, the front image of the forklift 10 and the distance to the front object can be obtained. Detect and display the processed video on the display. In addition, the processed video that adds the image of the distance value based on the acquired distance measurement point group data to the video acquired by the camera is displayed on the display. Thereby, the effect similar to 1st Embodiment can be acquired.

In addition, in the example of FIG. 12, although the example using one camera 21 was shown, it is not restricted to this, You may use the two cameras 21 and 22 arrange | positioned at one fork. That is, the projector 22c is further added to the configuration of the first embodiment (FIG. 2 and the like). In this case, for example, the projector 22 c is arranged at the same position as the camera 22 in the width direction of the fork 15, and the distance of each pixel is calculated from the image acquired by the camera 21 that is separated from the base line length. As described above, the distance measurement using two cameras and the distance measurement using the projector 22c can be used together to perform the distance measurement with higher accuracy.

(First modification)
FIG. 13 is a block diagram illustrating a hardware configuration and a functional configuration of the processing unit according to the first modification.

In the first modification, a position detection sensor 26 that acquires the position state of the fork 15 provided with two cameras 21 and 22 is further provided in the configuration of the first embodiment.

The position detection sensor 26 may be, for example, a sensor that detects the inclination angle (tilt) of the fork 15. Further, the position detection sensor 26 may be a sensor that detects the height of the fork 15 relative to the mast 13, that is, the height relative to the ground. These sensors are composed of an actuator and an optical element, for example. With these sensors, the relative positions of the cameras 21 and 22 provided on the fork 15 with respect to the main body 11 can be detected. Further, the position detection sensor 26 may be an acceleration sensor or a gyro sensor. Angular velocity information and turning angular velocity information can be acquired by the acceleration sensor and the gyro sensor, and the relative positions of the cameras 21 and 22 provided on the fork 15 with respect to the main body 11 or the surroundings can be grasped. The relative position mentioned here includes grasping the angle (horizontal or inclined) of the fork 15 (cameras 21 and 22) and the horizontal plane.

Even in such a modification, the same effect as that of the first embodiment can be obtained, and the processing load of the processing unit 23 that superimposes the additional image or converts the viewpoint by providing the position detection sensor 26. Can be reduced. In the modification, the image processing apparatus 20 applies the position detection sensor 26 to the configuration of the first embodiment. However, the present invention is not limited to this, and the position detection is performed in the second and third embodiments. The sensor 26 may be applied.

(Second modification)
FIGS. 14A and 14B are schematic views showing a state in which the first camera and the second camera are attached to the tip of one sheath fork 19 in the second modification. When the sheath fork 19 is attached to the fork 15 (or the fork 16) on the distal end side as in this embodiment, the distal end portion of the fork or the distal end of the fork is the distal end portion of the sheath fork 19 or the sheath fork. 19 points to the tip. The sheath fork 19 is mounted so as to cover the fork 15 and is fixed by a holder (not shown) such as a screw.

FIG. 14A is a side view showing a sheath fork 19 attached to the front end side of the fork 15, and FIG. 14B is a plan view.

The sheath fork 19 has a tip s11, an upper surface s12, a lower surface s13, a side surface s14, and a tapered portion s151 of the tip portion. Each of the left and right tapered portions s151 of the fork 19 is provided with a cylindrical hole, and the cameras 21 and 22 are respectively embedded in the holes. The arrangement positions of the cameras 21 and 22 and the effects thereof are the same as those in the first embodiment described with reference to FIGS.

The cameras 21 and 22 are connected to the main body 11 of the forklift 10 by a cable or wirelessly, and transmission of the video signal to the processing unit 23 is performed. In the case of cable connection, power is supplied from the main body 11, and in the case of wireless connection, power is supplied by a battery attached together with the cameras 21 and 22.

As described above, by providing the first and second cameras 21 and 22 on the sheath fork 19 as in the second modification, the same effect as that of the first embodiment can be obtained. Further, in the second and third embodiments, a sheath fork 19 may be used instead of the fork 15. That is, the camera 21 and the distance measuring sensor 22 b or the projector 22 c are provided at the tip of the sheath 19. Even if it does in this way, the effect similar to 2nd, 3rd embodiment can be acquired.

(Third Modification)
FIG. 15 is a block diagram illustrating a hardware configuration and a functional configuration of a processing unit according to the third modification. In the third modification example, one (monocular) camera 21 and a position detection sensor 27 are provided. In the third modification, the processing unit 23 includes an optical flow processing unit 320. In this modification, the image sensor 200, the position detection sensor 27, and the processing unit 23 (optical flow processing unit 320) of the camera 21 function as a detection sensor for detecting the distance to the object.

The position detection sensor 27 has the same configuration as the position detection sensor 26 according to the modification. From the position detection sensor 27, position data relating to the traveling direction and movement amount of the camera 21 when the forklift 10 moves is acquired.

The optical flow processing unit 320 acquires distance measurement information using a known optical flow process. Specifically, a difference between a plurality of frames (a plurality of images) in time series of the camera 21 is detected, and a distance value to each object in the image is acquired using position data from the position detection sensor 27 at that time. . Even if it does in this way, the effect similar to each above-mentioned embodiment can be acquired.

(Fourth modification)
FIG. 16 is a block diagram illustrating the hardware configuration of the image processing apparatus, the functional configuration of the processing unit, and the configuration of the HUD according to the fourth modification. FIG. 17 is a schematic diagram showing the configuration of the HUD. The fourth modification has a HUD 25b as a display. The combiner 522 of the HUD 25b is located at the same position as the display 25 in FIG. 1 and can pass through the front side of the forklift. The driver can see the front real image through the combiner while projecting an image such as the additional images 401 to 407 shown in FIG. 10 as a virtual image by the HUD 25b. In addition, you may apply HUD25b in this 4th modification to the 1st-3rd embodiment mentioned above and each modification.

As shown in FIG. 17, the HUD 25b enlarges a display element 51 having a two-dimensional display surface, an image i formed on the display element 51, converts it into a virtual image f, and projects the mirror 521 and a combiner 522. Including a virtual image projection optical system 52 and a moving mechanism 53. Note that the mirror 521 may be omitted. The display element 51 may be a liquid crystal, an OLED (Organic Light Emitting Diode), or an intermediate screen. The light reflected by the combiner 522 is guided to the driver's pupil 910 and recognized as a virtual image f. The virtual image projection optical system 52 includes a mirror 521 and a combiner 522, enlarges the image i formed on the display element 51, converts the image i into a virtual image f, and projects the image onto the combiner 522. The moving mechanism 53 moves the mirror 521 or the display element 51 to adjust the sitting height in accordance with the sitting height of the driver.

The virtual image on the HUD25b needs to work by alternately looking at the luggage, luggage rack, and fork toe. For setting the virtual image distance, it is easy to see the fork toe, luggage rack, and luggage. Something is desirable. With the display light from the display element 51 guided to the driver's pupil 910 sitting in the driver's seat, the driver can observe the virtual image f as a display image as if it is in front of the body of the forklift. The distance to the observed virtual image f is the virtual image distance, and corresponds to the distance between the pupil 910 and the virtual image f in FIG. Because the distance to the object to be visually recognized and the virtual image distance are short, the object and the virtual image can be seen with almost no change in the focal distance of the eyes, so the information on the virtual image understands without changing the focus of the eye Because it can, recognition time can be shortened. In addition, the degree of fatigue can be reduced by reducing the burden on the eyes. The virtual image distance is preferably set according to the size of the forklift 10 or the distance between the tip of the fork 15 and the driver's seat. For example, in the case of a small forklift of about 1.5 tons, since a virtual image can be observed without removing the viewpoint from the toe, the projection distance is preferably set within a range of 1 m to 3 m.

In addition, HUD is good also as a structure which can change a virtual image projection distance. For example, a moving mechanism for changing the position of the display element 51 in the optical axis AX direction or a moving mechanism for moving the mirror 521 may be provided, thereby changing the projection distance to the virtual image f. For example, the projection distance is changed according to the distance to the luggage 92 or the pallet insertion port 91a. Furthermore, as another embodiment, an intermediate screen may be provided, and the projection distance may be changed by changing the position of the intermediate screen in the optical axis direction. The intermediate screen is a member that projects an image formed on the display surface of the display element on the surface thereof, and is a member having a diffusion function such as ground glass, and is disposed between the display element 51 and the mirror 521.

Further, the HUD is configured as a 3D-HUD so that a virtual image is displayed three-dimensionally at a virtual image distance corresponding to the distance to the actual object (object) such as a front luggage, a load shelf, or an operator. May be. For example, the virtual image distance is changed by moving the elephant (display element 51 or mirror 521) by a moving mechanism at a period of several tens of Hz, and the display control unit forms the display element in accordance with the movement timing of the elephant. Control the image. This makes it appear to the driver that multiple virtual images with different virtual image distances are displayed simultaneously.

(Fifth modification)
FIG. 18 is a schematic diagram illustrating a state in which the first camera and the second cameras 21 and 22 are attached to the tip of one sheath fork 19 in the fifth modification. The fifth modified example uses the sheath fork 19 as in the second modified example, and further has a configuration in consideration of impact relaxation at the time of collision.

FIG. 18 is a side sectional view showing the sheath fork 19 attached to the front end side of the fork 15. The sheath fork 19 includes a main body 191, a lid 192, a transparent plate 193, an impact relaxation member 194, and a heat conduction member 195.

The substrate unit 40 includes an aluminum plate 41, first and second cameras 21 and 22, a camera substrate 42, and an IMU (Internal Measurement Unit) substrate 43 disposed on the aluminum plate 41. On the camera substrate 42, some or all of the functions of the processing unit 23 are arranged. The IMU substrate 43 corresponds to the position detection sensor 27 described above. The lid 192 is fixed to the main body 191 with a bolt 90 or the like, so that the board unit 40 is accommodated inside the space of the main body 191.

The main body 191 and the lid 192 are made of steel. The transparent plate 193 is a member that transmits light, and is made of, for example, polycarbonate. The first and second cameras 21 and 22 photograph the outside through the transparent plate 193. The impact relaxation member 194 is made of an elastic body such as silicon rubber or a gelled material.

The heat conduction member 195 is a member made of a flexible and high heat conductivity material such as aluminum, copper, carbon, or a heat pipe, or a heat pipe, and is attached to the aluminum plate 41 and the main body 191. The heat is transmitted to the main body 191 through the aluminum plate 41 and the heat conducting member 195, and the heat generated from each electronic component of the board unit 40 is radiated. As the heat conductive member 195, for example, a carbon-based graphite sheet (or heat conductive sheet) may be used, or a flexible heat pipe may be used.

The impact relaxation member 194 is affixed on the inner surface of the lid portion 192, and its end portion is also in contact with the main body portion 191. With this impact relaxation member 194, the camera and the electronic components constituting the detection sensor are indirectly connected to the main body 191 through the impact relaxation member 194. Specifically, in the example of FIG. 18, the entire board unit 40 is covered with the impact reducing member 194, and the impact caused by the collision of the object with the sheath fork 19 is reduced, and each electronic component of the board unit 40 is applied. It is transmitted.

Thus, in the fifth modified example, the camera and the detection sensor are protected by the impact relaxation member 194. Thereby, the influence given to these components by the impact to a sheath can be relieved. Furthermore, by using a flexible and flexible heat conducting member 195, vibration and impact on the electronic component can be mitigated and exhaust heat from the electronic component can be removed.

In the fifth modification, the example in which the member for impact relaxation and heat dissipation is applied to the sheath fork 19 is described. However, the present invention is not limited to this, and the configuration shown in FIG. 16 is shown in FIGS. The fork 15 may be applied. Moreover, you may apply to each embodiment, such as the above-mentioned 1st Embodiment.

(Fourth embodiment)
(Image display processing)
FIG. 19 is a flowchart showing display processing performed by the image processing apparatus 20 according to the fourth embodiment. Hereinafter, the display process will be described with reference to FIG.

(Step S101)
First, the processing unit 23 (image acquisition unit 301) of the image processing apparatus 20 controls the first and second cameras 21 and 22 to acquire video captured at a predetermined frame rate.

(Step S102)
Next, the processing unit 23 acquires a distance map by processing two corresponding images acquired from the respective cameras 21 and 22. This processing is performed by the above-described preprocessing unit 302, feature point extraction unit 303, and distance map generation unit 304. That is, after pre-processing the acquired image, a distance map is generated by calculating the distance value of each pixel based on the correspondence between the feature points of the two images using the baseline length.

(Step S103)
Subsequently, the processing unit 23 performs viewpoint conversion processing on the two-dimensional video acquired in step S101 using the generated distance map. This processing is performed by the object position determination unit 305, the association unit 307, and the viewpoint conversion unit 308. That is, the position of the object in front is determined in the three-dimensional space, and each determined object is associated with each pixel of the two-dimensional image. Then, by converting the coordinates and the distance and position of each pixel point with respect to the angle and direction when viewed from the virtual viewpoint position specified in advance, corresponding to the viewpoint position of the driver sitting on the cab 12, etc. Performs image viewpoint conversion processing.

(Step S104)
The processing unit 23 creates an additional image. Specifically, the additional image generation unit 306 generates an additional image corresponding to the distance between the object in front of the forklift 10 and the tips of the forks 15 and 16. Further, the processing unit 23 performs approach prediction. That is, when the fork 15 or the like gets too close to the object and the distance to the object is equal to or less than a predetermined value, it is determined that the reporting condition is satisfied.

(Step S105)
The processing unit 23 performs a processing process for adding (superimposing) the additional image generated in step S104 to the image subjected to the viewpoint conversion process in step S103. Specifically, the image composition unit 309 generates a composite image in which the additional image is superimposed at a display position corresponding to the position of the object. Note that the viewpoint conversion processing in step S103 may be omitted, and the video may be processed only by the processing processing in step S104.

(Step S106)
The image output unit 310 of the processing unit 23 outputs the composite image generated in step S105 to the display 25 in real time and displays it to the driver. For example, in step S106, a screen 250 as shown in FIG. If it is determined in step S104 that the notification condition is satisfied as the approach prediction, the processing unit 23 outputs a warning sound from a speaker provided around the display 25 or displays a predetermined warning display on the display. Or perform a notification process. When a person approaches within a predetermined range and makes an approach prediction, a warning sound is emitted from a speaker attached to the side of the display 25 as a notification process. At this time, the additional image 407 may be changed in color or blinked to indicate a warning.

Further, information on the shortest distance from the tips of the forks 15 and 16 may be output. As this information, for example, the distance value of the shortest distance is superimposed on the front image, the distance information is output by voice, or a warning lamp (see FIG. (Not shown). Further, since not only the position of the pallet 91 but also the spatial information on the top, bottom, left, and right of the pallet 91 (the position of the pallet 91 in the pallet 91) is required, not a single point of distance information but a range having a certain extent The distance information of (camera shooting range) may be displayed. Specifically, for example, distance information for a plurality of grid-like points arranged at predetermined intervals (in the angle of view) on the top, bottom, left, and right of the additional image 402 is displayed. In addition, the distance in the YZ plane is displayed as the relative position to the palette 91.

As described above, in the fourth embodiment, a pair of cameras 21 and 22 provided on the fork 15 is used, and the images up to the object in front of the forklift 10 are obtained based on the images acquired by the imaging elements of the cameras 21 and 22. Distance measurement, distance measurement point group data indicating the distribution of distance values is acquired, and processing of adding an image of distance values based on the acquired distance measurement point group data to the image acquired by the camera is performed The processed video is displayed on the display. By performing the processing for adding the distance value image in this manner, the scenery in front can be grasped as the distance, so that the work can be performed more safely than simply watching the video. Further, the driver can easily confirm the front situation on the screen displayed on the display 25 even when the front is difficult to see due to loading on the forks 15 and 16. In addition, by displaying the viewpoint-converted image on the display 25, the driver can use the processed image that has been subjected to the viewpoint conversion process, even when it is difficult to see the front due to loading on the forks 15 and 16. You can check the front more easily.

(Display modification)
FIG. 20 is a modification of the screen 251 displayed on the display 25. In the modification, an overhead image is added, and the screen 251 includes a front image 2501 and an overhead image 2502 in front view. A front image 2501 is the same as the screen 250 of FIG. 10, and is a reduced display of this. The overhead image 2502 is an image in which the additional image generation unit 306 generates an overhead image with a top viewpoint (upper viewpoint) in accordance with the distance value to the object, and displays the overhead image on the front image 2501 side by side. The overhead image 2502 displays an additional image 409 indicating a distance and an additional image 410 that is an illustration image corresponding to the forks 15 and 16. Usually, the distance value of the front side of the object facing the forklift 10, or the front side, side surface, and upper surface is obtained, but no information on the distance value is obtained for the back side of the object. Therefore, there is no display data on the back side. However, regarding the operation of the forklift 10, it is sufficient if the distance value on the front side can be determined without knowing the distance value on the back side. For the back side, by referring to the distance map stored in the storage unit 24 as shown in the example of FIG. 20, the contour of the object existing in the blind spot area is generated and superimposed. Also good. The broken line in FIG. 20 shows the outline of the object (truck, worker) in the blind spot area. In this way, by displaying the bird's-eye view image 2502, the driver can correctly grasp the surrounding situation and can drive the forklift more safely.

(Processing of blind spot area)
Next, the process of the blind spot area will be described with reference to FIGS. 21 (a) to 21 (d). FIG. 21A is an image (original image) taken by the cameras 21 and 22, and FIG. 21B is an illustration of each object in the image for the purpose of explanation with respect to the image of FIG. It is the schematic diagram which provided the code | symbol. Objects 971 to 975 are rectangular parallelepiped objects, and the object 971 is closest to the cameras 21 and 22, and is disposed in the order of objects 972, 973, 974, and 975 in the following order.

FIG. 21C is an image obtained by performing viewpoint conversion with the viewpoint conversion unit 308 using a position higher than the cameras 21 and 22 as the virtual viewpoint position with respect to the original image in FIG. The back side of the object is a blind spot area that cannot be captured by the cameras 21 and 22. Therefore, in the upward viewpoint conversion, the pixel information on the back side of the object is in a state where the pixel information is missing (Null). FIG. 21D is a schematic diagram in which a reference numeral is given to the image in FIG. In FIG. 21D, blind spot areas b1 to b3 are generated on the back side of each object. Black pixels may be assigned to the blind spot areas b1 to b3, respectively, but the following pixels may be assigned for ease of viewing.

For example, (1) The texture of the surface of the object above the object where the blind spot area is formed and at a distance farther than this object, that is, the color and pattern of the pixels constituting the surface are assigned to the blind spot area. By doing so, even if a blind spot area is generated, it melts into the surface of the object on the far side, so that the discomfort in the display can be alleviated. Alternatively, (2) using the contour information of the object in the three-dimensional distance map stored in the storage unit 24, the contour of the object existing in the blind spot area is generated with respect to the blind spot area and superimposed on the image. Even in this way, the uncomfortable feeling caused by the blind spot area in the display can be alleviated.

(Sixth Modification)
FIG. 22 is a block diagram illustrating a hardware configuration and a functional configuration of the processing unit according to the sixth modification.

In the sixth modification, a position detection sensor 26 that acquires posture information of the two cameras 21 and 22 provided on the fork 15 is further provided in the configuration of the first embodiment. In addition, a distance measurement map based on distance measurement point group data acquired by an external distance measurement sensor 80 (see FIG. 8) installed in a building or facility is stored in the storage unit 24. Here, the posture information includes information on an inclination angle and height with respect to the ground, or a height of the forklift 10 with respect to the main body 11, an opening angle between both forks 15 and 16, and an interval.

For example, the position detection sensor 26 is a sensor that detects an inclination angle (tilt) of the fork 15 with respect to the ground. The position detection sensor 26 is a sensor that detects the height of the fork 15 relative to the mast 13, that is, the height relative to the ground. These sensors are composed of an actuator and an optical element, for example. Moreover, you may include the sensor which detects the opening angle of both the forks 15 and 16, and the space | interval. These sensors are encoders provided in a motor provided in the forklift 10. The position detection sensor 26 can detect the relative positions of the cameras 21 and 22 provided on the fork 15 with respect to the main body 11 by these sensors. Further, the position detection sensor 26 may be an acceleration sensor or a gyro sensor. Angular velocity information and turning angular velocity information can be acquired by the acceleration sensor and the gyro sensor, and the relative positions of the cameras 21 and 22 provided on the fork 15 with respect to the main body 11 or the surroundings can be grasped. The relative position mentioned here includes grasping the angle (horizontal or inclined) of the fork 15 (cameras 21 and 22) and the horizontal plane.

In the sixth modified example, the same effects as those of the first embodiment can be obtained, and the position detection sensor 26 is provided to superimpose an additional image or convert the viewpoint. 23 processing load can be reduced. In addition, by storing the distance measurement map obtained from the external distance measurement sensor 80 in the storage unit 24, the distance measurement accuracy can be further improved when distance measurement is performed on an object in front of the forklift 10. Further, by using this distance measurement map, it is possible to obtain shape information such as the contour of the back side of the object that cannot be taken by the cameras 21 and 22.

(Seventh Modification)
FIG. 23 is an example of the proximity screen displayed on the display in the seventh modified example. In step S104 of FIG. 19 described above, when the distance to the object is equal to or less than a predetermined value, it is determined that the reporting condition is satisfied, and the warning processing such as warning display is performed. In the seventh modified example, when the distance from the tip of the fork 15 to the object is equal to or less than a predetermined distance, for example, 2 m or less, the processing unit 23 replaces the alarm processing with proximity as shown in FIG. Screen 252 is generated, and the display on the display is switched to this proximity screen 252. A proximity screen 252 shown in FIG. 23 shows a state immediately before the tip of the fork 15 is inserted into the insertion port of the pallet 91 on which the luggage 92 is placed.

In this proximity screen 252, unlike the screen 250 of FIG. 10 displayed so far, an object close to the fork 15 is displayed and an additional image indicating the virtual position of the tip of the fork 15 with respect to this object is added. Specifically, in the proximity screen 252, the distance ladder of the additional images 403, 404, and 405 is lost, and instead, an additional image 411 indicating the virtual tip of the fork 15 is displayed. When the fork 15 comes close to the pallet 91, the positional relationship between the insertion port 91a of the pallet 91 and the tip of the fork 15 becomes the greatest concern rather than the distance to the front object. Therefore, switching to the proximity screen 252 is effective. The example of the proximity screen 252 shown in FIG. 23 indicates that the tip of the fork 15 is located slightly to the left of the center of the insertion port 91a, and the driver shifts the tip of the fork 15 slightly to the right. Thus, the fork 15 can be inserted into the center of the insertion port 91a. In the example shown in the figure, since the camera is mounted on only one fork 15, only one virtual position of the tip of the fork is displayed. However, the present invention is not limited to this. Two virtual positions may be displayed. As described above, when the distance from the tip of the fork 15 is equal to or less than the predetermined value, the driver can switch to the proximity screen 252 so that the driver can move the tip of the fork 15 to the insertion port 91a of the pallet 91. It is possible to easily grasp whether it is in such a position.

(Eighth modification)
FIG. 24 is a block diagram illustrating the hardware configuration of the image processing apparatus according to the eighth modification, the functional configuration of the processing unit, and the configuration of the HUD. FIG. 25 is an example of a virtual image related to the package content information, empty shelf information, and cargo handling procedure information displayed on the HUD.

8th modification has HUD25b as a display. The HUD 25b has the same configuration as the HUD 25b shown in FIG. Further, the image processing apparatus 20 in the eighth modified example is connected to an external physical distribution system 60 via a network. Stored are location information of packages in the building, content information of packages, empty shelf information indicating the availability of shelves, cargo handling procedure information indicating procedures for cargo handling, and the like.

FIG. 25 is an example of a virtual image regarding the package content information, empty shelf information, and cargo handling procedure information displayed on the HUD in the eighth modification. The driver can directly see the floor, the shelf, the luggage 92, and the pallet 91, which are real images (objects), through the combiner 522. Further, the virtual images f11 to f13 are projected onto the combiner 522 by the HUD 25b, and the driver can see the virtual image at a predetermined virtual image distance. This virtual image distance is set to a value close to the measured distance to the real image. As long as the virtual image is displayed in a three-dimensional manner, different virtual image distances may be set according to the distance to each target real image. The virtual image f11 is package contents (“product number”, “raw material”), the virtual image f12 is empty shelf information (“space for raw material BB”), and the virtual image f13 is cargo handling procedure information (“loading ranking”). These pieces of information are information acquired from the external logistics system 60. FIG. 25 shows a display example in the HUD, but the present invention is not limited to this, and the display may be applied to the liquid crystal display 25 used in the first embodiment and the like. For example, additional images relating to the package content information, empty shelf information, and cargo handling procedure information corresponding to the virtual images f11 to f13 are added to the video displayed on the liquid crystal display 25 used in the first embodiment and the like.

As described above, in the eighth modification example, since the entity and the virtual image can be seen by using the HUD as the display, the information on the virtual image can be understood without the need to change the focus of the eyes, so that the recognition time can be shortened. I can do it. In addition, the degree of fatigue can be reduced by reducing the burden on the eyes. Moreover, by using the package content information, empty shelf information, and cargo handling procedure information as additional information, and projecting this as a virtual image, the driver can carry out cargo handling work more safely and smoothly without being confused by the work. .

The configuration of the image processing apparatus 20 for a forklift described above is the main configuration in describing the features of the above-described embodiment, and is not limited to the above-described configuration. Various modifications can be made within the scope of the claims. can do. Further, the configuration of a general image processing apparatus is not excluded.

For example, in the present embodiment, the display 25 attached to the forklift 10 is used, but is not limited thereto. In addition to or instead of the display 25, a display may be provided in a management office provided in the work space used by the forklift so that the video signal transmitted by the processing unit 23 by radio or the like is displayed on the display. Good. By doing so, it is possible to supervise the work situation in the management office or to perform an operation to leave a work record.

The means and method for performing various processes in the image processing apparatus according to the above-described embodiments can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc) -ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The program may be provided as a single application software, or may be incorporated in the software of the apparatus as a function of the image processing apparatus.

This application is based on a Japanese patent application (Japanese Patent Application No. 2018-030793) filed on February 23, 2018 and a Japanese patent application (Japanese Patent Application No. 2018-42292) filed on March 8, 2018. The disclosure of which is incorporated by reference in its entirety.

DESCRIPTION OF SYMBOLS 10 Forklift 11 Main body 12 Driver's cab 13 Mast 14 Finger bar 15, 16 Fork 19 Sheath fork 191 Main body part 192 Lid part 193 Transparent plate 194 Impact relaxation material 195 Thermal conduction member 20 Image processing device 21, 22 Camera 23 Processing part 301 Image acquisition Unit 302 preprocessing unit 303 feature point extraction unit 304 distance map generation unit 305 object position determination unit 306 additional image generation unit 307 association unit 308 viewpoint conversion unit 309 image synthesis unit 310 image output unit 24 storage unit 25 display 40 substrate unit 41 Aluminum plate 42 Camera board 43 IMU board 25b HUD
51 Display Element 52 Virtual Image Projection Optical System 522 Combiner 53 Movement Mechanism 91 Palette 401 to 407, 409 to 411 Additional Image

Claims (41)

1. An image processing apparatus used for a forklift,
   A camera that is provided at a front end portion of one fork among a plurality of forks supported to be movable up and down on the front side of the forklift, and shoots the front of the forklift;
   A detection sensor for detecting a distance to an object in front of the forklift, provided at the tip portion of the fork provided with the camera;
   Based on detection information of the detection sensor, a processing unit that processes the video acquired by the camera;
   A display for displaying a processed image processed by the processing unit;
   An image processing apparatus comprising:
2. A first image sensor and a second image sensor are provided on one fork so that at least a part of each imaging region overlaps,
   At least one of the first and second imaging elements functions as a part of the camera, and both the first and second imaging elements function as the detection sensor,
   The processing unit detects a distance to an object in front of the fork based on images acquired from both the first and second imaging elements, and processes the image based on the detected distance. The image processing apparatus according to claim 1.
3. A projector that emits light toward the front of the forklift, or that emits a two-dimensional pattern light toward the front of the forklift,
   With
   The projector and the processing unit also function as the detection sensor, and the processing unit is in front of the forklift based on an image from the camera that has captured the light emitted from the projector or the pattern light. The image processing apparatus according to claim 1, wherein a distance to an object is detected, and the video is processed based on the detected distance.
4. The image processing apparatus according to claim 1, wherein the detection sensor is a distance measurement sensor that detects a distance to an object in front of the forklift and acquires a plurality of distance measurement point group data.
5. A tapered portion at the tip of the fork, the width of which gradually decreases toward the tip when viewed from above and / or the thickness of the taper gradually decreases toward the tip when the bottom surface is inclined when viewed from side. The image processing apparatus according to claim 1, wherein the camera and the detection sensor are provided.
6. The front of the forklift is provided as an imaging area so that the first imaging element and the second imaging element share at least a part of each imaging area with one fork,
   At least one of the first and second imaging elements functions as a part of the camera, and both the first and second imaging elements function as the detection sensor,
   The processing unit detects a distance to an object in front of the forklift based on images acquired from both the first and second imaging elements,
   The tapered portion at the tip of the fork has a width that gradually decreases toward the tip when viewed from above and a thickness that gradually decreases toward the tip when the bottom surface is inclined when viewed from the side. The image processing apparatus according to claim 1, wherein the first and second imaging elements are arranged on both sides.
7. Furthermore, a storage unit is provided,
   The processing unit generates a distance map that is a distance measuring point group indicating a position or a shape of an object in a work space where the forklift operates based on an image from the camera and detection information of the detection sensor. The image processing apparatus according to claim 1, wherein the image processing apparatus accumulates in the storage unit.
8. The processing unit corrects part of the distance data to the object detected based on the detection sensor or the detection information of the detection sensor and the image from the camera with the distance map stored in the storage unit. The image processing apparatus according to claim 7.
9. Furthermore, a position detection sensor for acquiring the position state of the fork is included,
   The image processing device according to claim 1, wherein the processing unit acquires a position state of the fork provided with the camera by the position detection sensor.
10. The processing unit, as the processed video,
    The image processing apparatus according to claim 1, wherein an image obtained by adding additional information corresponding to a distance to a front object to the image acquired by the camera is displayed on the display.
11. The processing unit, as the processed video,
    The image processing apparatus according to claim 1, wherein an image obtained by performing viewpoint conversion on an image acquired by the camera is displayed on the display.
12. The image processing apparatus according to any one of claims 1 to 11, wherein the camera performs exposure using a central portion of a shooting angle of view.
13. The image processing apparatus according to any one of claims 1 to 12, wherein the display is a head-up display that projects a virtual image on a combiner attached to the forklift.
14. The combiner is disposed at a position where the front side of the forklift can be seen through,
    The image processing apparatus according to claim 13, wherein the head-up display has a virtual image projection distance set in a range of 50 cm to 20 m.
15. The image processing according to any one of claims 1 to 14, wherein the camera and the detection sensor provided at a tip portion of the fork are attached to a main body of the fork via an impact reducing member. apparatus.
16. The camera and at least a part of the electronic components constituting the detection sensor are connected to the main body of the fork via a heat conducting member made of a flexible and highly heat conductive material. Item 15. The image processing apparatus according to Item 15.
17. An image processing apparatus used for a forklift,
    A camera for photographing the front of the forklift;
    A distance sensor to measure the distance to an object in front of the forklift, and a distance sensor data indicating a distance value distribution, and a detection sensor;
    A processing unit that performs processing for adding an image of a distance value based on the acquired distance measuring point group data to the video acquired by the camera;
    A display for displaying the processed video processed by the processing unit;
    An image processing apparatus comprising:
18. The image processing apparatus according to claim 17, wherein the camera includes an image sensor having sensitivity in a visible light region.
19. The image processing apparatus according to claim 17 or 18, wherein the camera is installed on a fork supported so as to be movable up and down on the front side of the forklift so as to photograph the front.
20. 20. The image processing apparatus according to claim 19, wherein the camera performs exposure using a central portion of a shooting angle of view.
21. The image processing apparatus according to any one of claims 17 to 20, wherein the processing unit further performs a viewpoint conversion process on the video based on the distance measuring point group data as the processing process.
22. Furthermore, a position detection sensor for acquiring posture information of the camera is provided,
    The image processing apparatus according to claim 21, wherein the processing unit performs the viewpoint conversion process using the posture information acquired from the position detection sensor.
23. Furthermore, a storage unit is provided,
    The image processing apparatus according to claim 21 or 22, wherein the processing unit creates a three-dimensional distance map using distance measurement point group data acquired by the detection sensor and stores the three-dimensional distance map in the storage unit.
24. The three-dimensional distance map stored in the storage unit includes drawing data relating to a building or facility in which the forklift is used, distance measurement point group data obtained from sensors installed in the building, and position information of other vehicles. 24. The image processing apparatus according to claim 23, wherein position information of a package acquired from a physical distribution information system used in the building is reflected.
25. 25. The image processing apparatus according to claim 24, wherein the three-dimensional distance map includes position information of a floor surface, a wall surface, a window, or a lighting device related to the building or equipment.
26. The viewpoint conversion process is a viewpoint conversion process in which the viewpoint position of the driver sitting on the cab of the forklift is a virtual viewpoint position, a viewpoint conversion process in which a position higher than the viewpoint position of the driver is a virtual viewpoint position, or The image processing apparatus according to any one of claims 21 to 25, wherein the image processing apparatus performs viewpoint conversion processing in which a position away from the forklift is a virtual viewpoint position.
27. The viewpoint conversion process in which the driver's viewpoint position is a virtual viewpoint position is the viewpoint conversion process by keystone correction according to the angle or height of the camera with respect to the ground, or the distance measurement point group data, or the storage unit 27. The image processing apparatus according to claim 26, which is a viewpoint conversion process using a three-dimensional distance map stored in.
28. 27. The viewpoint conversion process using a position higher than the viewpoint position of the driver as a virtual viewpoint position is a viewpoint conversion process using the distance measurement point group data or a three-dimensional distance map stored in a storage unit. An image processing apparatus according to 1.
29. In the viewpoint conversion process in which a position higher than the viewpoint position of the driver is a virtual viewpoint position, or in the viewpoint conversion process in which a position away from the forklift is a virtual viewpoint position, with respect to the blind spot area of the camera, 29. The texture of the surface of an object at an angle of view above the object where the blind spot area is formed and at a distance farther than the object is arranged in the blind spot area. Image processing device.
30. In the viewpoint conversion process in which a position higher than the viewpoint position of the driver is a virtual viewpoint position, or in the viewpoint conversion process in which a position away from the forklift is a virtual viewpoint position, the blind spot area of the camera is stored in the storage unit. The image processing apparatus according to claim 26 or 28, wherein an outline of the object existing in the blind spot area is superimposed on the blind spot area using outline information of the object in the stored three-dimensional distance map.
31. The image processing apparatus according to any one of claims 21 to 30, wherein the viewpoint conversion process changes presence or absence of a viewpoint conversion process or intensity according to a distance to an object.
32. The display is a transparent screen or a head-up display attached to the forklift so that the front of the forklift can be seen through.
    The processing unit recognizes an object in front of the forklift, generates an additional image corresponding to the distance and / or direction to each of the recognized objects, and superimposes the generated additional image on each of the objects. The image processing apparatus according to any one of claims 17 to 20, wherein the image is displayed on the transparent screen or the head-up display.
33. The processing unit recognizes an object in front of the forklift, and generates an additional image corresponding to the recognized type of the object, the distance to the object, or the position in the video as the processing. The image processing apparatus according to any one of claims 17 to 32, which is added to the video.
34. When the processing unit recognizes a pallet as the object, the processing unit determines an inclination with respect to the pallet based on a shape of the insertion port of the pallet, and generates the additional image according to the determined inclination amount of the horizontal surface of the pallet. 34. The image processing apparatus according to claim 33.
35. In the additional image generated by the processing unit, the contents information of the package acquired from the physical distribution information system used in the building where the forklift is used, the empty shelf information indicating the availability of the shelf, the cargo handling procedure indicating the procedure for cargo handling. The image processing apparatus according to claim 32, wherein at least one piece of information is included.
36. The image according to any one of claims 17 to 35, wherein the processing unit generates an overhead image of an upper viewpoint according to a distance to the object, adds the generated overhead image, and displays the image on the display. Processing equipment.
37. The processing unit recognizes an object in front of the forklift, and issues a warning when the distance from the forklift or the fork tip of the forklift is a predetermined value or less, or closes the display on the display The image processing device according to any one of claims 17 to 36, wherein the image processing device is switched to a display screen.
38. The processing unit according to any one of claims 17 to 37, wherein the processing unit recognizes an object in front of the forklift and outputs information on the shortest distance from the fork tip of the object recognized in the image of the distance value. An image processing apparatus according to 1.
39. An image processing apparatus used for a forklift, which measures a distance between a camera for photographing the front of the forklift and an object in front of the forklift, and acquires distance measurement point group data indicating a distribution of distance values A control program executed by a computer for controlling an image processing apparatus comprising:
    Acquiring an image by the camera (a);
    A step (b) of obtaining ranging point cloud data with the detection sensor;
    A step (c) of performing processing for adding an image of a distance value based on the acquired distance measuring point group data to the video acquired by the camera;
    Displaying the processed image on a display (d);
    A control program for causing the computer to execute a process including:
40. 40. The control program according to claim 39, wherein in the step (c), as the processing, a viewpoint conversion process is further performed on the video based on the distance measuring point group data.
41. The process further includes
    Recognizing an object in front of the forklift (e),
    40. In the step (c), as the processing, an additional image corresponding to the recognized type of the object, the distance to the object, or the position is generated in the video and added to the video. The control program according to claim 40.
    

Images