JP2008143701A - View improving system and method for industrial vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a view improving system and method capable of improving the front view of an industrial vehicle, and capable of improving operability and safety. <P>SOLUTION: The system for improving the view of the industrial vehicle having a shield in front of a seat is provided with a head mount display 11 worn on an operator 5 of the industrial vehicle and transmitting the outside scene and projecting an image, a camera A10 fixed ahead of the shield and taking the image of the front scene, a camera B12 fixed on the head mount display and taking the image of the front scene, and an image processing device 13. The image processing device 13 is provided with an image deformation part 15 for deforming the taken image A of the camera A based on the taken image B of the camera B, an image cut-out part 16 for cutting out a region corresponding to the shield from the deformed image, and an image output part 17 for outputting the cut partial image to the head mount display 11. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an industrial vehicle visibility improvement system and method for improving visibility and operability in an industrial vehicle such as a forklift having a luggage lifting mast in front.

  Among industrial vehicles used in factory premises, warehouses, etc., there are vehicles equipped with shielding objects that obstruct forward visibility. For example, as shown in FIG. 7, the forklift 50 includes a seat 51 on which an operator 60 sits, a claw 52 that is provided in front of the seat 51 and places a load 61, and a load lifting mast 53 that lifts and lowers the claw 52. And an operation unit (not shown) for controlling the lifting and lowering of the luggage 61, and the mast 53 blocks the forward view of the operator 60. In the case of an industrial vehicle having such a structure, there is a problem that the visibility of the operator is poor and the safety and operability are poor.

  Various methods for improving the field of view of a vehicle have been conventionally proposed as described in Japanese Patent Application Laid-Open No. 2006-165946. Patent Document 1 discloses a vehicle equipped with an imaging unit that can capture the front and rear sides of a vehicle, a display monitor that a driver can visually recognize, and an image processing unit that performs image processing on the captured image and displays the captured image on the display monitor. A visibility assist device is disclosed. As described above, by capturing and displaying a part that is a blind spot for the vehicle driver, it is possible to obtain excellent visibility and improve safety.

  Furthermore, Patent Document 2 (Japanese Patent Application Laid-Open No. 2001-344597) discloses an apparatus mainly for improving the visibility of a transportation facility such as an aircraft. This is a configuration in which a stereoscopic image including an obstacle is acquired from a pair of left and right images captured by a stereo camera, and a three-dimensional terrain image indicating peripheral terrain information is fused with this to generate fused view information. Yes. As a result, even when the visibility is poor, it is possible to provide a field of view with the same reality as when the visibility is good to assist in maneuvering and to reliably detect an obstacle.

JP 2006-165946 A JP 2001-344597 A

As described above, various devices for improving the field of view of the vehicle have been proposed in the past. However, the device described in Patent Document 1 is mainly intended for general vehicles, and is difficult to visually confirm. The rear image is projected. Further, the device described in Patent Document 2 is mainly intended for aircraft, so that the view of the aircraft that travels at high speed is merged with map information to improve the view and to reliably detect obstacles. I have to. As described above, a device aiming at improving the field of view of a vehicle requires a configuration in accordance with the characteristics and driving environment of each vehicle.
On the other hand, in industrial vehicles such as forklifts, there was a problem that the front field of view was blocked by a shielding object such as a mast as described above, and operability and safety were poor. Is not applicable to this, and a visual field improvement device specialized for the problem of industrial vehicles has been desired.
SUMMARY OF THE INVENTION In view of the above-described problems of the prior art, an object of the present invention is to provide an industrial vehicle visibility improving system and method that improve the forward visibility of an industrial vehicle and improve operability and safety. .

Therefore, in order to solve such a problem, the present invention is a system for improving the visibility of an industrial vehicle having a shield in front of a seat,
A head-mounted display that is mounted on an operator of the industrial vehicle and projects an image while transmitting an external scenery;
A first imaging device that is fixed in front of the shield and images a front view;
A second imaging device that is fixed to the head-mounted display and images a front scene similar to the field of view of the operator;
An image processing device that performs image processing on the first captured image acquired by the first imaging device and the second captured image acquired by the second imaging device;
The image processing device deforms the first captured image based on the second captured image, and an image cutout unit that cuts out a range corresponding to a shield from the deformed image obtained by the image deforming unit. And an image output unit that outputs the partial image cut out by the image cutout unit to the head mounted display.

  According to the present invention, an image on the back side of the shielding object is deformed by image processing, and a range corresponding to the shielding object is cut out from the deformed image and displayed on the transmissive head-mounted display. Can be projected over the external scenery, and the operator can obtain a clear view regardless of the shielding object. Therefore, the operability and safety of the industrial vehicle can be improved.

Moreover, it is a system in which marking is given to the shielding object,
The image transformation unit extracts the marking position, calculates a positional relationship between the first imaging device and the second imaging device based on the extracted marking size and inclination angle, and based on the positional relationship The first captured image is reduced at a predetermined ratio, and a deformation process including a process of tilting the first captured image based on the tilt angle is performed.
Thus, by marking the shielding object in advance, the image can be easily deformed.

Further, the image deformation means performs a matching process for identifying a corresponding region by obtaining a correlation between the first captured image and the second captured image regardless of a size ratio, and the matching A deformation process including a process of reducing and tilting the first captured image is performed based on the process.
In the present invention, since the first captured image is deformed by performing the matching process based on the second captured image, the deformation process can be performed without marking the shielding object.

  Furthermore, in these inventions, the image processing apparatus acquires position and angle information of the second captured image, and designates a display position and angle on the head mounted display based on the position and angle information. It is preferable to provide an angle indicator. By acquiring the position and angle of the second captured image in this way, it is possible to grasp what field of view the operator has at the present time, and by displaying a partial image on the head mounted display so as to correspond to this It is possible to display the partial images so as to be accurately superimposed on the external scenery.

Further, as a method invention, a method for improving the visibility of an industrial vehicle having a shield in front of a seat,
A front image is captured by a first imaging device fixed in front of the shield, and the same field of view as the operator by a second imaging device fixed to a head mounted display attached to the operator of the industrial vehicle. Imaging the scenery, and
Transforming the first captured image acquired by the first imaging device based on the captured image acquired by the second imaging device;
Cutting out a range corresponding to a shield from the deformed image;
And displaying the clipped partial image on the head-mounted display so as to be superimposed on an external scenery.

  As described above, according to the present invention, in an industrial vehicle having a shielding object in front, a scene ahead of the shielding object is imaged, and after the captured image is deformed to be the same as the field of view as viewed from the operator, the shielding is performed. By displaying a partial image cut out of the area corresponding to the object overlaid on the external scenery on the head-mounted display, the operator can see the image behind the shield at the same time, improving operability and safety. Become.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, relative arrangements, and the like of the components described in this embodiment are not intended to limit the scope of the present invention unless otherwise specified, but are merely illustrative examples. Not too much.
FIG. 1 is a side view of a forklift equipped with a visual field improvement system according to the present embodiment, FIG. 2 is a schematic plan view including the internal configuration of the image processing apparatus according to the visual field improvement system according to the present embodiment, and FIG. FIG. 4 is a diagram showing a processing flow by the visual field improvement system of the first embodiment, FIG. 5 is a diagram for explaining image processing in the first embodiment, and FIG. 6 is an image in the second embodiment. It is a figure explaining a process.

  In addition, although a case where the present system is applied to a forklift is shown as an example in the present embodiment, the present invention can be applied to all industrial vehicles whose forward visibility is hindered by a shielding object. In addition, an industrial vehicle means the vehicle for conveyance handling used in a factory premise, a warehouse, etc.

With reference to FIG.1 and FIG.2, the apparatus structure of the visual field improvement system of a present Example is demonstrated.
The forklift 1 includes a seat 2 on which an operator 5 is seated, a claw 4 on which a load is placed, a mast 3 having a lifting member that lifts and lowers the claw 4, and an operation unit (not shown) that controls the lifting and lowering of the claw 4. Further, as a characteristic configuration of the present embodiment, a camera A10 fixed to a forklift part in front of the mast 3, a head mounted display 11 worn by an operator 5, and a camera fixed to the head mounted display 11 B12 and an image processing device 13 that performs image processing of images captured by the cameras 11 and 12 are provided.

As shown in FIG. 1, the camera A <b> 10 is provided, for example, in front of the mast 3 and at a stationary part, and the front scene from the mast 3 is an imaging range. At this time, when it is desired to visually check the lifting / lowering operation of the luggage, it is preferable to install the imaging range so as to include the nail.
The camera B12 is fixed to the head mounted display 11, and has a field of view substantially the same as the field of view of the operator 5 as an imaging range.

  The head mounted display 11 is mounted on the head of the operator 5 and is provided by the operator via an optical system having an eyepiece, a half mirror, and the like, displayed on a display device such as a liquid crystal panel (LCD). It is a device. The image processing unit includes an image processing unit and an image display unit. The image processing unit receives an image signal from the image processing device 13 and a necessary control signal via a wireless or line, and displays the image signal on the image display unit. Processing to convert to an image is performed. This image is displayed on the image display unit through the optical system and is visually recognized by the operator 5. The head mounted display 11 can transmit the external scenery together with the image information subjected to the image processing. That is, as shown in FIG. 3, the head mounted display 11 is a semi-transparent image display screen that is displayed from the image processing unit so as to be superimposed on a transparent part (outlined part in FIG. 3) 41 through which an external scenery can be seen. (Shaded portion in FIG. 3) is a transmission type that projects 42.

  The image processing device 13 is connected to the camera A10, the head mounted display 11, and the camera B12 via a wired or wireless line, and is obtained by the camera A10 and an image acquisition unit 14 that captures captured images captured by the camera A10 and the camera B12. The obtained captured image A is deformed based on the captured image B obtained by the camera B12, the image deforming unit 15 that cuts the deformed deformed image only in a range corresponding to the mast 3, and the cut image And an image output unit 17 for outputting the partial image to the head mounted display 11.

  The image deforming unit 15 deforms the captured image A by a method of deforming the captured image A based on the marking applied to the mast 3 and a matching process between the captured image A and the captured image B as in Example 1 described later. The method etc. are mentioned. When a stereo camera is used, edge 3D data may be detected, and the wire frame may be deformed in accordance with the image of the head mounted display 11. In this case, distortion of an image of an object at a short distance is suppressed. be able to. Further, when a camera capable of measuring a distance for each pixel is used, the captured image A may be deformed based on the marking after the image is converted into an image viewed from the head mounted display 11.

With reference to FIG. 4, the process flow by the visual field improvement system of Example 1 is demonstrated.
First, the front scenery is imaged by the cameras A and B, and the captured image is taken into the image processing apparatus 13 (S1). The captured image A21 captured is shown in FIG. 5A, and the captured image B22 is shown in FIG. 5B. Since the captured image B22 is located farther than the captured image compared to the captured image A21, the captured image is displayed in a smaller size, and the image is partially missing due to the mast 3. The mast 3 is provided with a marking 30 in advance. Although the marking 30 is represented by a square shape in the present embodiment, the marking 30 is not limited to this and may be any point, line, or shape. When marking is set as a point, it is preferable to set the number to 3 or more, whereby the position information and angle information of the camera B12 can be accurately obtained.

Then, the marking 30 given to the mast 3 is extracted from the captured image B22 acquired by the camera B12 (S2), and the position of the extracted marking 30 in the xy direction (in the plane perpendicular to the imaging direction of the camera B) Is calculated (S3). The position information obtained here is used when an image is displayed on the head mounted display 11 later.
Further, the distance between the camera A10 and the camera B12 is calculated from the length of the side of the marking 30. In this embodiment, since the marking 30 has a quadrangular shape, the side length is used. However, in the case of a point, the distance is calculated from the length or size between the points (S4). This is because the distance between the camera B12 and the mast 3 is obtained by comparing the size of the marking 30 measured in advance with the size of the marking 30 on the captured image B22. Further, since the camera A10 is fixed to the mast 3, the distance between the camera A10 and the camera B is calculated.

Similarly, by comparing the angle of the marking 30 measured in advance with the angle of the marking 30 on the captured image B22, the angle of how much the marking 30 is inclined on the captured image B22 is calculated (S5). This angle refers to an angle in a vertical plane with respect to the imaging direction of the camera B.
Next, the captured image A21 is reduced at a predetermined ratio based on the distance between the camera A10 and the viewing target and the distance between the camera B12 and the viewing target (S6). Further, the captured image A is tilted based on the calculated marking angle (S7). By this reduction and inclination processing, the captured image A becomes an image corresponding to the captured image B. That is, the captured image A matches the scene viewed from the operator 5. This deformed image is shown in FIG.

Then, a captured image A21 in a range corresponding to the size of the mast 3 is cut (S8), and the cut partial image 23 is displayed on the head mounted display 11 based on the calculated position information of the marking 30.
Thereby, the back image of the mast 3 can be visually recognized by the operator 5 together with the external scenery.

With reference to FIG. 6, the process flow by the visual field improvement system of Example 2 is demonstrated. In the second embodiment, detailed description of the same configuration as that of the first embodiment is omitted. In the second embodiment, matching processing is performed between the captured image A21 acquired by the camera A10 and the captured image B acquired by the camera B12 without performing marking on the mast 3, and image processing is performed. I am doing so.
First, the front scene is imaged by the cameras A10 and B12, and the captured image is taken into the image processing device 13 (see FIG. 6A). And the matching process which specifies the area | region corresponding by calculating | requiring the mutual correlation irrespective of a ratio of magnitude | size between the picked-up image A21 acquired with camera A10 and the picked-up image B acquired with camera B is performed. The captured image A21 is deformed so as to correspond to the captured image B by the matching process, and the deformed captured image A21 is superimposed on the captured image B (see FIG. 6B). Further, the captured image A21 in the range corresponding to the mast 3 is cut (see FIG. 6C), and the cut partial image 24 is displayed on the head mounted display 11 (see FIG. 6D).
According to the present embodiment, it is not necessary to mark the mast 3 as in the first embodiment, and the captured image can be deformed.

  According to the present invention, a front view can be ensured even in a vehicle with a shield in front, and a visibility assisting device with high safety and operability can be obtained. As described above, the present invention can be widely applied to industrial vehicles in which a front object is present and front visibility is hindered.

It is a side view of the forklift provided with the visual field improvement system of a present Example. 1 is a schematic plan view including an internal configuration of an image processing apparatus according to a visual field improvement system of an embodiment. It is a figure which shows a visual field when a transmissive | pervious head mounted display is mounted | worn. It is a figure which shows the processing flow by the visual field improvement system of Example 1. FIG. FIG. 6 is a diagram illustrating image processing in Embodiment 1. FIG. 10 is a diagram for explaining image processing in a second embodiment. It is a side view which shows the conventional forklift.

Explanation of symbols

1 Forklift 3 Cargo lifting mast 5 Operator 10 Camera A
11 Transmission Head Mount Display 12 Camera B
13 Image Processing Device 14 Image Acquisition Unit 15 Image Deformation Unit 16 Image Clipping Unit 17 Image Output Unit 21 Captured Image A
22 Captured image B
23, 24 Partial image 25 External scenery 30 Marking

Claims (5)

A system for improving the visibility of an industrial vehicle having a shield in front of a seat,
A head-mounted display that is mounted on an operator of the industrial vehicle and projects an image while transmitting an external scenery;
A first imaging device that is fixed in front of the shield and images a front view;
A second imaging device that is fixed to the head-mounted display and images a front scene similar to the field of view of the operator;
An image processing device that performs image processing on the first captured image acquired by the first imaging device and the second captured image acquired by the second imaging device;
The image processing device deforms the first captured image based on the second captured image, and an image cutout unit that cuts out a range corresponding to a shield from the deformed image obtained by the image deforming unit. And an image output unit for outputting the partial image cut out by the image cutout unit to the head mounted display. The system of claim 1, wherein the shielding is marked.
The image transformation unit extracts the marking position, calculates a positional relationship between the first imaging device and the second imaging device based on the extracted marking size and inclination angle, and based on the positional relationship 2. The field of view improvement of an industrial vehicle according to claim 1, wherein the first captured image is reduced at a predetermined ratio and a deformation process including a process of tilting the first captured image based on the tilt angle is performed. system.   The image deforming means performs a matching process for specifying a corresponding region by obtaining a correlation between the first captured image and the second captured image regardless of a ratio of the sizes. 2. The industrial vehicle visibility improving system according to claim 1, wherein a deformation process including a process of reducing and tilting the first captured image is performed.   The image processing apparatus includes a position / angle instruction unit that acquires position and angle information of the second captured image and indicates a display position and angle on the head mounted display based on the position and angle information. The industrial vehicle visual field improvement system according to any one of claims 1 to 3. A method for improving the visibility of an industrial vehicle having a shield in front of a seat, comprising:
A front image is captured by a first imaging device fixed in front of the shield, and the same field of view as the operator by a second imaging device fixed to a head mounted display attached to the operator of the industrial vehicle. Imaging the scenery, and
Transforming the first captured image acquired by the first imaging device based on the captured image acquired by the second imaging device;
Cutting out a range corresponding to a shield from the deformed image;
A method of improving the visibility of an industrial vehicle, comprising: displaying the cut partial image on the head mounted display so as to be superimposed on an external scenery.

Images