JP5442353B2 - Image display apparatus and method for vehicle control - Google Patents

Description

  The present invention relates to an image display apparatus and method for vehicle operation for maneuvering a vehicle under a poor visibility condition.

In order to supplement human vision at night or in situations with poor visibility such as dense fog, an infrared camera is installed in the vehicle and displayed on the driver's seat monitor and head-up display, making it safe and easy to drive. Can be improved.
For this purpose, an apparatus using an infrared camera has already been proposed (for example, Patent Documents 1 to 3).
Further, technologies related to the present application are disclosed in Patent Documents 4 and 5 and Non-Patent Document 1. Patent Document 5 is not disclosed at the time of filing of the present application.

Patent Document 1 includes an infrared sensor, an imaging unit, a high-temperature region extraction unit, and a distance measurement unit, and presents an infrared image for assisting driving to the driver of the vehicle. In this document, driving by a driver on a vehicle is assumed. However, the present invention can also be applied to remote control by transmitting an image wirelessly.
Patent Document 2 includes a visible imaging unit, an infrared imaging unit, a superimposing unit, a display unit, and a region setting unit. The visible image and the infrared image are selected and superimposed and displayed. In addition, by transmitting such an image wirelessly, remote control can be performed only by looking at one screen, so that information more suitable for remote control can be presented.
Patent Document 3 is intended to make the infrared image easier to see by emphasizing the temperature of the detection target and excluding the temperature outside the target, assuming the temperature of the target (person) to be detected and the temperature outside the target. is there. Furthermore, it has also been proposed to switch the emphasis target and the exclusion target depending on the situation.

Japanese Patent Application Laid-Open No. 2-158900, “Obstacle Detection Device” Japanese Patent Application Laid-Open No. 2008-230358, “Display Device” Japanese Patent Laid-Open No. 2001-23095, “Vehicle Surrounding Information Notification Device” JP 2008-202971 A, “Infrared Camera Adjustment Device and Infrared Camera Adjustment Method” Japanese Patent No. 2008-079784, “Mobile Robot Travel Area Discrimination Device and Travel Area Discrimination Method”, unpublished

“Distance Image Sensor”, Internet <http: // denko. panasonic. biz / Ebox / kyorigazou / feature. html>

  Infrared light does not require a light source, and therefore has the advantage that it can be viewed in any dark place and has a long wavelength so that it can be transmitted and observed even in environments where smoke and fog are dense. Patent Documents 1 to 3 described above using this merit try to overcome a state in which it is difficult for a human to see and control because of the inability to see humans. However, these conventional techniques have the following weak points.

FIGS. 1 to 3 are images of visible light (A) and infrared light (B) at night, FIG. 1 is a paved road, FIG. 2 is an unpaved road, and FIG. 3 is a grass road (wadder).
The brightness of infrared images is greatly influenced by temperature, and there are some that can see the difference in material and surface roughness, and others that do not look very clearly. For this reason, in a situation where there is no temperature difference such as at night, it is possible to distinguish only materials that have a significant difference in infrared radiation (radiation) rate between concrete and asphalt, such as the paved road in FIG. Such materials as soil and grass are buried in noise, making it very difficult to distinguish the state of the road and its surroundings.
In addition, the infrared image captures the radiant heat of a substance, and a shadow cannot be seen like a visible light image, so that the shape cannot be captured. For this reason, when the temperature difference between the surrounding objects is small and made of the same material (for example, an infrared image of a road (claw) covered with grass as shown in FIG. 3), the shape changes greatly. The parts that should be there are also indistinguishable.
This makes it difficult to drive safely while viewing infrared images, especially on dirt roads and grass roads.

Further, the means of Patent Document 3 requires an image quality improvement rule as to how to improve the image quality of a part having a characteristic (specific brightness or the like).
However, when maneuvering a vehicle under poor visibility conditions, the infrared camera image is a general outdoor environment (whether paved or unpaved, night or weather), and the road surface ( In addition, since the appearance (brightness and contrast) of the road surface boundary) varies, it has been impossible to apply certain image quality improvement rules, making it difficult to see the road surface and its surroundings.

  The present invention has been developed to solve the above-described problems. That is, an object of the present invention is to display an image of the road surface area in front of the vehicle at night or under poor visibility such as dense fog, thereby maneuvering the moving body without distinguishing between paved and unpaved roads. An object of the present invention is to provide an image display apparatus and method for vehicle control that can be used.

According to the present invention, an infrared camera that is mounted on a vehicle and shoots an infrared image of a specific range in front of the vehicle;
A three-dimensional shape measuring device mounted on a vehicle and measuring a three-dimensional shape of the specific range;
An arithmetic processing unit that detects a road surface area where a vehicle easily passes from the three-dimensional shape, and forms a corrected infrared image by image processing that emphasizes a luminance change in a portion other than the road surface area on the infrared image;
There is provided a vehicle control image display device comprising a display device for displaying the corrected infrared image.

Further, according to the present invention, an infrared image of a specific range in front of the vehicle is taken by an infrared camera mounted on the vehicle,
Measure the three-dimensional shape of the specific range with a three-dimensional shape measuring device mounted on the vehicle,
A road surface area where a vehicle can easily pass from the three-dimensional shape is detected by the arithmetic processing device, and a corrected infrared image is formed by image processing that emphasizes a luminance change in a portion other than the road surface area on the infrared image,
A vehicle image display method is provided, wherein the corrected infrared image is displayed by a display device.

According to an embodiment of the present invention, the arithmetic processing unit
(A) detecting the road surface area where the vehicle is easy to pass from the three-dimensional shape;
(B) The road surface area on the infrared image and the other road area other than that are obtained from the installation position and orientation of the three-dimensional shape measuring apparatus and the infrared camera,
(C) Performing noise removal and smoothing processing of the infrared image of the road surface area and / or performing edge enhancement processing of the infrared image of the road area,
(D) The infrared image of the said road surface area | region and the said road area is synthesize | combined, and a correction | amendment infrared image is formed.

According to another embodiment of the present invention, the arithmetic processing unit
(A) detecting the road surface area where the vehicle is easy to pass from the three-dimensional shape;
(B) The road surface area on the infrared image and the other road area other than that are obtained from the installation position and orientation of the three-dimensional shape measuring apparatus and the infrared camera,
(C) Create a smoothed image obtained by removing and smoothing the entire infrared image and an edge enhanced image obtained by performing edge enhancement on the entire infrared image,
(D) The infrared image of the road surface area on the smoothed image and the road area on the edge-enhanced image are combined to form a corrected infrared image.

Further, according to the present invention, an infrared image of a specific range in front of the vehicle is taken by an infrared camera mounted on the vehicle,
Measure the three-dimensional shape of the specific range with a three-dimensional shape measuring device mounted on the vehicle,
The ease of passing the vehicle on the road surface is calculated from the three-dimensional shape by the arithmetic processing unit. On the infrared image, the higher the ease of passing the vehicle on the road surface, the smoother the image becomes, and the easier the vehicle passes on the road surface. A corrected infrared image is formed by image processing that emphasizes changes in the image,
A vehicle image display method is provided, wherein the corrected infrared image is displayed by a display device.

  According to the embodiment of the present invention, the ease of passing the vehicle calculated by the arithmetic processing unit is set as the degree of undulation of the ground.

According to the above-described apparatus and method of the present invention, the corrected infrared image obtained by performing image processing so that the road surface area looks smooth is formed by the arithmetic processing unit, and the corrected infrared image is displayed by the display device. In the infrared image that is difficult to appear above, the road surface area looks smoother than the original infrared image, and the road outside area looks more emphasized than the original infrared image.
For this reason, it is possible to cause the driver to have an illusion that the flatness of the road surface area of the corrected infrared image looks higher and the undulation of the road area appears to be larger. And since this is close to an actual three-dimensional shape, control is performed based on information that takes into account information seen in the infrared (original infrared image) and three-dimensional relief size information (three-dimensional shape). Will be able to.

In this way, by displaying images that are close to human sensibility and sensation, it becomes possible to make it appear as if there are unevenness and undulations in places with undulations such as areas outside the road, and this is the appropriate route This will lead to the provision of information that enables driving along the road.
This kind of image processing makes it easy for the operator to operate an infrared image (corrected infrared image) that is easy to control even in environments with poor temperature changes, which were originally not good at infrared rays, or in environments where similar materials have created large shape changes. Can be presented.

It is an image of visible light (A) and infrared rays (B) of a paved road at night. It is an image of visible light (A) and infrared rays (B) on an unpaved road at night. It is the image of the visible light (A) and infrared rays (B) of the grass road (wadder) at night. 1 is an overall layout diagram of a vehicle steering image display device according to the present invention. It is a block diagram at the time of boarding operation which a person gets on and controls directly. It is a block diagram at the time of the remote control which performs unmanned remote control. It is a schematic diagram of the line scan of the laser by LRF. It is a schematic diagram of the surface scan of the laser by LRF. It is 1st Embodiment figure which shows an infrared image correction method. (A) is an image obtained by detecting a road surface area 4 where a vehicle can easily pass from the three-dimensional shape 3 by the road surface detection means 32, and (B) is a method in which the road surface area 4 is superimposed on the infrared image 2 photographed by the infrared camera 10. It is an image. It is a schematic diagram which shows the relationship between the three-dimensional coordinate system (X, Y, Z) on a vehicle, and the two-dimensional coordinate system (x, y) on an infrared image. It is a schematic diagram which shows the relationship between the two-dimensional coordinate system (X, Y, 0) on a bird's-eye view map, and the two-dimensional coordinate system (x, y) on an infrared image. It is a flowchart which shows the process example by the image display apparatus for vehicle steering of this invention. It is 2nd Embodiment figure which shows an infrared image correction method.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In each figure, common portions are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 4 is an overall layout diagram of a vehicle steering image display device according to the present invention, and FIGS. 5 and 6 are overall configuration diagrams. FIG. 5 is a configuration diagram at the time of boarding operation in which a person gets on and directly controls the vehicle, and FIG. 6 is a configuration diagram at the time of remote control in which the vehicle is unmanned and remotely controlled.

  As shown in FIGS. 4 to 6, the vehicle display image display device of the present invention includes an infrared camera 10, a three-dimensional shape measurement device 20, an arithmetic processing device 30, and a display device 40.

  The vehicle 1 is a vehicle that travels outdoors at night or under poor visibility conditions such as dense fog, and travels unattended and remotely operated even on a vehicle (for example, a passenger car or a truck) that a person rides and directly controls. It may be a body (for example, a remotely operated vehicle or a robot).

  The infrared camera 10 is mounted on the vehicle 1 and captures an infrared image 2 in a specific range in front of the vehicle. “Vehicle front” refers to the front when moving, and includes the rear when the vehicle moves backward. The “specific range” is a view range required when maneuvering the vehicle, and this range may be variable in the front / rear and left / right directions, or the range of the view range may be variable.

The three-dimensional shape measuring device 20 is mounted on the vehicle 1 and measures the three-dimensional shape 3 in a specific range in front of the vehicle. Note that the specific range preferably matches the range determined by the three-dimensional shape measuring apparatus 20 and the range determined by the infrared camera 10, but does not completely match, and it is only necessary to match within a necessary range.
The arithmetic processing unit 30 detects a road surface area 4 through which the vehicle 1 easily passes from the three-dimensional shape 3, and forms a corrected infrared image 5 that is image-processed so that the road surface area 4 looks smooth on the infrared image 2.
The display device 40 displays the corrected infrared image 5.

5 and 6, arrows indicate data flow (and data contents).
Each component is arranged in the vehicle 1 as shown in FIG. 4 and senses the environment in the direction shown in the figure. In FIG. 5, information is presented inside the vehicle. In FIG. Present information to the terminal.

  The infrared camera 10 is a camera that photographs far-infrared light (wavelength: 8 μm to 13 μm) emitted (radiated) according to the temperature of an object. The infrared camera 10 is characterized in that a hot object, an object with a rough surface, and an object with a high emissivity (infrared light emissivity) appear bright.

The three-dimensional shape measurement apparatus 20 is a laser range finder (hereinafter referred to as “LRF”) or a range image camera. The range image camera is a device for remotely measuring the shape having a function of measuring a round trip time of light from a light source on a pixel of the camera, and is disclosed in Non-Patent Document 1, for example.
The three-dimensional shape measuring device 20 is a device for measuring a three-dimensional shape around the vehicle 1 as long as it can measure up to about 15 m to 40 m (depending on the maximum speed) in the traveling direction of the vehicle 1, and the measurement range is forward. Even if it is limited, the entire circumference may be measured.

7 and 8 are schematic diagrams showing laser scanning when the three-dimensional shape measuring apparatus 20 is LRF. FIG. 7 shows line scanning, and FIG. 8 shows surface scanning.
In the case of LRF, the laser 6 is scanned in a line shape (FIG. 7) or a planar shape (FIG. 8), and the distance is measured by the time until the laser 6 hits an object and a reflection is obtained.
In the case of a laser range finder that scans in a line shape, as shown in FIG. 7, the vehicle 1 moves forward while scanning, thereby measuring the surrounding three-dimensional shape (a specific range in front of the vehicle).
In the case of a laser range finder that scans in a planar shape, the laser scans the surface as shown in FIG. 8, so that the surrounding three-dimensional shape (a specific range in front of the vehicle) can be automatically measured.

  The arithmetic processing device 30 performs the following two processes based on the sensor information (the infrared image 2 and the three-dimensional shape 3) from the infrared camera 10 and the three-dimensional shape measuring device 20.

(A) Road surface detection function by the road surface detection means 32 The road surface detection means 32 is based on the three-dimensional shape 3 around the vehicle 1 and is an area in which a vehicle can easily pass, that is, an area that is gentle and continuous in the traveling direction. It is detected as a road surface area 4. That is, the road surface detection means 32 evaluates the undulations of the three-dimensional shape 3 and classifies the vehicle 1 into a region where the vehicle 1 can travel and a region where the vehicle 1 cannot travel, and detects the travelable region as the road surface region 4.
Therefore, in the present application, the “road surface area” means an area where a vehicle can easily pass through, an area which is continuous in the traveling direction, and an area where the vehicle can travel, as determined from the three-dimensional shape 3. Does not have to match.
For example, from the three-dimensional shape 3 of the specific range ahead of the vehicle, the gradient and the surface roughness of the range are obtained, and the region that is the gradient and the surface roughness that the vehicle can travel and is continuous in the traveling direction is the road surface region. Detect as.
A specific means for detecting the road surface is disclosed in Patent Document 5.

(B) Infrared Image Correction Function by Infrared Image Correction Unit 34 The infrared image correction unit 34 makes the road surface region 4 appear smooth based on the road surface information detected by the road surface detection unit 32 of the infrared image 2 of the infrared camera 10. Is subjected to image processing.
Specific operation of the infrared image correction function will be described later.

The display device 40 is equipped with either or both of the following, and displays the result of image processing to the operator.
(A) When a person gets on the vehicle 1 to drive as shown in FIG. 5, the screen is provided on the driver's seat, or is projected on the glass surface of the driver's seat and is displayed around the driver as a translucent image. Present information.
(B) As shown in FIG. 6, when driving a vehicle by remote control, the corrected infrared image 5 obtained by the processing in the arithmetic processing device 30 is wirelessly transmitted through the wireless transmission device 42 and the wireless reception device 44. To the remote display terminal 46. The remote display terminal 46 receives the image information and displays it on the screen display device 40.
The communication may use a digital system such as a wireless LAN or an analog system such as a television image transmitter.

(Infrared image correction method)
FIG. 9 is a first embodiment diagram illustrating an infrared image correction method.
In this figure, the infrared image correction by the infrared image correction means 34 is composed of six steps (steps) S1 to S6, and is converted so as to make it easy to see the road surface portion of the captured infrared image.

1. Display position association (step S1)
10A is an image obtained by detecting the road surface area 4 where the vehicle easily passes from the three-dimensional shape 3 by the road surface detection means 32, and FIG. 10B shows the road surface area in the infrared image 2 photographed by the infrared camera 10. 4 is a superimposed image.
As shown in FIG. 10A, the road surface region 4 obtained from the information of the LRF 20 is obtained as a three-dimensional positional relationship with the vehicle 1 as an axis, or as a two-dimensional positional relationship when the vehicle 1 is viewed from directly above. As shown in FIG. 10B, it is determined which range in the image corresponds to the area corresponding to the road surface area 4 on the infrared image captured by the infrared camera 10.
Hereinafter, an area other than the road surface area 4 in the infrared image 2 is referred to as an “outside road area 4b”.

FIG. 11 is a schematic diagram showing the relationship between the three-dimensional coordinate system (X, Y, Z) on the vehicle and the two-dimensional coordinate system (x, y) on the infrared image. FIG. 12 is a schematic diagram showing the relationship between the two-dimensional coordinate system (X, Y, 0) on the bird's-eye view map and the two-dimensional coordinate system (x, y) on the infrared image.
An arbitrary point (X, Y, Z) in a coordinate system (X: traveling direction, Y: left, Z: up) with the direction of the vehicle body 1 in FIG. 11 as an axis, or a point ( Which pixel (x, y) on the infrared image corresponds to the data observed in (X, Y, 0) can be obtained by Expression (1) of Equation 1.
Here, A is a camera parameter, and is expressed as in Expression (2) of Formula 1 by the optical axis centers Cx and Cy on the image and the focal lengths fx and fy.
R is a matrix for converting the advancing axis coordinate system of the vehicle body and the image plane coordinate system of the camera (right: x, down: y, near side: z), and is obtained from Equation (3) of Equation 1.

  The parameter Rx in the equation (3) becomes the equation (4) based on the roll deviation angle between the vehicle body and the camera, the parameter Ry becomes the equation (5) by the elevation angle of the camera from the vehicle body, and the parameter Rz is the camera body. Rotation type (6) due to deviation in yaw direction from

  In addition, t in equation (1) is the installation position of the infrared camera viewed from the center of the vehicle body, and is obtained from equation (7) of equation 3 from the camera installation offset positions tx, ty, tz from the vehicle body center.

If necessary, distortion is removed as a pre-process after shooting. At that time, the camera obtains a distortion parameter in advance and removes the distortion. (For example, Patent Document 4 presents a method using a calibration plate).
Even when the camera orientation is accurately obtained, a calibration plate as shown in Patent Document 4 is installed at a known position, the positional relationship between the camera and the component plate is obtained by image processing, and the camera orientation / position is determined by the calibration plate. R and t are determined by calculating backward from the position of.

  In FIG. 10 (B), the road surface region 4 is given in the form of a contour line segment or a distribution whole area divided by a mesh (mesh) and indicating the presence or absence of a road surface portion for each section. Find out where the points of the contour and the central point of each square correspond on the image with the above-mentioned corresponding pixel calculation formula, convert the contour shape to obtain the shape, or information on presence / absence to the corresponding position in all sections Project to get the distribution.

  Based on the road surface distribution information described above, an out-of-road mask in which the field of view at the viewpoint that can be superimposed on the infrared image 2 is the same, the out-of-road area 4b is 1, and the out-of-road area 4a is 0 is generated.

  When the road surface area 4 is a contour, an outside road mask is generated by painting the inside of the contour with 1. When the road surface area 4 is projecting information on the mesh, the nearest projection point is searched from each pixel of the mask image, and the pixel value is determined using the information on the presence or absence of the road surface.

2. Extract road surface area from infrared image (step S2)
In step S2, an image of only a portion of the infrared image corresponding to the road surface area 4a (referred to as a “road surface infrared image”) is created.
The above-described portion where the road mask is 0 is expanded by the filter size of step S4, and then only the pixel value of the infrared image that overlaps the pixel having the value of 0 is copied to obtain a partial image (road surface infrared image) of the road surface region. ).
The reason why the filter size is expanded is that if there is no surrounding pixel value, the filter in step S4 cannot be applied, so that the surrounding pixel value at the position where the value after the filter application is desired is left.

3. Extraction of out-of-road area from infrared image (step S3)
In step S3, an image of only the infrared image of the road area (referred to as an “road infrared image”) is created.
The portion where the above-mentioned road mask is 1 is expanded by the filter size in step S5, and thereafter, only the pixel value of the infrared image that overlaps the pixel having the value of 1 is copied, and a partial image (road red) Outside image).
The reason why the filter size is expanded is that if there is no surrounding pixel value, the filter cannot be applied, so that the surrounding pixel value at the position where the value after the filter application is desired is left.

4). Noise removal and smoothing of road surface area (S4 step)
In step S4, a noise removal / smoothing filter is applied to the road surface infrared image, and noise removal is performed by smoothing.
Only applies if all surrounding pixels according to the filter size contain values.

Implement the filter of Table 1 that convolves the following weights:

5. Edge enhancement in off-road area (step S5)
In step S5, an edge enhancement filter is applied to the out-of-road infrared image, and the contour is clarified by edge enhancement.
Only applies if all surrounding pixels according to the filter size contain values.
For the edge-enhanced image, the calculation of “original value—Laplacian (secondary differential) processing value” is performed for each pixel to obtain the result.

This is realized by the filter of Table 2 that convolves the following weights.

6). Composition (Step S6)
In step S6, the road surface infrared image after the filter application and the image of the road infrared image after the filter application are superimposed.
For pixel values at the same place in both images, basically only one of the two images (the road surface infrared image and the roadside infrared image) contains a value, and that value is adopted. Create a composite image. However, depending on the filter shape, a pixel having both values may remain in the boundary portion. In this case, an average value is adopted.

FIG. 13 is a flowchart showing an example of processing performed by the vehicle control image display device of the present invention.
The operation of the apparatus of the present invention will be described with reference to this figure below.
(1) The LRF is continuously operated by LRF measurement (T1), and the shape (undulations and obstacles) of a specific range in front of the vehicle (especially about 0 to 30 m ahead and about 20 m wide) is measured.
(2) The road surface area 4 (area estimated to be a road surface) is extracted from the measurement result of the LRF 20 by road surface detection (T2). T1 and T2 are repeated until the entire target region is measured and extracted.
(3) The road surface distribution totalization (T3) continuously accumulates and aggregates the distribution of the road surface area 4 to create a road surface distribution map.
(4) In infrared image capturing (T4), the far-infrared camera captures the infrared image 2 in the specific range in front of the vehicle at regular time intervals.
(5) Calculate how the road surface distribution looks from the same viewpoint as the infrared camera 10 by superimposing on the infrared image (T5), and generate a road exterior mask image with the same viewpoint and the same field of view as the infrared image To do.
(6) In the road surface infrared image extraction (T6), an image of the road surface region 4 of the infrared image is extracted to generate an image (road surface infrared image).
(7) In the road surface infrared image noise removal (T7), a noise removal / smoothing filter is performed to remove noise in the road surface region 4 (road surface infrared image) of the infrared image.
(8) In the out-of-road infrared image extraction (T8), the image of the out-of-road area 4b of the infrared image is extracted to generate an image (out-of-road infrared image).
(9) The edge enhancement filter is implemented in the roadside infrared image contour enhancement (T9), and the edge enhancement is performed on the infrared image (roadside infrared image) in the roadside region 4b.
(10) In the synthesis (T10), the filtered road surface infrared image and the filtered out-of-road infrared image are superimposed.
(11) In the image display (T10), the processing result image is presented to the operator sequentially in the wireless terminal or in the vehicle.

Hereinafter, another embodiment of the present invention will be described.
(1) Replacement of Processing Procedure FIG. 14 is a second embodiment diagram showing an infrared image correction method.
In the first embodiment of the infrared image correction method described above, an image is extracted with an out-of-road mask depending on whether or not it is on the road surface area, and different filter processing is performed.
On the other hand, in 2nd Embodiment, the road surface area | region 4 where the vehicle 1 passes easily from the three-dimensional shape 3 is detected in S10,
In S11, the road surface area 4 on the infrared image 2 and the other outside road area 4b are obtained from the installation positions and postures of the three-dimensional shape measuring apparatus 20 and the infrared camera 10,
In S12 and S13, a smoothed image obtained by removing and smoothing the entire infrared image and an edge enhanced image obtained by performing edge enhancement processing on the entire infrared image are created.
In S14 to S16, the infrared image of the road surface area 4 on the smoothed image and the road area 4b on the edge-enhanced image are combined to form a corrected infrared image 5.

That is, as shown in FIG. 14, each filter (noise removal / smoothing and edge enhancement) is applied to the infrared image 2, and the image after the filter application is cut out using a mask.
In this case, compared with the first embodiment, the performance is not affected, and only the procedure is different.

(2) Continuously changing the influence of a filter with a parameter With respect to the above-described embodiment, a continuous shape change (an increase in undulation amount or the like) can be taken into account by changing the following two points.
A. Road surface (travelability)
Instead of the binary value indicating whether or not the road surface is output, which is output from the road surface detection means 32 in FIG. 5, a coefficient is applied to the parameter used to determine the road surface, and a value indicating a driving difficulty level of 0% to 100% is output. . For example, this corresponds to outputting the maximum value of the two parameters in Example [0031] of unpublished Patent Document 5 by applying a coefficient.

B. Applying a filter whose characteristics can be continuously changed by parameters (between noise removal and edge enhancement) Instead of switching the filter, a filter that convolves the filter shown in Table 3 in which the weight shown below is variable depending on the location is implemented. . The filter is applied to each pixel of the entire image.
The filter parameter α indicates a degree on the road and takes a value from 0 to 1. 1 means that it is almost surely considered as a road area, and 0 means that it is almost certainly considered as a road surface area. This corresponds to a driving difficulty level of 0.1 to 0.4.
This filter has functions of noise removal and edge enhancement, and the two strengths are changed by α. If α is large, the edge enhancement function is strong, and if α is small, the noise removal (smoothing) function is strong.
As a result, it is possible to improve and display an image in consideration of a continuous change and an intermediate undulation amount.

  Further, for reference, a filter when α is 0 is shown in Table 4, and a filter when α is 1 is shown in Table 5. It can be seen that when α is 0, the filter is a smoothing filter, and when α is 1, the filter is an edge enhancement filter.

(3) Utilization of undulations The factor of undulation (the distance profile measured by the LRF), which is a characteristic that can be easily obtained from the presence or absence of the road surface (the road surface degree) as a factor for switching the filter type or switching the filter parameters. The filter is changed (changed) using the absolute value of the differential value or the absolute value of the differential value of the height profile.

  When the means of (2) and (3) of the above-described embodiment are used, the image of the portion with a large undulation has a strong contrast according to the degree, so that the driver feels unevenness. The small portion of the portion looks smooth according to the degree, so that the pilot feels flat, and the degree of undulation that does not originally appear in the image can be conveyed to the pilot with a natural feeling.

According to the apparatus and method of the present invention described above, the corrected infrared image 5 is processed by the arithmetic processing device 30 so that the road surface area 4 looks smooth, and the corrected infrared image is displayed by the display device 40. In the infrared image 2 in which the three-dimensional shape is difficult to appear on the image, the road surface area 4 looks smoother than the original infrared image, and the road area 4b appears to be emphasized from the original infrared image.
For this reason, it is possible to cause the driver to have an illusion that the flatness of the road surface area 4 of the corrected infrared image 5 looks higher and the undulation of the road area 4b appears larger. And since this is close to an actual three-dimensional shape, control is performed based on information that takes into account information seen in the infrared (original infrared image) and three-dimensional relief size information (three-dimensional shape). Will be able to.

In this way, by displaying images that are close to human sensibility and sensation, it becomes possible to make it appear as if there are unevenness and undulations in places with undulations such as areas outside the road, and this is the appropriate route This will lead to the provision of information that enables driving along the road.
This kind of image processing makes it easy for the operator to operate an infrared image (corrected infrared image) that is easy to control even in environments with poor temperature changes, which were originally not good at infrared rays, or in environments where similar materials have created large shape changes. Can be presented.

  In addition, this invention is not limited to embodiment mentioned above, Of course, it can change variously in the range which does not deviate from the summary of this invention.

1 vehicle (moving body), 2 infrared image, 3 3D shape,
4 road surface area, 4b outside road area, 5 corrected infrared image,
10 Infrared camera,
20 3D shape measuring device (laser range finder),
30 arithmetic processing units, 32 road surface detection means,
34 infrared image correction means, 40 display device,
42 wireless transmitters, 44 wireless receivers,
46 Remote display terminal

Claims (6)

An infrared camera that is mounted on a vehicle and shoots an infrared image of a specific range in front of the vehicle;
A three-dimensional shape measuring device mounted on a vehicle and measuring a three-dimensional shape of the specific range;
An arithmetic processing unit that detects a road surface area where a vehicle easily passes from the three-dimensional shape, and forms a corrected infrared image by image processing that emphasizes a luminance change in a portion other than the road surface area on the infrared image;
A vehicle control image display device comprising: a display device that displays the corrected infrared image. Taking an infrared image of a specific range ahead of the vehicle with an infrared camera mounted on the vehicle,
Measure the three-dimensional shape of the specific range with a three-dimensional shape measuring device mounted on the vehicle,
A road surface area where a vehicle can easily pass from the three-dimensional shape is detected by the arithmetic processing device, and a corrected infrared image is formed by image processing that emphasizes a luminance change in a portion other than the road surface area on the infrared image,
A vehicle image display method, wherein the corrected infrared image is displayed by a display device. By the arithmetic processing unit,
(A) detecting the road surface area where the vehicle is easy to pass from the three-dimensional shape;
(B) The road surface area on the infrared image and the other road area other than that are obtained from the installation position and orientation of the three-dimensional shape measuring apparatus and the infrared camera,
(C) Performing noise removal and smoothing processing of the infrared image of the road surface area and / or performing edge enhancement processing of the infrared image of the road area,
(D) The vehicle image display method according to claim 2 , wherein a corrected infrared image is formed by combining infrared images of the road surface area and the outside road area. By the arithmetic processing unit,
(A) detecting the road surface area where the vehicle is easy to pass from the three-dimensional shape;
(B) The road surface area on the infrared image and the other road area other than that are obtained from the installation position and orientation of the three-dimensional shape measuring apparatus and the infrared camera,
(C) Create a smoothed image obtained by removing and smoothing the entire infrared image and an edge enhanced image obtained by performing edge enhancement on the entire infrared image,
3. The vehicle according to claim 2 , wherein the road image area on the smoothed image and the infrared image of the road area on the edge-enhanced image are combined to form a corrected infrared image. Image display method. Taking an infrared image of a specific range ahead of the vehicle with an infrared camera mounted on the vehicle,
Measure the three-dimensional shape of the specific range with a three-dimensional shape measuring device mounted on the vehicle,
The ease of passing the vehicle on the road surface is calculated from the three-dimensional shape by the arithmetic processing unit. On the infrared image, the higher the ease of passing the vehicle on the road surface, the smoother the image becomes, and the easier the vehicle passes on the road surface. A corrected infrared image is formed by image processing that emphasizes changes in the image,
A vehicle image display method, wherein the corrected infrared image is displayed by a display device. The vehicle image display method according to claim 5 , wherein the ease of passing the vehicle calculated by the arithmetic processing unit is set as the degree of undulation of the ground.

Images