KR20170119508A - Detection System for Vehicle Surroundings and Detection Method for Vehicle Surroundings Using thereof - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a system and method for detecting a periphery of a vehicle, and a system for detecting a periphery of a vehicle according to an embodiment of the present invention includes a plurality of A plurality of visible image measurement sensors, a plurality of thermal image measurement sensors disposed between the plurality of visible image measurement sensors, and a plurality of visible image measurement sensors and a plurality of thermal image measurement sensors, And a control unit for providing a surround fusion image for the surround region around the vehicle through image fusion.

Description

TECHNICAL FIELD [0001] The present invention relates to a vehicle surroundings detection system and a vehicle surround detection method using the same,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a vehicle surroundings detection system and a vehicle surroundings detection method using the same, and more particularly to a vehicle surroundings detection system and a detection method for detecting a blind spot and surroundings of a vehicle using a visible image measurement device and a thermal image measurement device .

Recently, various technologies have been developed to detect the surroundings of vehicles by the development of ADAS (Advanced Driver Assistance Systems) technology. In particular, the Around View Monitoring (AVM) system created at least four front, rear and left / right side cameras around the vehicle to create an around view.

In Patent Documents 1 to 3, it is disclosed that a virtual surround view image is generated by using an image sensor or an ultrasonic measurement sensor. However, in the case of Patent Documents 1 to 3, a camera for image measurement is installed in various places to acquire an image, and in order to acquire an image of the surroundings of the vehicle, a wide viewing angle, i.e., an Ultra-Wide Field of View Use a lens.

Specifically, a fish-eye lens is mainly used. Fisheye lenses are often used to create an aural view with a large wide angle of view, but the biggest drawback is the problem of image distortion. Also, even if the distortion is corrected, the focal length is very short, so that the image is seriously distorted and the object around the vehicle seems to fall on the road surface.

Especially, in case of a large vehicle, considering the wide viewing angle to be measured when constructing a camera array like a general vehicle in order to create an around view, it is necessary to increase the number of cameras or to have an ultra-wide viewing angle lens. In this case, the problem of distortion as described above will become more serious.

On the other hand, in the case of Patent Document 3, an ultrasonic measurement sensor is applied to obtain distance information with respect to an obstacle around the vehicle. However, ultrasonic measurement sensor has low resolution and narrow field of view because it has a short distance to detect an object and a low spatial resolution. Therefore, in order to be applied to the vicinity of a large-sized vehicle, a plurality of ultrasonic measurement sensors must be disposed and processed, and thus the entire system becomes complicated.

KR10-2010-0113959A KR10-2012-0060283A KR10-1511610 B1

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a vehicle surroundings detection system and a detection method capable of detecting all-weather blind and night blind zones and surroundings of a large vehicle through fusion and matching processes of multi- .

It is another object of the present invention to provide a vehicle surroundings sensing system and method for detecting an object or a user by matching an ultrasonic measurement sensor with image information of a multi-wavelength sensor.

It is another object of the present invention to provide a highly flexible vehicle surroundings detection system that can be installed without being restricted by the shape of a vehicle.

A vehicle perimeter detection system according to an embodiment of the present invention includes: a plurality of visible image measurement sensors mounted on an upper portion of a vehicle, the plurality of visible image measurement sensors being arranged at predetermined intervals over an entire area around the vehicle; A plurality of thermal image measurement sensors disposed between the plurality of visible image measurement sensors; And a control unit connected to the plurality of visible image measurement sensors and the plurality of thermal image measurement sensors and providing a surround fusion image for the area around the vehicle through the image fusion of the measured visible image and the deteriorated image.

Wherein the plurality of visible image measurement sensors are four visible image measurement sensors arranged respectively in a first direction, a second direction opposite to the first direction, and a third direction and a fourth direction between the first direction and the second direction, The plurality of thermal image measurement sensors may be four thermal image measurement sensors each disposed between the visible image measurement sensors.

Wherein the control unit receives visible image data and thermal image data measured by the plurality of visible image measurement sensors and a plurality of thermal image measurement sensors to perform lens distortion correction of the visible image data and thermal image data, It is possible to generate a surround visible image by performing image matching of first correction data subjected to geometric alignment correction after distortion correction and second correction data subjected to photometric alignment correction with the visual image data after the lens distortion correction .

Wherein the geometric alignment correction is performed by obtaining two first homography matrices for each of the plurality of thermal image measurement sensors through homography-based position estimation between adjacent visual image measurement sensors on both sides, Based on the geometric correction parameters based on the multilevel transformation matrices, second multilevel transformation matrices obtained by obtaining second homography matrices between the neighboring visible image measurement sensors on the basis of one homography matrix, Can be calculated.

The photometric alignment correction may include analyzing photometric alignment of the visible image data in the overlap region between adjacent visible image measurement sensors and adjusting the photometric calibration parameters set to minimize the difference in brightness and hue in the overlap region, It is possible to calculate the second correction data by applying it to the image data.

The image matching may be performed by performing feature extraction and matching between the visible image and the thermal image or the visible image based on the first correction data and the second correction data and obtaining a final projection transformation matrix, The surround visible image can be calculated through blending.

In the image fusion, the visible image data _Iv (x, y) and the thermal image data I ' _M (x, y) are subjected to wavelet transform (W) And reconstructs the surround fusion image I _F (x, y) by the inverse wavelet transform (W ^-1 ), and can be expressed by the following equation (1).

[Formula 1]

Here, C _Y and C _M are the wavelet coefficients of the visible image data I _v (x, y) and the thermal image data I _M (x, y), and C _F is the wavelet coefficient after the fusion rule? Where a and d are approximate and detailed coefficients, respectively, and N is the size of the image in the y direction.

(X, Y, Z) on the world coordinate system through tracking of the ultrasonic measurement sensor, and a plurality of ultrasonic measurement sensors arranged at predetermined intervals around the vehicle, was converted to a point P _img (x, y), it is possible to provide a position information on the vehicle in the plane of the surrounding obstacles.

The control unit is connected to the control unit, and selects at least one selected from the surround visual image information, the surround fusion image information, the position information of the obstacle around the vehicle, the surround visual image or the image information of the obstacle in the surround fusion image, And an information providing unit for providing the information.

The information providing unit may include at least one display unit selected from the group consisting of a flat panel display and a headset mount display.

And a mount plate coupled to the other end of the connection unit and having the plurality of visible image measurement sensors and the plurality of thermal image measurement sensors mounted thereon have.

The connecting portion or the connecting portion and the mounting plate may be formed of a magnetic frame, and may be magnetically coupled to the vehicle.

According to another aspect of the present invention, there is provided a method for detecting a vehicle periphery, comprising: providing a plurality of visible image data measured at predetermined intervals over an entire area around a vehicle at an upper portion of the vehicle; Providing a plurality of thermal image data measured over a whole area around the vehicle at predetermined intervals; And providing a surround fused image for a surround region around the vehicle through the measured visual fusion and image fusion of the deteriorated image.

According to the present invention, it is possible to provide a vehicle surroundings detection system and a detection method capable of detecting all-weather blind zones and surroundings of a large vehicle through fusion and matching processes of multi-wavelength sensors.

In addition, according to the present invention, it is possible to provide a system and method for detecting a periphery of a vehicle that can detect an object or a user by matching an ultrasonic measurement sensor with image information of a multi-wavelength sensor.

According to the present invention, it is possible to provide a highly flexible vehicle surroundings detection system that can be installed without being restricted by the shape of the vehicle.

1 is a perspective view (a) and a side view (b) showing a vehicle surroundings detection system mounted on a vehicle according to an embodiment of the present invention.
FIG. 2 is a plan view schematically showing the arrangement of a visible image measurement sensor and a thermal image measurement sensor mounted in a vehicle surroundings sensing system according to an embodiment of the present invention.
3 is a block diagram schematically showing a vehicle perimeter detection system according to an embodiment of the present invention.
FIG. 4 is a flowchart illustrating a method for detecting a vehicle periphery of a vehicle periphery sensing system according to an embodiment of the present invention.
5 is a flowchart illustrating a geometric alignment correction and a photometric alignment correction method of a visible image measurement sensor in a vehicle surroundings sensing system according to an exemplary embodiment of the present invention.
6 is a schematic diagram illustrating a multi-step alignment process of geometric alignment correction according to an embodiment of the present invention.
7 is a schematic diagram illustrating an overlap area for photometric alignment correction according to an embodiment of the present invention.
FIG. 8 is a flowchart illustrating an image matching method of a vehicle surroundings sensing system according to an embodiment of the present invention.
9 is a schematic view illustrating a process of converting an obstacle detected by an ultrasonic measurement sensor into a plane coordinate system in a world coordinate system in a vehicle perimeter sensing system according to an embodiment of the present invention.

These embodiments are capable of various modifications and various embodiments, and specific embodiments will be described in detail with reference to the drawings. It is to be understood, however, that it is not intended to limit the scope of the specific embodiments but includes all transformations, equivalents, and alternatives falling within the spirit and scope of the disclosure disclosed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the embodiments of the present invention,

In the embodiment, 'module' or 'sub' performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module except for 'module' or 'module' which need to be implemented by specific hardware, and implemented by at least one processor (not shown) .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Referring to the accompanying drawings, the same or corresponding elements are denoted by the same reference numerals, and a duplicate description thereof will be omitted.

1 is a perspective view (a) and a side view (b) of a vehicle surroundings sensing system according to an embodiment of the present invention. 2 is a plan view showing the arrangement of sensors of a vehicle surroundings sensing system according to an embodiment of the present invention.

Referring to FIG. 1, a vehicle perimeter detection system 2 according to an embodiment of the present invention is mounted on an upper portion of a vehicle 1.

A vehicle perimeter detection system according to an embodiment of the present invention includes a plurality of visible image measurement sensors mounted on an upper portion of a vehicle 1 and arranged over a whole area around a vehicle at predetermined intervals, A plurality of thermal image measurement sensors disposed between the measurement sensors and a plurality of visible image measurement sensors and a plurality of thermal image measurement sensors, And a control unit for providing a surround fusion image with respect to the surround image.

Specifically, in the case of the vehicle perimeter detection system according to the embodiment of FIG. 1, the vehicle 1 is mounted on the vehicle 1 and includes a first direction D ₁ around the vehicle, a third direction D ₃ opposite thereto, The four visible image measurement sensors C ₁ , C ₂ , C ₃ , and C 4 disposed respectively in the second direction D ₂ and the fourth direction D ₄ between the first direction D ₁ and the third direction D ₃ , C _3, C _4), the visible image-measuring sensor _{(C 1, C 2, C} 3, C 4) 4 -column image sensor that is disposed between _{(T 1, T 2, T} 3, T 4), And four thermal image measurement sensors (T ₁ , T ₂ , T ₃ , T ₄ ) connected to the four visible image measurement sensors (C ₁ , C ₂ , C ₃ , C ₄ ) And a control unit 30 for providing a surround fusion image around the vehicle.

Referring to FIGS. 1 and 2, a vehicle perimeter detection system according to an embodiment of the present invention provides an image or an image of an entire area around a vehicle, that is, a "surround ". Thereby providing information about the obstacles that may exist in the entire area around the vehicle.

According to an embodiment of the present invention, the surround region is divided into four regions at predetermined intervals, the visible image and the thermal image of each region are measured, and the visible image and the thermal image of each region are fused, Image, and thermal image information of the region, i.e., the surround region, or fused image information thereof.

In the case of the embodiment of FIG. 1, four regions are divided into four regions according to four directions, a first direction D ₁ , a second direction D ₂ , a third direction D ₃ and a fourth direction D ₄ , Images and thermal images are measured and information is provided. In this case, the image is divided into four directions in consideration of the view angle of the visible image and the thermal image sensor, and the allowable data capacity, but the present invention is not limited thereto, and the number may be varied depending on the specific situation.

In the present specification, the first direction D ₁ refers to one direction around the vehicle, the third direction D ₃ refers to the direction opposite to the first direction D ₁ , D ₂₎ and the fourth direction (D ₄₎ refers to a direction between the first direction (D ₁₎ and a second direction (D _2), respectively. For example, the front area, the rear area, and the left / right area of the vehicle may be referred to.

The vehicle perimeter detection system according to an embodiment of the present invention includes a rod-shaped connection part 21 connected to the upper part of the vehicle and a mounting plate 23 connected to the other end of the connection part 21 . The connecting portion 21 of the embodiment may be mounted on the roof portion of the vehicle. More specifically, the connecting portion 21 or both the connecting portion 21 and the mounting plate 23 may be formed of a magnetic frame, So as to be fixed to the roof portion of the vehicle.

Accordingly, it is possible to provide a vehicle surroundings detection system that is easy to install and can be applied to various vehicles by allowing vehicles of various shapes and sizes to be fixedly mounted on a partial area of the roof.

In particular, in the case of a large vehicle such as a train or a bus, it is difficult to provide an entire surround image because the entire area around the vehicle is wide. However, since the present invention is formed so as to be easily fixed to the roof portion of the vehicle, it can be easily fixed to a large-sized vehicle and can be used for providing a surround image around the vehicle.

2 is a view schematically showing an arrangement of a visible image measurement sensor and a thermal image sensor mounted on a mount plate 23 according to an embodiment of the present invention.

The visible image measurement sensor and the thermal image sensors may be mounted on the mounting plate 23 and configured to measure the top view of the vehicle. The top view is divided into four regions and the top view in the first direction D ₁ , the second direction D ₂ , the third direction D ₃ and the fourth direction D ₄ is divided into four regions The first visible image measurement sensor C ₁ , the second visible image measurement sensor C ₂ , the third visible image measurement sensor C ₃ and the fourth visible image measurement sensor C ₄ are arranged in the first direction (D ₁ ), the second direction (D ₂ ), the third direction (D ₃ ), and the fourth direction (D ₄ ). Referring to FIG. 2, one visible image measurement sensor may be arranged for each direction with a diamond-shaped arrangement structure.

Thermal image measurement sensors T ₁ , T ₂ , T ₃ , and T ₄ may be disposed between adjacent visible image measurement sensors C ₁ , C ₂ , C ₃ , and C ₄ . Specifically, a first thermal image measurement sensor T ₁ is disposed between the first visible image measurement sensor C ₁ and the second visible image measurement sensor C ₂ , and a second visible image measurement sensor C ₂ is disposed between the first visible image measurement sensor C ₁ and the second visible image measurement sensor C ₂ . A second thermal image measurement sensor T ₂ is disposed between the third visible image measurement sensor C ₃ and the third visible image measurement sensor C ₃ and a second thermal image measurement sensor T ₃ is disposed between the third and fourth visible image measurement sensors C ₃ and C ₄ , sensor (T ₃₎ that can be arranged is, the arrangement of the fourth infrared sensor (T ₄₎ between the fourth and first visible image measuring sensor (C _4, C _1).

That is, each of the four visible image measurement sensors and the four thermal image measurement sensors may be disposed with a field of view of about 95 degrees or more with respect to the surround area around the vehicle. Specifically, a visible image measurement sensor and a thermal image measurement sensor having an angle of view of approximately 95 degrees can be alternately arranged. Accordingly, one visible image measurement sensor can be assigned to an area over an approximately 95 degree angle and be configured to measure the image. Of course, the allocation area of the visible image measurement sensor and the allocation area of the thermal image measurement sensor do not necessarily coincide with each other, but may be integrated into the 360-degree surround area through the image registration and image fusion process to provide information on the surround area .

According to an exemplary embodiment, a CCD camera may be used as the visible image measuring sensor, and an infrared ray camera may be used as the infrared image measuring sensor. A lens having a size of 4 mm and a 2/3 "image sensor can be used, and an aspect ratio of 4: 3, a resolution of 1280 × 960, a horizontal viewing angle of 95.5 °, a vertical viewing angle of 79 ° An image measurement sensor and a thermal image measurement sensor may be used.

Accordingly, obstacles at distances up to 48.5 m can be monitored, and can be configured to detect obstacles up to 24.2 m.

In the embodiment of FIG. 1, a vehicle detection system is applied to a train. Can be applied to trams with a height of 3 m (y ₁ ) and can detect obstacles from 1.92 m to 17.1 m from the vehicle. An obstacle of 2 m height (y ₁ ) can be detected within a range of 45.2 degrees within the range of the vertical angle of view (θ).

3 is a block diagram schematically showing a vehicle perimeter detection system according to an embodiment of the present invention.

3, the system for detecting peripherals of the present invention includes a first visible image measuring sensor C ₁ and a first thermal image measuring sensor T ₁ as a first multi-modal sensor unit 11, . ≪ / RTI > In addition, the first multimodal sensor unit may further include a first ultrasonic measurement sensor E ₁ . Accordingly, it is possible to measure the presence or absence of an obstacle or the distance to the obstacle through the ultrasonic information for each direction.

According to an embodiment of the present invention, the first ultrasonic measurement sensor E ₁ may be disposed in the first direction D ₁ . The second ultrasonic measurement sensor E ₂ may be disposed in the second direction D ₂ and the third ultrasonic measurement sensor E ₃ may be disposed in the third direction D ₃ , ultrasound sensor (E ₄₎ may provide the ultrasound information of the obstacle relative to the fourth direction (D ₄₎ are disposed in each arrangement direction.

The first multimodal sensor unit 11 may provide the information of the obstacle in the first direction D ₁ , but is not limited thereto. For example, the first multimodal sensor unit 11 may display the visible image in the first direction D ₁ , Information, thermal image information between the first direction D ₁ and the second direction D ₂ , and ultrasonic information in the first direction D ₁ .

According to an embodiment of the present invention, sensors having an angle of view of approximately 95 degrees can be arranged at an interval of approximately 90 degrees with respect to a 360 degree surround region around the vehicle, whereby each angle of view has an angle of view of approximately 95 degrees, Modal sensor unit 11, the second multimodal sensor unit 13 and the third multimodal sensor unit 15 and the fourth multimodal sensor unit 17 that are arranged in order at intervals have. Thereby, it is possible to secure sufficient visible image data or thermal image data for providing information on the 360-degree surround area around the vehicle, and to optimize the data throughput and the number of sensors (or cameras). However, the present invention is not necessarily limited thereto, and may vary depending on the installation environment, the specifications of the sensor, and the like.

The first multimodal sensor unit 11, the second multimodal sensor unit 13, the third multimodal sensor unit 15 and the fourth multimodal sensor unit 17 are connected to the control unit 30, , Thermal image, and ultrasound information may be transmitted.

In addition, the controller 30 may be connected to the information providing unit 40. The information providing unit 40 may include at least one selected one of a surround visual image information, a surround fusion image information, a position information of an obstacle around the vehicle, a surround visible image or an image information of an obstacle in a surround fusion image, . ≪ / RTI >

The control unit 30 processes the visible image, the thermal image, and the ultrasound information to transmit at least one of the surround visible image information, the surround fusion image information, and the ultrasound-based obstacle position information to the information providing unit 40, To be displayed in the form of a sound, a display image, or the like.

In particular, the information providing unit 40 displays visual information such as surround visual image information, surround fusion image information, position information of an obstacle around the vehicle, and surround image information or image information of an obstacle in a surround fusion image in a display form A display unit, and more specifically, a display unit including at least one selected from the group consisting of a flat panel display and a headset mount display.

FIG. 4 is a flowchart illustrating a method for detecting a vehicle periphery of a vehicle periphery sensing system according to an embodiment of the present invention.

According to an embodiment of the present invention, a vehicle surroundings sensing system measures a visible image of a plurality of regions constituting a surround region in a plurality of visible image measurement sensors (S11), and forms a surround region in the plurality of thermal image measurement sensors Thermal images of a plurality of regions can be measured (S13).

The control unit 30 receives the visible image data and the thermal image data measured by the plurality of visible image measurement sensors and the plurality of thermal image measurement sensors and performs the lens distortion correction of the measured visible image data and the thermal image data (S20). The lens distortion correction may use a method known in the art and is for correcting the distortion of the lenses used in the visible image measurement sensor and the thermal image measurement sensor.

Then, the camera position based geometric alignment correction after the lens distortion correction is performed (S21) to calculate the first correction data, and the alignment of the photometer with the visual image data measured after the lens distortion correction is corrected (S23) The correction data can be calculated.

The first correction data and the second correction data are data obtained by correcting the position, distortion, brightness, color, and the like of a plurality of the visible images and the thermal images measured. Using the corrected visual images and thermal images, Matching (S30) and image fusion (S50) can be performed.

The first correction data and the second correction data are subjected to image matching (S30), and a surround visible image is calculated (S40). Image Synthesis is a process of combining regions that overlap each other based on position information of a plurality of visible images or a plurality of degradation-phase images. According to an embodiment of the present invention, visible image information for a 360-degree surround region can be generated (S40) by combining overlapping regions in visible images having an image angle of approximately 95 degrees.

In the image matching process S30, a plurality of visible image data whose position, distortion, brightness, color, etc. are corrected are calculated based on position information between the corrected images, and warping and blending Blending) to match one surround visual image. In the image matching step S30, positional information is calculated between the plurality of visible images and the plurality of thermal images through image matching of the corrected visible images and the thermal images, and the image fusion process (S60) as basic information for image fusion.

In other words, the surround visible image generated through the image matching is fused with the thermal image (S50), thereby generating the surround fusion image (S60). Here, the thermal image data that has undergone the correction process (S20 to S21) and the surround visual image generated based on the positional information derived in the image matching process (S30) are fused so that the visible image and the thermal image are fused It is possible to generate a surround fusion image for all directions of 360 degrees.

When providing the image information for the surround region only with the visible image, accurate ambient information can not be provided when the weather is bad or dark and the visibility is bad. However, in the case of the present invention, it is possible to provide a peripheral system for detecting the surroundings of a vehicle which can accurately detect the surroundings even when dark or poor visibility is provided by providing a surround fusion image in which a visible image and a thermal image are fused.

Hereinafter, the photometric alignment correction S23 and the geometric alignment correction S21 will be described in more detail with reference to FIGS. 5 and 6. FIG.

The visible image and the thermal image data measured by the visible image measurement sensor and the thermal image measurement sensor are subjected to a lens distortion correction process S20 and then a photometric alignment correction process S23 and a geometric alignment correction process S21.

,

;

,

;

,

;

,

)을 구하고, 상기 2개의 제1 호모그래피 행렬들에 기초하여 상기 인접한 가시영상 측정 센서들 사이의 제2 호모그래피 행렬들(

,

,

,

)에 기초하여 이들의 다단계 변환 행렬을 구할 수 있다(S211).In the case of the geometric alignment correction S21, for each of the plurality of thermal image measurement sensors T ₁ , T ₂ , T ₃ and T ₄ , the visible image measurement sensors C ₁ , C ₂ , C ₃ , C ₄ ), the two first homography matrices ((1), (2), and (3) are obtained through homography based camera position estimation

,

;

,

;

,

;

,

) Between the neighboring visible image measurement sensors based on the two first homography matrices and second homography matrices (

,

,

,

(S211). ≪ / RTI >

Specifically, the process of homography-based camera position estimation is performed on the basis of a homography matrix for an image obtained by photographing a plane of a three-dimensional space at a position of the thermal image measurement sensor T ₁ adjacent to the position of the visible image measurement sensor C ₁ A can be expressed as the following [Equation 2].

[Formula 2]

m ₂ = Am ₁

The rotation matrix R and the transformation vector t, which convert from the position of the visible image measurement sensor C1 to the position of the adjacent thermal image measurement sensor T ₁ , are in the relationship of [Equation 3] below and satisfy R and t I ask.

[Formula 3]

X ₂ = RX ₁ + t, n ^t X ₁ = d

Here, n and d represent the distance d from the normal vector n of the plane to the origin with reference to the positional coordinates of the visible image measurement sensor C ₁ . Accordingly, we can obtain R and t from A as well.

A plurality of transformation matrices related to a plurality of visible image sensors and a plurality of thermal image sensors are obtained, and a lookup table (geometric LUT) for geometric parameters for geometric correction parameters is obtained, The geometric correction (S213) can be performed.

By performing the alignment process step by step, the multilevel transformation matrices can be obtained and the first correction data of the visible image and the degradation image can be generated based on the geometric correction parameters based on the multilevel transformation matrices (S213).

Hereinafter, with reference to FIG. 5 and FIG. 7, the photometric alignment correction (S23) process will be described in detail.

Since it is necessary to correct the brightness and color between the measured images, a photometric alignment analysis (S231) can be performed. It can be corrected by referring to the parameter that minimizes the difference between the overlap areas O _{( m, n )} of four adjacent visible image measurement sensors. Here, the photometric correction parameters can be represented, for example, as O _{( m, n )} , where m = 1, 2, 3, 4 and n = (m + 1) modulo 4. That is, a photometric LUT of the photometric parameter can be obtained and applied to perform photometric correction (S233). This can be performed for both the visible image and the thermal image, and the second correction data in which brightness and color are corrected through photometric alignment correction can be generated.

Through the above process, the visible image and the thermal image in which the lens distortion, the camera alignment, the brightness and the color are corrected can generate a surround visible image for the 360-degree omnidirection through the image matching (S30) (S40).

8 is a flowchart illustrating an image matching process (S30) according to an exemplary embodiment of the present invention in more detail.

Referring to FIG. 8, the image matching (S30) may be performed by combining the data of the overlap region among the measured thermal image and visible image data. The feature extraction and matching in the overlap region between the visible image and the thermal image or the visible image are performed based on the corrected image data S31 including the first correction data and the second correction data (S32) Projection transformation matrices between the visible image and the thermal image or the visible image can be obtained (S33).

Then, a plurality of visible images based on the final projection transformation matrices, more specifically, overlapping areas are subjected to warping and blending (S34) to generate a surround visual image for all directions (S34).

In other words, since a single sensor can not capture an omni-directional image, a visible image or a thermal image for the divided regions is obtained, and a final projection transformation matrix between the plurality of images is obtained. Images can be generated. Accordingly, the surround visible image can be generated through the image matching (S30) of the corrected visible images.

In the case of the projection transformation matrices in the image matching step (S30), it includes positional information between respective images, specifically between visible images or between a visible image and a thermal image, and based on the positional information, Since the image fusion is performed, the image matching process is also performed on the visible image and the thermal image, and can be used to acquire the position information.

The surround visualized image data and the thermal image data can generate a surround fused image through an image fusion process. That is, it is possible to emphasize an infrared thermal image to provide an image of all-weather day and night, and to provide a surround fusion image emphasizing the appearance information for the situation recognition.

According to an embodiment of the present invention, the image fusion process can be roughly divided into three stages: a decomposition process, a fusion process, and a restoration process.

According to one embodiment, wavelet transformation (W) of visible image data I _v (x, y) and thermal image data I ' _M (x, y) is performed by using Symlets wavelets, to go through the fusion process by the fusion rule (Φ), inverse wavelet transform (W ^-1) fused to surround the image _{(I F (x, y)} ) to have a fused image can be performed to restore, which by the following [expression 1].

[Formula 1]

Here, C _Y and C _M are the wavelet coefficients of the visible image data I _v (x, y) and the thermal image data I _M (x, y), and C _F is the wavelet coefficient after the fusion rule? Where a and d are approximate and detailed coefficients, respectively, and N is the size of the image in the y direction.

In the image fusion step, the approximation coefficient is used as the maximum value, and the detailed coefficient can be weighted according to the position. In the convergence step where different weights are given to the position, the infrared image image can be emphasized and the appearance information of the general image (image) can be emphasized. In addition, the image reconstructed through the inverse transform can be output as a color fusion image by adding the saturation component of the visible image.

The above-described correction method, image matching and fusion method can be used as an embodiment of the present invention. Of course, various correction methods, image matching methods, and fusion methods applicable in the art may be applied.

9 is a schematic view illustrating a process of converting an obstacle detected by an ultrasonic measurement sensor into a plane coordinate system in a world coordinate system in a vehicle perimeter sensing system according to an embodiment of the present invention.

As described above, in the case of the ultrasonic measurement sensors E ₁ , E ₂ , E ₃ , and E ₄ , they can be mounted together on the mounting plate, or they can be installed in a part of the side surface of the vehicle in each direction.

9 (a), it can be mounted on the vehicle side in the first direction D ₁ , the second direction D ₂ , the third direction D ₃ and the fourth direction D ₄ And may be configured to sense ultrasonic signals in the respective directions.

One point P (x, y, z) on the world coordinate system through obstacle tracking of the ultrasonic measurement sensor can be converted into a point P _img (x, y) on the image plane (pixel coordinate system) The relational expression can be transformed into homogeneous coordinates as shown in [Equation 4].

[Formula 4]

Here, T can be expressed as a 4 × 3 matrix as described above, and [R | t] is a rigid transformation that transforms the world coordinate system (FIG. 9A) into a camera coordinate system (FIG. 9B) And T _{pers (1)} is a projection matrix for projecting the 3D coordinate on the camera coordinate system to the normal image plane (Fig. 9 (c)). K is the intrinsic parameter matrix and T _{pers (1)} is the projection transformation to a plane with d = 1, i.e. Z _c = 1.

Through this process, the obstacle can be converted to a point on the pixel coordinate system, i. E., On the surround visible image plane. The position information of the obstacle may be transmitted to the user through the display unit 41 (see FIG. 3) or may be provided to the user through the information providing unit 40 including the alarm device . The control unit 30 may transmit the image information of the obstacle in the surround fusion image to the information providing unit 40 or the display unit 41 together with the position information on the plane of the obstacle around the vehicle.

According to another aspect of the present invention, there is provided a method for detecting a vehicle periphery, comprising the steps of: providing a plurality of image data measured at predetermined intervals over an entire area around a vehicle at an upper portion of the vehicle; Providing a plurality of thermal image data measured over a predetermined interval and providing a surround fused image for a surround region around the vehicle through the image fusion of the measured image and the deteriorated image.

In the method of detecting the periphery of a vehicle according to an embodiment of the present invention, the methods applied in the above-described perimeter sensing system may be applied.

As described in the vehicle perimeter detection system, the entire area around the vehicle is divided into four regions of a first direction D ₁ , a second direction D ₂ , a third direction D ₃ , and a fourth direction D ₄ be the direction, and the visible image data and thermal image data may be the data for the same direction, or also one direction, as in the embodiment of the first (D _1), the second direction (D _2), the third direction ( D ₃₎ and the fourth may be the visible image data in the direction (D _4), the thermal image data for a region between about two direction adjacent in the first direction (D ₁₎ to the fourth direction (D ₄₎ .

These images may be provided as visual or auditory information through the information providing unit as the surround fusion image information through the image matching and image fusion process as described above.

In the case of the present invention, even if the shape of the vehicle itself can not be formed by installing the camera, it can be installed by being applied to a part of the roof of the vehicle. Accordingly, it is possible to provide a vehicle surroundings detection system and method with high degree of design freedom.

In addition, in the case of using a fisheye lens, even if distortion correction is performed, the distance is extremely shortened and the image is seriously distorted, and it is difficult to detect the distance. However, according to the embodiment of the present invention, Through the convergence process, 360 degree surround visibility image can be realized.

In the case of a camera of a general peripherals sensing device, the field matching process is complicated in the correction of external parameters, which is not suitable for a large vehicle. However, according to the embodiment of the present invention, the photometric alignment correction, geometric correction, It is possible to provide a vehicle surroundings detection system and method that can be easily applied to a large-sized vehicle through a relatively simple process.

Also, it is possible to provide a vehicle surroundings detection system and method capable of monitoring a blind spot with respect to an omnidirectional area of a vehicle regardless of the time of day or night, through image fusion of a general image and an infrared thermal image.

In addition, it can provide a perimeter sensing system and method that can provide monitoring over a full 360 degree range of surroundings while at a lower cost than radar (Radio Detection and Ranging).

In addition, it is possible to provide a system and method for detecting the surroundings of a vehicle, which can simultaneously monitor events occurring in the world coordinate system by using an ultrasonic measurement sensor.

In addition, it is possible to provide a system and method for detecting a periphery of a vehicle which is constructed of a magnet frame in the case of a mounting plate and a connecting portion for mounting sensors on a vehicle, and which is durable and easy to install.

1: vehicle
2: Peripheral detection system
21: Connection
23: mounting plate
30:
40: Information provision
41:
D ₁ : the first direction
D ₂ : the second direction
D ₃ : Third direction
D ₄ : fourth direction

Claims (13)

A plurality of visible image measurement sensors mounted on the top of the vehicle and arranged over a whole area around the vehicle at predetermined intervals;
A plurality of thermal image measurement sensors disposed between the plurality of visible image measurement sensors; And
And a control unit connected to the plurality of visible image measurement sensors and the plurality of thermal image measurement sensors and providing a surround fusion image for an area around the vehicle through the visual fusion of the measured visual image and the deteriorated image, Around vehicle detection system.
The method according to claim 1,
Wherein the plurality of visible image measurement sensors are four visible image measurement sensors arranged respectively in a first direction, a third direction opposite to the first direction, and a second direction and a fourth direction between the first direction and the third direction,
Wherein the plurality of thermal image measurement sensors are four thermal image measurement sensors respectively disposed between the visible image measurement sensors.
The apparatus of claim 1,
Receiving a plurality of visible image measurement sensors and visible image data and thermal image data measured by the plurality of thermal image measurement sensors,
Performing a lens distortion correction of the visible image data and the thermal image data,
A surround viewing image is generated by image matching of first correction data that has undergone geometric alignment correction after the lens distortion correction and second correction data that has been subjected to photometric alignment correction with the visible image data after the lens distortion correction Wherein the vehicle surroundings detection system comprises:
The method of claim 3,
The geometric alignment correction may include:
For each of the plurality of thermal image measurement sensors, two first homography matrices are obtained through homography-based position estimation between the neighboring visible image measurement sensors, and two first homography matrices Calculating a first correction data based on the geometric correction parameters based on the multilevel transformation matrices, obtaining second homography matrices between the neighboring visible image measurement sensors based on the multilevel transformation matrixes, Around the vehicle detection system.
The method of claim 3,
The photometric alignment correction,
Analyzing photometric alignment of the visible image data in the overlap region between neighboring visible image measurement sensors,
Wherein the second correction data is calculated by applying a photometric correction parameter set to minimize a difference in brightness and hue in the overlap region to visible image data.
The method of claim 3,
Wherein the image registration is performed by:
Based on the first correction data and the second correction data,
A feature extraction and matching is performed between the visible image and the thermal image or the visible image, a final projection transformation matrix is obtained, and a surround visible image is calculated through the warping and blending of the visible images.
The method according to claim 1,
In the image fusion,
The coefficients of the visible image data I _v (x, y) and the thermal image data I _M (x, y) are converged after the wavelet transform W by the convergence law Φ, Is reconstructed into a surround fused image image (I _F (x, y)) by a transform (W ^-1 ), and is expressed by the following equation (1).
[Formula 1]

Here, C _Y and C _M are the wavelet coefficients of the visible image data I _v (x, y) and the thermal image data I _M (x, y), and C _F is the wavelet coefficient after the fusion rule? Where a and d are approximate and detailed coefficients, respectively, and N is the size of the image in the y direction.
The method according to claim 1,
Further comprising a plurality of ultrasonic measurement sensors disposed at predetermined intervals around the vehicle,
The control unit converts a point P (X, Y, Z) on the world coordinate system to a point P _img (x, y) on the surround visible image plane through tracking of the ultrasonic measurement sensor, Wherein the information is provided to the vehicle.
The method according to claim 1 or 8,
Connected to the control unit,
An information providing unit for providing at least one selected one of the surround visible image information, the surround fusion image information, the position information of the obstacle around the vehicle, the surround visual image, or the image information of the obstacle in the surround fusion image and the auditory information on the obstacle Wherein the vehicle surroundings detection system comprises:
10. The method of claim 9,
Wherein the information providing unit comprises at least one display unit selected from the group consisting of a flat display and a headset mount display.
The method according to claim 1,
A connecting portion having one end detachably connected to the roof portion of the vehicle,
And a mounting plate coupled to the other end of the connection portion, wherein the plurality of visible image measurement sensors and the plurality of thermal image measurement sensors are mounted.
12. The method of claim 11,
Wherein the connection portion or the connection portion and the mount plate are formed of a magnetic frame and are magnetically coupled to the vehicle.
Providing a plurality of visible image data measured at predetermined intervals over the entire area around the vehicle at an upper portion of the vehicle;
Providing a plurality of thermal image data measured over a whole area around the vehicle at predetermined intervals; And
And providing a surround fused image for a surround area around the vehicle through the image fusion of the measured visible and deteriorated images.

Images