KR20170095005A - Image Sensor Apparatus using Sensor Fusion and Control Method thereof - Google Patents

Abstract

The present invention relates to an image sensor device and a control method which use a sensor fusion to control a setting of an interest region of an image sensor, and control an image parameter of the image sensor within the interest region to improve an object recognition ability. In the present invention, disclosed is an image sensor device using a sensor fusion, comprising: a heterogeneous sensor which recognizes a forward object for sensing an object in the forward of a vehicle, and gains information on a distance to the object and/or information on a track of the object; an image sensor for gaining a forward image; and a sensor control unit which controls the image sensor on the basis of the information on the distance to the object and/or information on the track of the object received from the heterogeneous sensor. According to the present invention, cooperation of the heterogeneous sensor by means of a sensor fusion improves an object recognition in the image sensor. In addition, by applying the present invention to AES or SCC, more convenient and safe vehicle system can be implemented.

Description

TECHNICAL FIELD [0001] The present invention relates to an image sensor device using a sensor fusion,

The present invention relates to an image sensor apparatus and a control method for sensing a forward object, and more particularly, to a system and method for controlling an object sensor using a sensor fusion to control a region of interest of an image sensor, And an image sensor device and a control method thereof.

Recently, various devices have been developed for the safe operation of vehicles. Among them, autonomous emergency braking (braking) which informs the driver of the risk of collision when the risk of collision is anticipated and the brakes are operated when the collision is predicted without the driver's response. AEB) or an adaptive cruise control (SCC) device that detects the vehicle or object ahead and automatically controls the speed of the vehicle while controlling the operation of the vehicle.

In the case of such an AEB or SCC, a device capable of recognizing a preceding vehicle is indispensable. For this purpose, an apparatus and method for detecting an object by fusion of two sensors using a radar and a camera are proposed. Korean Patent Laid-Open Publication No. 10-2014-0123270 (published on October 22, 2014) discloses a radar sensor and a method and system for detecting an object using the radar device and the image mapping. The radar sensor unit receives a radio signal reflected by at least one object. There is provided an object sensing apparatus for generating sensing information on at least one object through mapping of an object detected by a wireless signal to an image, and an image processing unit for generating an image of a predetermined space including at least one object, have.

However, in order to increase the possibility of recognizing an object through sensor fusion, it is necessary to image the periphery of an object such as a camera. In the conventional system, the image sensor sets the ROI area based on the vanishing point of the front road, There is a problem that control of the image sensor through the sensor fusion can not be improved.

Korean Patent Registration No. 10-2014-0123270 (Publication date: October 22, 2014)

An object of the present invention is to provide an image sensor apparatus and a control method using a sensor fusion which improves recognition performance by using an object recognition result obtained from a heterogeneous sensor in an image sensor.

According to one aspect of the present invention, there is provided an image sensor device using sensor fusion for sensing an object existing in front of a vehicle, the sensor device recognizing an object in front of the sensor and detecting distance information to the object and / A method of controlling an image pickup apparatus, comprising: a heterogeneous sensor for acquiring track information of an object; an image sensor for acquiring an image in front of the image; and a controller for controlling the image sensor based on distance information from the heterogeneous sensor to the object and / And a sensor control unit for controlling the sensor.

Here, the heterogeneous sensor may be a radar or a lidar sensor, and the image sensor may be a CMOS sensor or a CCD camera.

Further, the sensor control unit can set the position and size of the region of interest of the image sensor based on distance information from the heterogeneous sensor to the object and / or track information of the object, and furthermore, If the distance to the object is greater than the distance to the object in the previous frame and the distance from the heterogeneous sensor to the object is closer to the object in the previous frame, The area size can be enlarged.

Also, the sensor control unit may analyze the type of the object from the track information of the object coming from the heterogeneous sensor, set the size of the region of interest of the image sensor based on the distance information to the object and the type of the object, When the type of object is a pedestrian, a pedestrian reference margin is added to the size of the ROI, and when the type of the object is a vehicle, a vehicle reference margin can be added to the ROI size.

In addition, when the heterogeneous sensor recognizes an object but the object does not recognize an object in the current frame of the image sensor, the sensor control unit can adjust the image sensing parameter of the image sensor, which includes a histogram And the histogram of the region of interest of the current frame is more dark than the histogram of the region of interest of the previous frame, the image sensing parameter of the image sensor may be adjusted based on the result of the edge analysis. The exposure of the image sensor may be made larger and the exposure of the image sensor may be made smaller if there are many bright portions.

According to another aspect of the present invention, there is provided a method of controlling an image sensor of an image sensor device using sensor fusion for sensing an object existing in front of a vehicle, Acquiring distance information of the object and / or track information of the object, controlling the image sensor based on distance information to the object and / or track information of the object And acquiring an image of the front using the image sensor.

Here, the step of controlling the image sensor based on the distance information to the object and / or the track information of the object may include a step of controlling the image sensor based on the distance information to the object and / Setting the position and size of the region of interest of the sensor, and setting the position and size of the region of interest of the image sensor based on the distance information to the object and / or the track information of the object If the distance to the object is smaller than the distance to the object in the previous frame, and if the distance from the heterogeneous sensor to the object is closer to the object in the previous frame, And enlarging the area of interest, and may further include a step of acquiring distance information to the object and / or track information of the object The step of controlling the image sensor with the water comprises the steps of analyzing the type of the object from the track information of the object and setting the size of the region of interest of the image sensor based on the distance information to the object and the type of the object . The step of setting the size of the region of interest of the image sensor based on the distance information to the object and the type of the object may include adding a pedestrian reference margin to the size of the region of interest when the type of the object is a pedestrian, And adding a vehicle reference margin to the region of interest if the type of the vehicle is a vehicle.

According to another aspect of the present invention, there is provided a method of controlling an image sensor using an image sensor using sensor fusion for sensing an object existing in front of a vehicle, Recognizing the object, recognizing the object in the current frame using the image sensor, and adjusting the imaging parameter of the image sensor when the image sensor fails to recognize the object.

Wherein adjusting the imaging parameter of the image sensor may include adjusting an imaging parameter of the image sensor based on a histogram and an edge analysis of the region of interest of the current frame, Wherein the adjusting the image pickup parameter of the image sensor based on the histogram and the edge analysis on the ROI of the current frame includes adjusting a histogram of the ROI of the current frame to a portion May include adjusting the imaging parameters of the image sensor so as to increase the exposure of the image sensor in many cases and to reduce the exposure of the image sensor in the presence of a large number of bright portions.

According to the present invention, there is an effect of enhancing the object recognizing ability in the image sensor due to the cooperation of the heterogeneous sensor by the sensor fusion. By applying this to AEB or SCC, it is possible to implement a more convenient and safe vehicle system.

1 is a block diagram of an image sensor control apparatus using a sensor fusion according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a method of setting a region of interest of the image sensor 100 using a sensor fusion performed by the sensor control unit 300 according to an exemplary embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of setting a size of a region of interest based on a type of object and distance information by the sensor control unit 300 according to an embodiment of the present invention.
4 is a flow chart illustrating a method for changing imaging parameters of an image sensor 100 based on histogram and edge analysis of a region of interest in accordance with an embodiment of the present invention.

In the following description, well-known functions or constructions are not described in detail to avoid unnecessarily obscuring the subject matter of the present invention.

Although the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, But should be understood to include all modifications, equivalents, and alternatives.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description with reference to the accompanying drawings, the same or corresponding components will be denoted by the same reference numerals, and redundant description thereof will be omitted.

1 is a block diagram of an image sensor control apparatus using a sensor fusion according to an embodiment of the present invention.

Referring to FIG. 1, an image sensor control apparatus using sensor fusion according to an embodiment of the present invention includes an image sensor 100 for obtaining an image in front of a vehicle, a heterogeneous sensor 200 for measuring a distance to an object in front And a sensor control unit 300 for controlling the image sensor based on positional information, distance information, and / or object track information from the heterogeneous sensor 200.

Here, object track information can be referred to as information indicating a position where an object has passed, and the speed and direction of an object can be detected using the information.

The heterogeneous sensor 200 may be a radar system or a lidar system for recognizing an object in front and measuring an approximate distance or relative position from the vehicle to the object. The heterogeneous sensor 200 may transmit object track information, position information, and / or distance information to the sensor control unit 300.

The image sensor 100 may be a CMOS sensor or a CCD camera as an apparatus for acquiring an image of the front of the vehicle. The image sensor 100 may capture an image of a region of interest (ROI) based on a control signal from the sensor controller 300 and transmit the image to the sensor controller 300.

The sensor control unit 300 is a device that controls the image sensor 100 using a sensor fusion in order to increase the recognition performance of an object. The sensor control unit 300 can increase the object recognition capability by the image sensor by setting the region of interest of the image sensor 100 using object track information, position information, and / or distance information from the heterogeneous sensor 200 . Also, when the brightness around the vehicle suddenly changes, such as during a tunnelled exit / exit section, it may be difficult to distinguish an object from the image sensor 100 even though the object is detected by the heterogeneous sensor 200. At this time, the sensor control unit 300 can increase the object recognition capability by extracting and analyzing the histogram and the edge data in the area of interest with respect to the image data of two or more frames and controlling the image sensor 100 differently.

FIG. 2 is a flowchart illustrating a method of setting a region of interest of the image sensor 100 using a sensor fusion performed by the sensor control unit 300 according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the sensor control unit 300 first obtains object track information and / or distance information from the heterogeneous sensor (S210) in order to set a region of interest of the image sensor 100. FIG. Then, the type of the object is analyzed from the acquired object track information (S220). The type of object may be a pedestrian or a vehicle. The location and size of the region of interest can be set (S230) based on the type and distance information of the analyzed object.

FIG. 3 is a flowchart illustrating a method of setting a size of an ROI based on the type and distance information of an object according to an embodiment of the present invention.

Referring to FIG. 3, the sensor control unit 300 compares the distance information from the heterogeneous sensor with the distance information in the previous frame (S310). If the distance information of the current frame is farther, (S330). If the distance information of the current frame is closer, the size of the region of interest is enlarged (S320) than the size of the previous frame. If the distance is the same, the size of the region of interest set in the previous frame is maintained do. If the object is a pedestrian according to the type of the object analyzed based on the object track information, a pedestrian margin is added to the set area of interest (S360). If the object is a vehicle, a vehicle margin is added to the set area of interest S350). If the object is other objects, the margin may not be added to the set size of the region of interest. Since the moving speed of the vehicle and the pedestrian is different, it is possible to increase the recognizing ability of the object by adding an appropriate size margin to the speed. Generally, the vehicle reference margin may be larger than the pedestrian reference margin.

Referring again to FIG. 2, after setting the size of the ROI based on the type of the object and the distance information, a reference position of the ROI analyzed by the object track information is set. At this time, if it is the same as the reference position of the previous frame, it is not necessary to change it, but if it is different from the reference position of the previous frame, it can be set again according to the reference position in the current frame.

As described above, by using the object track information and the distance information acquired from the heterogeneous sensor 200, the region of interest of the image sensor 100 is precisely set to match the position of the object, The recognition ability can be enhanced.

In addition, when the ambient brightness of the vehicle suddenly changes, such as the tunnel entrance / exit, the sensor control unit 300 may recognize an object in the heterogeneous sensor 200, but the object may not be recognized in the image sensor 100, The recognition performance of the object is improved by adjusting the imaging parameters including the gain or exposure of the image sensor 100 by referring to the histogram or the edge analysis result of the region of interest . As an example, if the histogram of the current frame is more dark than the histogram of the previous frame due to the vehicle entering the tunnel, the exposure of the image sensor 100 can be changed to be larger. On the other hand, when the vehicle comes out of the tunnel, the brightness of the histogram increases, so that the exposure of the image sensor 100 can be changed to be smaller.

4 is a flow chart illustrating a method for changing imaging parameters of an image sensor 100 based on histogram and edge analysis of a region of interest in accordance with an embodiment of the present invention.

Referring to FIG. 4, the sensor control unit 300 first determines whether the heterogeneous sensor 200 recognizes an object (S410). If it is not recognized, it is not necessary to proceed further. If the object is recognized, it is determined whether the object is recognized in the current frame of the image sensor 100 (S420). If the object is recognized and the object is not recognized, the heterogeneous sensor 200 recognizes that the object is recognized because the object is recognized, so that the histogram analysis in the region of interest is performed in order to improve the recognition performance (S430). A histogram represents the distribution of pixels from the darkest part to the brightest part of the image. If the darkness of the entire image is large, the left part of the histogram becomes high, and if there are many bright parts, the right part of the histogram becomes high. Therefore, if the brightness around the vehicle suddenly changes, such as tunnel entrance / exit, there may be a large difference in frame histogram. Histogram analysis shows that the brightness changes rapidly.

In operation S440, it is determined whether the histogram of the current frame is different from the histogram of the previous frame through the histogram analysis in the region of interest. If the histogram is different, the image sensor parameter is changed in operation S450. More specifically, when the vehicle enters the tunnel, the exposure of the image sensor 100 can be greatly changed if the histogram of the current frame becomes more dark than the histogram of the previous frame through the histogram analysis. On the other hand, when the vehicle comes out of the tunnel, the bright portion of the histogram increases, so that the exposure of the image sensor 100 can be changed to be small. In addition, the object recognition ability can be increased by increasing or decreasing the gain of the image.

After the parameter is changed, edge analysis of the image coming from the image sensor 100 is performed (S460). Edge analysis shows that there are heterogeneous objects within the area of interest. Also, it can be confirmed through edge analysis that vehicles or pedestrians traveling from the front in general road environment. If it is determined that there is a heterogeneous object as a result of the edge analysis, it is recognized that there is a vehicle or a pedestrian in the front (S470). If the difference is less than the reference value, the image sensor parameter is changed again (S450) .

Referring to the method of the present invention described above, the object recognition capability of the image sensor 100 can be enhanced by using the heterogeneous sensor 200. [

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims and their equivalents. Only. It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

100: Image sensor
200: heterogeneous sensor
300:

Claims (18)

1. An image sensor apparatus using a sensor fusion for sensing an object existing in front of a vehicle,
A heterogeneous sensor for recognizing an object in front and obtaining distance information to the object and / or track information of the object;
An image sensor for acquiring an image of a front side; And
A sensor control unit for controlling the image sensor based on distance information from the heterogeneous sensor to the object and / or track information of the object;
And an image sensor device using the sensor fusion. 2. The hybrid type sensor according to claim 1,
A radar or lidar sensor,
Image sensor device using sensor fusion. 2. The image sensor according to claim 1,
CMOS sensor or CCD camera,
Image sensor device using sensor fusion. The apparatus as claimed in claim 1,
Setting a position and a size of a region of interest of the image sensor based on distance information from the heterogeneous sensor to the object and / or track information of the object,
Image sensor device using sensor fusion. 5. The apparatus according to claim 4,
If the distance from the heterogeneous sensor to the object is greater than the distance to the object in the previous frame,
And enlarging the size of the region of interest if the distance from the heterogeneous sensor to the object is closer to the object in the previous frame,
Image sensor device using sensor fusion. The apparatus as claimed in claim 1,
Analyzing the type of the object from the track information of the object coming from the heterogeneous sensor,
Setting a size of an area of interest of the image sensor based on distance information to the object and a type of the object;
Image sensor device using sensor fusion. 7. The apparatus of claim 6,
Wherein when the type of the object is a pedestrian, a pedestrian reference margin is added to the size of the region of interest,
Adding a vehicle reference margin to the region of interest if the type of the object is a vehicle,
Image sensor device using sensor fusion. The apparatus as claimed in claim 1,
When the heterogeneous sensor recognizes an object but fails to recognize an object in the current frame of the image sensor,
Image sensor device using sensor fusion. 9. The apparatus according to claim 8,
Wherein adjusting the imaging parameter of the image sensor comprises adjusting an imaging parameter of the image sensor based on a histogram and / or edge analysis of the region of interest of the current frame
Image sensor device using sensor fusion. 10. The apparatus according to claim 9,
The exposure of the image sensor is made larger when the histogram of the current frame is darker than the histogram of the region of interest of the previous frame, Adjusting an image pickup parameter of the image sensor so as to make it smaller,
Image sensor device using sensor fusion. A method of controlling an image sensor of an image sensor device using a sensor fusion for sensing an object existing in front of a vehicle,
Recognizing an object in front by using a heterogeneous sensor and obtaining distance information to the object and / or track information of the object;
Controlling an image sensor based on distance information to the object and / or track information of the object; And
Acquiring an image of the front using the image sensor;
Wherein the image sensor control method comprises: 12. The method of claim 11,
The step of controlling the image sensor based on the distance information to the object and / or the track information of the object may include:
And setting a position and a size of a region of interest of the image sensor based on distance information to the object and / or track information of the object.
Image sensor control method. 13. The method of claim 12,
The step of setting the position and size of the ROI of the image sensor based on the distance information to the object and / or the track information of the object may include:
If the distance to the object is greater than the distance to the object in the previous frame and the distance from the heterogeneous sensor to the object is closer to the object in the previous frame, And enlarging the area size.
Image sensor control method. 12. The method of claim 11,
The step of controlling the image sensor based on the distance information to the object and / or the track information of the object may include:
Analyzing the type of the object from the track information of the object; And
Setting the size of the region of interest of the image sensor based on the distance information to the object and the type of the object.
Image sensor control method. 15. The method of claim 14,
The step of setting the size of the region of interest of the image sensor based on the distance information to the object and the type of the object,
Adding a pedestrian reference margin to the size of the region of interest if the type of the object is a pedestrian and adding a vehicle reference margin to the size of the region of interest if the type of the object is a vehicle.
Image sensor control method. A method of controlling an image sensor of an image sensor device using a sensor fusion for sensing an object existing in front of a vehicle,
Recognizing an object in front by using a heterogeneous sensor;
Recognizing the object in a current frame using an image sensor; And
Adjusting imaging parameters of the image sensor when the image sensor fails to recognize the object;
Wherein the image sensor control method comprises: 17. The method of claim 16,
Wherein adjusting the imaging parameter of the image sensor comprises:
Adjusting an imaging parameter of the image sensor based on a histogram and an edge analysis result for a region of interest of the current frame,
Image sensor control method. 18. The method of claim 17,
Wherein adjusting an image pickup parameter of the image sensor based on a histogram and an edge analysis result on a region of interest of the current frame,
The exposure of the image sensor is made larger when the histogram of the current frame is darker than the histogram of the region of interest of the previous frame, And adjusting an imaging parameter of the image sensor so as to be smaller.
Image sensor control method.

Images