KR102031635B1 - Collision warning device and method using heterogeneous cameras having overlapped capture area - Google Patents

Abstract

Disclosed are a collision warning apparatus and method using a heterogeneous camera having an overlapping photographing area. The collision warning device includes a wide angle camera unit provided in a vehicle to generate a wide angle image including one or more wide angle cameras; A narrow angle camera unit provided in the vehicle to generate a narrow angle image including one or more narrow angle cameras; And detecting a lane from at least one of the wide-angle image and the narrow-angle image, calculating a lane curvature value of the detected lane, and when the lane curvature value is less than or equal to a predetermined first threshold, And a control unit that selects a base image for generation and selects a narrow angle image as the base image when the lane curvature value is greater than the first threshold value.

Description

Collision warning device and method using heterogeneous cameras having overlapped capture area}

The present invention relates to a collision warning device and method using a heterogeneous camera having an overlapping photographing area.

In general, a driver's clock inside a vehicle is mainly forward, and the driver's left and right and rear clocks are largely obscured by the vehicle body and thus have very limited visibility.

In order to solve this problem, a clock aid means such as a side mirror is used, and recently, a technique including a camera means for photographing an external image of a vehicle and providing the driver to a vehicle is being applied to a vehicle.

Among them, an around view monitoring system (hereinafter, referred to as an AVM system) that installs a plurality of cameras in a vehicle and displays a 360-degree omnidirectional image around the vehicle.

The AVM system combines images around the vehicle taken by cameras at each location to provide a top view image of the driver looking at the vehicle from the sky, displaying obstacles around the vehicle and blind spots. There is an advantage that can be solved.

In addition, recently, a narrow angle camera system has been additionally provided so that the driver can effectively recognize the road situation in front of the vehicle.

However, the AVM system and the narrow angle camera system, which are provided in the vehicle, are provided only to perform individual operations, respectively, and have limitations in that they cannot function complementarily to detect an object present in the driving direction and recognize a collision possibility. there was.

Korean Patent No. 10-1739394 (Object distance estimation device and method using a stereo camera of different angle of view) US Patent No. 8,633,810 (Rear-view multi-functional camera system) US Patent Publication No. 2017/0113611 (Method for stereo map generation with novel optical resolutions)

B-spline-based road model for 3D lane recognition (IEEE conference on intelligent transportation system October 2010) Lane and Curvature Detection Algorithm Based on Curve Template Matching Using Top View Image (Nov. 2010, Journal of the Institute of Electronics Engineers of Korea, Vol. 47, SP No. 6)

According to the present invention, a heterogeneous camera having an overlapping photographing area capable of utilizing an image optimal for driving conditions of a vehicle by selecting an appropriate image from a wide-angle image and a narrow-angle image in consideration of a path curvature and a curvature of a lane in operation. It is to provide a collision warning device and method using.

The present invention can increase the amount of information included in the overlap area by considering images captured by heterogeneous cameras having different view angle ranges together, thereby allowing heterogeneous cameras having an overlap shooting area to improve object detection accuracy. It is to provide a collision warning device and method using.

Other objects of the present invention will be readily understood through the following description.

According to an aspect of the present invention, a wide-angle camera unit provided in a vehicle to generate a wide-angle image including one or more wide-angle camera; A narrow angle camera unit provided in the vehicle to generate a narrow angle image including one or more narrow angle cameras; And detecting a lane from at least one of the wide-angle image and the narrow-angle image, calculating a lane curvature value of the detected lane, and when the lane curvature value is less than or equal to a predetermined first threshold, A collision warning device is provided that includes a control unit that selects a base image for generation and selects a narrow angle image as the base image when the lane curvature value is greater than the first threshold value.

The control unit selects one image for calculating a lane curvature value to be compared with the first threshold value according to a predetermined condition, wherein the predetermined condition is previously designated as one of a wide angle image and a narrow angle image, or the vehicle moves. One of the wide-angle image and the narrow-angle image may correspond to the speed, or the wide-angle image may be designated as the wide-angle image, and may be changed to the narrow-angle image only when the curvature value of the lane detected in the wide-angle image is greater than or equal to a predetermined threshold value. .

The control unit may generate a depth map of an overlapping photographing area of the wide angle camera and the narrow angle camera by using the wide angle image and the narrow angle image, and may change a depth change of an object in a depth map continuously temporally. The path curvature value corresponding to the movement trajectory of the vehicle may be calculated.

If the lane is not detected in the wide-angle image and the narrow-angle image, the control unit selects the wide-angle image as a base image for generating collision prediction information when the path curvature value is equal to or less than a second predetermined threshold value. When the lane curvature value is larger than the second threshold value, the narrow angle image may be selected as the base image.

One or more wide-angle cameras and one or more narrow-angle cameras are installed such that an overlapping shooting area exists in each shooting area, and the control unit generates the collision prediction information for objects existing in the overlapping shooting area and other shooting areas of the base image. The overlapping photographing area may generate collision prediction information using an overlapping area image generated by combining the auxiliary image, which is one of the wide-angle image and the narrow angle image, which is not selected as the base image, and the base image.

The overlap region image may be generated as a depth map.

The collision prediction information is generated to include at least one of a separation distance from the object and the remaining time until the collision with the object, the control unit through the output unit if the collision prediction information is less than a third predetermined threshold value; The alarm can be processed to be triggered.

The control unit may determine whether there is a collidable object in at least one of the wide-angle image and the narrow-angle image, and calculate the lane curvature value and select the base image only when there is a collidable object.

The wide angle camera may be a camera for an around view monitoring system.

According to another aspect of the present invention, in the collision warning method performed in the collision warning device, (a) a lane in at least one of the wide angle image generated by the wide angle camera and the narrow angle image of the narrow angle image generated by the narrow angle camera. Detecting and calculating lane curvature values of the detected lanes; (b) comparing the calculated lane curvature value with a first predetermined threshold value; And (c) selecting the wide-angle image as a base image for generating collision prediction information when the lane curvature value is less than or equal to the first threshold value, and when the lane curvature value is larger than the first threshold value, narrow-angle image is selected. There is provided a collision warning method comprising the step of selecting as the base image.

The step (b) is performed on the lane curvature value of the selected image according to a predetermined condition, wherein the predetermined condition is previously designated as one of a wide angle image and a narrow angle image, or the wide angle image to correspond to the moving speed of the vehicle. And the narrow angle image, or the wide angle image may be changed to the narrow angle image only when the curvature value of the lane detected in the wide angle image is greater than or equal to a predetermined threshold value.

The collision warning method may further include generating a depth map of an overlapping photographing area between the wide angle camera and the narrow angle camera using the wide angle image and the narrow angle image; And calculating a path curvature value corresponding to a moving trajectory of the vehicle using a change in depth of an object in a depth map continuously generated in time. However, if the lane is not detected in both the wide-angle image and the narrow-angle image in the step (a), when the path curvature value is less than or equal to a second predetermined threshold, the wide-angle image is selected as the base image, and the path curvature. If the value is larger than the second threshold value, the narrow angle image may be preset to be selected as the base image.

According to a predetermined determination method, steps (a) to (c) may be performed only when it is determined that an object present in at least one of the wide-angle image and the narrow-angle image is a collidable object.

The collision warning method may include: generating an overlap region image of an overlapping photographing area between the base image and the auxiliary image by synthesizing the auxiliary image, which is one of the wide-angle image and the narrow angle image, which is not selected as the base image, and the base image; ; And generating collision prediction information about an object present in the overlap region image and the other photographing region of the base image. Here, the collision prediction information may be generated to include at least one of a separation distance from the object and a remaining time until the collision with the object. The overlap region image may be generated as a depth map.

The collision warning method may include: comparing the collision prediction information with a third predetermined threshold value; And processing the alarm through an output unit when the collision prediction information is equal to or less than the third threshold value.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, by selecting an appropriate image from a wide angle image and a narrow angle image as a basic image in consideration of the path curvature of the vehicle and the curvature of the lane, it is possible to utilize an image that is optimal for driving conditions of the vehicle. have.

In addition, by considering the images captured by the heterogeneous cameras having different viewing angles together, the amount of information included in the overlap area can be increased, thereby improving the object detection accuracy.

1 is a block diagram of a collision warning device according to an embodiment of the present invention.
2 is a view for explaining a captured image of the wide-angle camera unit and the narrow-angle camera unit according to an embodiment of the present invention.
3 and 4 are views for explaining a synthesis method of a wide-angle image and a narrow-angle image according to an embodiment of the present invention.
5 is a view for explaining a path curvature calculation method using an overlap region image according to an embodiment of the present invention.
6 and 7 are views for explaining a lane curvature calculation method according to an embodiment of the present invention.
8 is a diagram for describing a primary image and an auxiliary image determination technique according to various conditions according to an embodiment of the present invention.
9 to 11 are flowcharts showing a collision warning method according to an embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, terms such as a first threshold value, a second threshold value, and the like, which will be described later, may be pre-specified as threshold values that are substantially different or partially the same value, but may be confused when expressed with the same word. For the sake of convenience, the terms "first" and "second" will be written together for convenience of classification.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

In addition, the components of the embodiments described with reference to the drawings are not limited to the corresponding embodiments, and may be implemented to be included in other embodiments within the scope of the technical spirit of the present invention. Even if the description is omitted, it is obvious that a plurality of embodiments may be reimplemented into one integrated embodiment.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same or related reference numerals and redundant description thereof will be omitted. In the following description of the present invention, if it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a block diagram of a collision warning device according to an embodiment of the present invention, Figure 2 is a view for explaining the captured image of the wide-angle camera unit and the narrow-angle camera unit according to an embodiment of the present invention. 3 and 4 are views for explaining a synthesis method of a wide-angle image and a narrow image according to an embodiment of the present invention, Figure 5 is a path curvature using an overlap region image according to an embodiment of the present invention ) Is a view for explaining the calculation method. 6 and 7 illustrate a method of calculating a lane curvature according to an embodiment of the present invention, and FIG. 8 illustrates determination of a primary image and an auxiliary image according to various conditions according to an embodiment of the present invention. 9 to 11 are flowcharts illustrating a collision warning method according to an embodiment of the present invention.

Referring to FIG. 1, the collision warning device 100 may include a wide angle camera unit 110, a narrow angle camera unit 120, a control unit 130, and an output unit 140.

Each of the wide-angle camera unit 110 and the narrow-angle camera unit 120 may be controlled by the controller 134 of the control unit 130 to generate the wide-angle image or the narrow-angle image in a predetermined condition and / or in real time.

The wide-angle camera unit 110 may include one or more wide-angle cameras, and a processor configured to generate a captured image (ie, a wide-angle image) according to a shooting operation of each wide-angle camera. The wide-angle image generated by the processor may be stored in a storage unit (not shown) for use in the control unit 130 or the like.

The wide-angle cameras are, for example, cameras for AVM systems, which are wide-angle cameras 211, 213, 215, and 217 respectively installed in a plurality of places of the vehicle (eg, positions designated to photograph front, rear, left, and right sides, respectively). It may be.

The wide-angle cameras 211, 213, 215, and 217 are, for example, wide-angle lenses having an angle of view of about 180 degrees (or about 180 degrees) so that a wide range of areas close to the vehicle can be taken as illustrated in FIG. Fisheye lens) is used. Wide-angle images generated by the wide-angle camera are illustrated in FIGS. 2A and 2B, respectively.

The narrow angle camera unit 120 may include one or more narrow angle cameras and a processor configured to generate a captured image (ie, a narrow angle image) according to a photographing operation of each narrow angle camera. The narrow angle image generated by the processor may be stored in a storage unit (not shown) for use in the control unit 130 or the like.

The narrow angle camera 240 is, for example, a camera in which a narrow angle lens having an angle of view of 30 to 50 degrees is used, so that the photographing area can be an area far from the vehicle as compared to the wide angle camera as illustrated in (d) of FIG. 2. to be. The captured image generated by the narrow angle camera is illustrated in Fig. 2C.

In FIG. 2 (d), the case where one narrow angle camera 240 for photographing the front of the vehicle is provided is illustrated, but one or more narrow angle cameras may be further provided to photograph another direction such as the rear of the vehicle. Of course.

The control unit 130 may include an image processor 132 and a controller 134.

The image processor 132 determines whether there is an object colliding with the object photographed in the wide-angle image and the narrow-angle image. When the image processor 132 determines that the collidable object exists, the image processor 132 generates a depth map using the wide-angle image and the narrow-angle image. The path curvature value is calculated using the generated depth map.

In addition, the image processor 132 recognizes lanes around the vehicle by using the wide-angle image and / or the narrow-angle image, calculates curvature values of the lanes, and selects the primary image and the auxiliary image, respectively.

In addition, the image processor 132 generates collision prediction information with an object existing in an overlap region between the primary image and the secondary image and other regions of the primary image. The collision prediction information may include one or more of a distance from the object, a remaining time until the collision, and the like, as described below.

Hereinafter, the operation of the image processor 132 will be described in detail.

First, the image processor 132 recognizes an object photographed in the wide angle image generated by the wide angle camera and the narrow angle image generated by the narrow angle camera, and determines whether there is a possibility of collision with the recognized object.

For example, the image processor 132 is an object having a change that increases in size when the photographed object refers to a wide angle image and / or a narrow angle image generated in real time, and the object is provided with the collision warning device 100. It may be set in advance to determine whether the object is a colliding object in consideration of whether it exists in the same lane (for example, a lane, a parking partition line, etc.) as the own vehicle.

Here, the lane may be detected by an image processing technique such as edge detection in the wide-angle image and / or the narrow-angle image, which is obvious to those skilled in the art, and thus description thereof will be omitted. However, in the case of the wide-angle image, a process of converting the image into the top view image may be performed to enable accurate lane detection and object detection.

For example, the image processor 132 may determine that there is a possibility of collision if an object existing in the wide-angle image and / or the narrow-angle image is changed so that its size continuously increases in the direction toward the vehicle. However, if the size of the object is gradually reduced or the size increases but the size increases in the direction away from the vehicle, it may be determined that there is no possibility of collision.

If it is determined that the object is collidable, the image processor 132 may generate a depth map using the wide-angle image and the narrow-angle image, and calculate a path curvature value using the generated depth map.

As illustrated in FIGS. 3A and 3B, the wide-angle image 310 is generated to easily detect an object within a relatively short range, and the narrow-angle image 320 is a relatively long distance object. It is created to be easy to detect. In this case, the wide-angle camera and the narrow-angle camera are respectively installed in the vehicle such that the overlap photographing area 260 exists between the wide-angle image 310 and the narrow-angle image 320 (see FIG. 2 (d)).

The image processor 132 generates a depth map by using a wide angle camera and a narrow angle camera as a stereo vision using a stereo mapping method. That is, the image processor 132 may use a wide angle image 310 and a narrow angle image 320 generated by each of the wide angle camera and the narrow angle camera to obtain information about a depth of each object in the image. 330 is generated (see FIG. 4).

Since the shapes of the wide-angle image 310 and the narrow-angle image 320 are different from each other, the wide-angle image 310 is narrowed to the narrow-angle image 320 by a pixel mapping method using a look-up table (LUT) or the like to generate the depth map 330. Of course, the conversion to the same or similar format (for example, planar image format) may be preceded. A detailed method of generating the depth map 330 is described in detail in US Patent No. 8,633,810, US Publication No. 2017/0113611, and so on, the description thereof will be omitted.

In addition, there are various methods of mapping and synthesizing pixels of the wide-angle image 310 and the narrow-angle image 320. For example, a synthesis process for generating the depth map 330 may be performed using Equations 1 to 3 below. It may be.

For reference, Equation 1 is to calculate the H matrix of the wide angle camera (ie, H ₁ ) to obtain the relationship between the flat floor of the real world and the wide angle camera, and Equation 2 is the H matrix of the narrow angle camera (ie, H _2). ) To calculate the relationship between the flat floor of the real world and the narrow angle camera.

Here, x and y are pixel coordinates in the captured image, x 'and y' represent real world coordinates (measurement coordinates) for distances, positions, etc. spaced from the vehicle, and w is a predetermined weight. Accordingly, unknown matrices H ₁ and H ₂ may be calculated by Equations 1 and 2, respectively.

In addition, the calculated H ₁ and H ₂ are calculated based on the same object (that is, the flat bottom), and the wide angle image by the wide angle camera and the narrow angle image by the narrow angle camera have a relationship as in Equation 3 below.

Accordingly, an arbitrary pixel position P _{Narrow of the narrow} angle image may be projected to one pixel position P _Wide of the wide angle image by Equation 3, and the image processor 132 uses this relationship to narrow the wide angle image and the narrow angle. An overlap region image obtained by synthesizing the overlap region 260 of the image may be generated. Similarly, it is obvious that one pixel position of the wide angle image may be projected to one pixel position of the narrow angle image. In this case, the wide-angle image may be mapped to each pixel of the narrow-angle image after a process of correcting a wide-angle image in the form of an image captured by a conventional lens using a predetermined lookup table (LUT) is preceded.

The image processor 132 calculates a path curvature value according to the driving of the vehicle using the generated depth map 330.

As illustrated in FIG. 5, the driving of the vehicle may be performed in the direction of forward straight, forward right turn, forward left turn, backward straight, backward right turn and backward right turn.

In each case, the depth map 330 used by the image processor 132 to calculate a path curvature value according to the driving of the vehicle is a composite image generated to correspond to the forward or backward direction. That is, the depth map 330 is a depth map of the wide-angle image and the narrow-angle image in which the front of the vehicle is photographed when the vehicle is driving in the forward direction, and the wide-angle image and the narrow-angle image of the rear of the vehicle is photographed when the vehicle is driving in the reverse direction. Depth map.

In this case, the information about whether the vehicle is moving forward or backward may be recognized by the control unit 130, for example, provided from one or more of a vehicle control system (eg ECU), a GPS system, a geomagnetic sensor, and the like. The processor 132 may generate the depth map 330 using an appropriate image.

The wide-angle image 310 and the narrow-angle image 320 are generated in real time while the vehicle is moving in an arbitrary direction, and the image processor 132 generates the depth map 330 in real time. The image processor 132 may calculate a path curvature value according to the driving of the vehicle by using a depth value change (ie, a change in relative distance) of an object in the depth map 330 generated in real time.

For example, when the vehicle moves in the direction ⓐ while the depth map 330 such as ⓓ of FIG. 5 is generated, the image processor 132 has no difference within the error range for the objects ①, ②, and ③ in front of it. You will notice a change in depth that recognizes a common approach.

However, when the vehicle moves in the ⓑ direction, the image processing unit 132 becomes the deepest (ie, relatively farthest) of the object ① in front of it, and the shallowest (ie, relatively closest) to ③. It can be recognized that the depth change occurs in each object in the direction of).

That is, the image processor 132 may recognize the movement and the trajectory of the vehicle by using the depth change of each object and the position change indicated by the movement of each object in the continuously generated depth map 330. Through this, a path curvature value indicating a flow in the direction of movement of the vehicle itself may be calculated. The calculation criterion for calculating the path curvature value for each magnitude of the continuous motion may be specified in advance.

Here, the path curvature value is a value indicating whether the amount of rotational movement of the vehicle is large or small, and the smaller the path curvature value, the faster the vehicle rotates.

In addition, in calculating the path curvature value according to the movement of the vehicle by using the depth change and the position change of each object in the depth map 330, a large amount of change due to the movement of a specific object itself included in the depth map 330. To prevent the path curvature value from being distorted and calculated, the path curvature value is calculated except for a change amount of an object in which the change amount of each object included in the depth map 330 is significantly changed by more than a predetermined reference value compared to the surrounding object. May be specified in advance.

Next, the image processor 132 calculates a curvature value of the lane detected in the wide angle image 310 and / or the narrow angle image 320. Here, the lane includes all the display information shown on the floor to induce the movement of the vehicle, for example, the lane of the road on which the vehicle travels (see Fig. 6 (a)), the parking partition line of the parking lot (Fig. 6 (b)).

As illustrated in FIG. 7A, the image processor 132 may detect lanes from the narrow angle image 320 and calculate a curvature value of the detected lanes. The curvature value of the lane extends horizontally, for example, the clothoid model by one commonly used B-spline curve, and the height of the road by two B-spline curves. It can be calculated using a 3D model, such as modeling the profile.

The clothoid model assuming a flat road may be expressed by Equation 4 below.

For reference, in Equation 4, X (L) is the position of the center point of the lane at length L, x _offset is the distance from the lane of the ego vehicle, Δψ is the yaw angle, and c ₀ is the distance of the lane. The curvature, c ₁ means the curvature change amount (closoidal parameter), where the lane width is predetermined. The B-spline-based road model for 3D lane recognition, IEEE conference on intelligent transportation system October 2010 has already been described in detail for calculating the curvature value (c ₀ ) of a recognized lane using a clothoid model. Since it is described, a description thereof will be omitted.

In addition, the image processor 132 may convert the wide-angle image 310 into a top view image and then detect lanes for the top view image and calculate curvature values of the detected lanes as illustrated in (b) of FIG. 7. have. Determination of whether a lane recognized in the top view image is a straight lane or a curved lane, and in the case of a curved lane, a method of detecting curvature through a matching process of a curve template is related to a lane and curvature based on matching a curve template using a top view image. Detection Algorithm-November, 2010 The Journal of the Institute of Electronics Engineers of Korea Vol.

As described above, the image processing unit 132 may calculate the curvature value of the lane for both the wide-angle image 310 and the narrow angle image 320, but calculates the curvature value of the lane according to a predetermined condition such as a moving speed of the vehicle. The object to be calculated may be previously designated to select one of the wide angle image 310 and the narrow angle image 320.

For example, when the vehicle is traveling at a predetermined speed or more such as driving on a highway, the curvature value of the lane is calculated for the narrow angle image 320, and when the vehicle is driving at or below a predetermined speed such as moving in a parking lot. The curvature value of the lane may be calculated in advance for the wide-angle image 310. For example, the movement speed of the vehicle may be recognized by the control unit 130 provided with relevant information from the vehicle's control system (eg, ECU), a GPS system, or the like.

As another example, when the curvature values of the lanes are calculated for both the wide-angle image 310 and the narrow-angle image 320, and the curvature values of the lanes calculated by the wide-angle image 310 are greater than or equal to a predetermined threshold value. Is determined by the curvature value of the lane calculated by the narrow angle image 320, and when the curvature value of the lane calculated by the wide-angle image 310 is less than or equal to a predetermined threshold, the curvature value of the lane calculated by the wide-angle image 310 is determined. The method of decision may be pre-specified to be applied.

Thereafter, the image processor 132 generates collision prediction information between the wide angle image 310 and the narrow angle image 320 with an object located in the vehicle traveling direction with reference to the calculated path curvature value and / or lane curvature value. Selected as the primary image to generate the collision prediction information for the object using the primary image and the secondary image.

In detail, the image processor 132 selects the base image by referring to the movement of the vehicle or the curvature of the surrounding lanes, and if the curvature is smaller than a predetermined threshold value (that is, the degree of warpage is large), the wide angle image and the narrow angle image are wide angle. An image is selected as the base image (see FIGS. 8A and 8C). In this case, the lane curvature value may be preset in comparison with the first predetermined threshold value, and the path curvature value may be compared with the second predetermined threshold value. Here, the first threshold value and the second threshold value may be designated as the same value or different values, respectively.

In this case, when the curvature is small, a sharp angle of rotation of the vehicle is required, so information around the vehicle may be more important, and thus collision prediction with an object in a driving direction, which will be described later, based on a wide-angle image photographed around the vehicle. This is because generating information is effective.

On the contrary, if the curvature is larger than the corresponding threshold (that is, the degree of warpage is small), the image processor 132 selects the narrow angle image as the basic image among the wide angle image and the narrow angle image (FIG. 8B and 8B). (d)).

This is because when the curvature is large, it is necessary to run a straight line or a smooth rotation of the vehicle. Therefore, since information of a far distance is more important than surrounding a vehicle, a driving direction to be described later based on a narrow angle image captured to a position far from the vehicle This is for generating collision prediction information with an object on the image.

However, as the information for selecting the base image, the lane curvature value may be assigned in advance to the path curvature value. This is because the vehicle generally moves along the lane marked to guide the direction of movement of the vehicle.

Accordingly, when the lane is detected using the wide-angle image 310 and / or the narrow-angle image 320 and the lane curvature value is calculated, the image processor 132 selects the base image by comparing the lane curvature value with a predetermined threshold value. However, when the vehicle is driven in an empty lot or the like where the lane is not displayed, since the lane is not recognized, the base image will be selected by comparing the path curvature value with a corresponding threshold value.

The image processor 132 divides the basic image selected from the wide angle image 310 and the narrow angle image 320 into an overlap region image and other region images, and detects an object from the overlap region image and the other region images, respectively. Generate collision prediction information for an object.

The overlap region image is an image corresponding to the overlap photographing region 260 shown in FIG. 2D, and the other region image is an image of a photographing region (ie, other regions) except for the overlap region image among the basic images. That is, the other area corresponds to, for example, a photographing area corresponding to ① and ② of FIG. 2 (d) when the base image is a wide-angle image, and corresponds to ③ of FIG. 2 (d) when the base image is a narrow angle image. Shooting area.

The image processor 132 may generate the overlap region image generated to correspond to the overlap photographing region 260 in the form of the depth map 330 as described above.

The image processor 132 attempts to detect an object in the overlap area image and calculates a spatial separation distance from the detected object. The distance from the object may be detected by changing in real time using an overlap region image generated using a wide-angle image and a narrow-angle image generated in real time.

Also, the image processor 132 may calculate the remaining time until the collision with the object by using the change in the size of the probable object and / or the change in the separation distance from the object in the overlap area.

As such, the image processor 132 generates an overlap region image by using images (that is, a wide angle image and a narrow angle image) respectively captured by heterogeneous cameras having different view angle ranges, thereby being included in the overlap region image. The amount of information can be increased, thereby improving the object detection accuracy.

Also, the image processor 132 attempts to detect an object in a region other than the overlapping region 260 in the basic image. As described above, the other region image corresponding to the other region is different depending on whether the base image is selected as the wide-angle image or the narrow-angle image.

If a collidable object is detected in the image of the other region corresponding to the other region, the remaining time until the collision with the object is determined by using the size change of the object and / or the change in the separation distance from the object as described above. Can be calculated.

The controller 134 may control the user to perform alarm processing when the collision prediction information (that is, the remaining time and / or the separation distance) generated by the image processor 132 is smaller than a predetermined third threshold value.

In this case, the alarm processing may be performed in a visual or / and audible manner, and may be processed through the output unit 140 that is, for example, a display device or a speaker device.

9 to 11 are flowcharts illustrating a collision warning method according to an embodiment of the present invention.

9 to 11, in step 910, the control unit 130 of the collision warning device 100 is provided with a wide-angle image and a narrow-angle image respectively generated by the wide-angle camera and the narrow-angle camera provided in the vehicle.

In operation 915, the control unit 130 determines whether an object is detected in the provided wide-angle image and / or the narrow-angle image. Since a method of detecting an object such as a vehicle or a traffic light in an image is obvious to those skilled in the art, a description thereof will be omitted.

If an object is detected in the wide angle image and / or the narrow angle image, the control unit 130 determines whether the object is capable of colliding with the vehicle in operation 920. For example, if the object is located in the same lane as the lane in which the vehicle is driven and the object has a change in the direction in which the size increases in the wide-angle image or the narrow-angle image generated in real time, it may be determined that collision is possible.

If it is determined that the detected object is a collidable object, in step 930, the control unit 130 generates a depth map using the wide-angle image and the narrow-angle image, and uses the generated depth map to determine a path curvature value. To calculate.

Here, the depth map is generated by using a wide angle camera and a narrow angle camera as a stereo vision and using a stereo mapping method, and has a feature of acquiring information about the depth of each object in the image.

Therefore, when the vehicle is moved in any direction, the depth value of each object included in the depth map is changed, and in which direction and in which track the vehicle is moving by using the difference in the relative change amount of each of the objects. Corresponding path curvature values can be calculated.

In operation 935, the control unit 130 determines whether a lane is detected in the wide angle image 310 and / or the narrow angle image 320. The lane may be, for example, a lane of a road on which a vehicle travels, a parking partition line of a parking lot, and the like.

 If a lane is detected, the control unit 130 calculates a curvature value of the detected lane at step 940. As described above, the curvature of the lane may be calculated using, for example, a clothoid model or a matching process of the curve template.

In operation 945, the control unit 130 determines whether the calculated lane curvature value is equal to or less than a first predetermined threshold value. Here, the lane curvature value is, for example, a value indicating the degree of curvature of the lane, and as the curvature value is smaller, the lane curvature is rapidly curved.

If the lane curvature value is less than or equal to the first threshold value, in step 950, the control unit 130 selects the wide-angle image as the base image from the wide-angle image and the narrow-angle image to be input, and overlaps the wide-angle image and the narrow-angle image on the basis of the wide-angle image. An overlap region image of the photographing region 260 is generated. Here, as described above, the overlap region image may be generated as a depth map.

The control unit 130 attempts to detect an object from the generated overlap area image, and generates collision prediction information corresponding to the object.

The collision prediction information may include, for example, the distance to the object, the remaining time until the collision with the object, and the like. That is, the control unit 130 calculates a spatial separation distance from the object detected in the overlap region image. The calculated separation distance may be used as information for calculating the remaining time required to collide with the object by using the size change amount of the object and / or the distance distance change with the object.

In addition, the control unit 130 additionally attempts to detect an object in a region other than the overlap image capturing region (see 1 and 2 of FIG. 2) in the wide-angle image which is the basic image in step 955. If a collidable object is detected in the other region image, the control unit 130 generates collision prediction information corresponding to the object as described above.

However, if it is determined in step 945 that the lane curvature value is greater than or equal to the first threshold value, in step 960, the control unit 130 selects the narrow angle image among the wide angle image and the narrow angle image input as a base image, and based on the narrow angle image. As a result, an overlap region image of the overlapping region 260 of the wide angle image and the narrow angle image is generated.

In addition, the control unit 130 attempts to detect an object in the generated overlap area image, and generates collision prediction information on the detected object.

Subsequently, in step 965, the control unit 130 further attempts to detect an object in an area other than the overlap area (see 3 in FIG. 2) in the narrow angle image, which is the basic image, and when the object is detected, the control unit 130. As described above, the collision prediction information corresponding to the object is generated.

As described above, when a lane is detected in the wide-angle image and / or the narrow-angle image, the lane curvature value is calculated as described in steps 940 to 965, and the base image and the lane curvature are compared with the lane curvature value and the first predetermined threshold value. An auxiliary image may be selected and collision prediction information about the detected object may be generated using the auxiliary image.

However, if the lane is not detected in the determination of step 935, in step 980, the control unit 130 determines whether the calculated path curvature value is equal to or less than a second predetermined threshold value.

If the path curvature value is less than or equal to the second threshold value, the control unit 130 performs the aforementioned steps 950 and 955 to generate collision prediction information.

However, if the path curvature value is greater than the second threshold value, the control unit 130 performs the above-described steps 960 and 965 to generate collision prediction information.

In step 970, the control unit 130 determines that the collision prediction information (e.g., the separation distance and / or remaining time until the collision) for the collidable object calculated in steps 950 to 965 is equal to or less than a predetermined third threshold value. Determine whether or not. Naturally, the third threshold value will be previously designated as a distance value and / or a time value.

If the collision prediction information is less than or equal to the third threshold, the control unit 130 processes the alarm to be triggered in a visual or audible manner via the output 140 to alert the driver (step 975).

Of course, the above-described collision warning method may be performed by an automated procedure in a time series order by a program embedded in or installed in the digital processing apparatus. Codes and code segments constituting the program can be easily inferred by a computer programmer in the art. The program is also stored in a computer readable media that can be read by a digital processing device, and read and executed by the digital processing device to implement the method.

Although described above with reference to embodiments of the present invention, those of ordinary skill in the art will be variously modified and modified within the scope of the present invention without departing from the spirit and scope of the invention described in the claims below It will be appreciated that it can be changed.

100: collision warning device 110: wide-angle camera unit
120: narrow angle camera unit 130: control unit
132: image processing unit 134: control unit
140: outputs 211, 213, 215, 217: wide angle camera
240: narrow angle camera 260: overlap shooting area
310: wide angle image 320: narrow angle image
330: depth map

Claims (17)

A wide-angle camera unit provided in the vehicle to generate a wide-angle image including one or more wide-angle cameras;
A narrow angle camera unit provided in the vehicle to generate a narrow angle image including one or more narrow angle cameras; And
And a control unit for selecting any one of the wide-angle image and the narrow-angle image as a base image for generating collision prediction information according to a predetermined criterion.
The control unit,
If a lane is detected in at least one of the wide-angle image and the narrow-angle image, a lane curvature value of the detected lane is calculated, and when the lane curvature value is less than or equal to a predetermined first threshold value, the wide-angle image is selected as the base image. If the lane curvature value is greater than the first threshold value, the narrow angle image is selected as the basic image.
If no lanes are detected in the wide-angle image and the narrow-angle image, when the calculated path curvature value is equal to or less than a second predetermined threshold value, the wide-angle image is selected as the base image, and the path curvature value is greater than the second threshold value. If large, select the narrow angle image as the base image,
To calculate the path curvature value, the control unit generates a depth map of an overlapping photographing area of the wide-angle camera and the narrow-angle camera using the wide-angle image and the narrow-angle image, and continuously depths that are temporally continuous. And calculating the path curvature value corresponding to the movement trajectory of the vehicle by using the relative depth change of one or more objects existing in the map.
The method of claim 1,
The control unit selects one image to calculate a lane curvature value to be compared with the first threshold value according to a predetermined condition,
The predetermined condition is one of a wide-angle image and a narrow-angle image, or one of the wide-angle image and the narrow-angle image so as to correspond to the moving speed of the vehicle, or the wide-angle image, and detected from the wide-angle image. And the narrow angle image is changed only when a curvature value of a lane is greater than or equal to a predetermined threshold value.
The method of claim 1,
The control unit,
And the path curvature value of the object included in the depth map is calculated except for an amount of change in depth of an object whose depth is changed by more than a predetermined reference value compared to a surrounding object.
delete The method of claim 1,
One or more wide angle cameras and one or more narrow angle cameras are installed such that an overlapping shooting area exists in each shooting area,
The control unit generates the collision prediction information for an object existing in the overlap shooting region and the other shooting region of the base image,
In the overlapping region, collision prediction information is generated by using an overlapping region image generated by combining an auxiliary image, which is one of a wide-angle image and a narrow-angle image, which are not selected as a base image, and the base image. Warning device.
The method of claim 5,
And the overlap area image for generating collision prediction information is generated as a depth map.
The method of claim 5,
The collision prediction information is generated to include at least one of a separation distance from the object and a remaining time until the collision with the object,
And the control unit is configured to process an alarm through an output unit when the collision prediction information is equal to or less than a third predetermined threshold value.
The method of claim 1,
And the control unit determines whether there is a collidable object in at least one of the wide-angle image and the narrow-angle image, and performs selection of the basic image only when there is a collidable object.
The method of claim 1,
And said wide angle camera is a camera for an around view monitoring system.
delete delete delete delete delete delete delete delete