KR20200070100A - A method for detecting vehicle and device for executing the method - Google Patents

Abstract

The present invention relates to a method for detecting a vehicle. The method includes the steps of: receiving continuously captured front images; setting a search area of the vehicle in a target image based on a location of the vehicle or a vehicle area detected from a previous image among the front images; detecting the vehicle in the search area according to a machine learning model; and tracking the vehicle in the target image by using feature points of the vehicle extracted from the previous image according to a vehicle detection result based on the machine learning model. Since the entire image is not used as a vehicle detection area, a processing speed may be increased, and a forward vehicle tracked in an augmented reality navigation may be continuously displayed without interruption, thereby providing a stable service to a user.

Description

A method for detecting vehicle and device for executing the method

It relates to a vehicle detection method and an electronic device.

As the Internet network is opened and laws related to location data are in place, industries related to location-based services (LBS) are being activated. As a field of location-based service, a vehicle navigation-related industry has been activated to measure the current location of a device-equipped vehicle, etc., or to guide a travel route to a destination. The vehicle's black box industry for recording driving images has also been activated, and vehicles equipped with a digital video recorder or a dashboard camera for vehicles are gradually increasing.

Recently, in addition to the primary function of photographing route guidance and driving images, application technologies for assisting the driver's driving based on various image processing technologies have been developed. Specifically, images obtained through various sensors installed in a vehicle Advanced Driver Assistance System (ADAS) has been developed and applied to analyze and analyze the objects included in the image to determine driving-related information and provide it to the driver.

The advanced driver assistance system provides guidance information to the driver according to various situations. One of them is the front vehicle collision warning system (FVCWS, which recognizes the vehicle in front and provides the guidance information to maintain an appropriate safety distance from the vehicle in front). Forward Vehicle collision warning system) has been developed and applied.

The front vehicle collision warning system may, for example, have a danger of collision due to the distance from the front vehicle being within a predetermined distance while the host vehicle is running, or may collide with the front vehicle if the host vehicle maintains the current speed, or other front vehicle It is possible to identify in advance when an accident is expected as an unexpected factor caused by and provide warning information about it. In addition, technologies that provide driving guidance information to the driver on the premise of recognition of the front vehicle, such as a function of informing a departure of the front vehicle in a signal waiting state or when the vehicle is stopped, are continuously being developed in addition to collision with the front vehicle.

Therefore, in order to increase the accuracy of these technologies, it is essential to accurately detect a vehicle in front of the captured driving image.

The object of the present invention is to propose a method for performing a stable and accurate detection of a front vehicle. In addition, an object of the present invention is to propose an electronic device and a method for guiding a forward collision of the electronic device to enable safe driving through the detected front vehicle.

A vehicle detection method according to an embodiment of the present invention for solving the above technical problem includes receiving a continuously photographed front image; Setting a search area of the vehicle in a target image based on a position of the vehicle or a vehicle area detected in a previous image among the front images; Detecting the vehicle in the search area according to a machine learning model; And tracking the vehicle in the target image using feature points of the vehicle extracted from the previous image according to the vehicle detection result according to the machine learning model.

Preferably, the search area is enlarged based on the vehicle area detected in the previous image.

It is preferable that the search area is enlarged according to the size of the detected vehicle.

In the tracking, it is preferable to track the vehicle by extracting feature points of the vehicle from the vehicle area detected in the previous image.

When the vehicle detection according to the machine learning model fails or the reliability of the detected vehicle is below a reference, it is preferable to track the location of the vehicle in the target image using feature points of the extracted vehicle.

The tracking step tracks the vehicle in parallel with the vehicle detection of the detecting step, but when the vehicle detection according to the machine learning model of the detecting step is successful or the reliability of the detected vehicle is higher than the reference, tracking the vehicle It is desirable to end.

The vehicle detection method further includes displaying the detected or tracked vehicle according to a predetermined user interface.

In the displaying step, it is preferable to display a front collision-related notification according to the vehicle according to a predetermined user interface.

The detecting step further includes obtaining a motion vector of a feature point of the vehicle from a plurality of previous images and generating a modified search area based on the motion vector and the search area, and the vehicle in the modified search area And detecting according to the machine learning model.

Preferably, the motion vector is generated based on a relationship of positions where a feature point of the vehicle is expressed in each of a plurality of previous images.

The center position of the correction search area is determined based on the center position of the search area and the motion vector, and the width of the correction search area is preferably determined based on the direction or size of the motion vector.

A vehicle detection apparatus according to an embodiment of the present invention for solving the above technical problem includes an image input unit that receives a continuously photographed front image; A region setting unit configured to set a search region of the vehicle in a target image based on a location of the vehicle or a vehicle region detected from a previous image among the front images; A vehicle detection unit detecting the vehicle in the search area according to a machine learning model; And a vehicle tracking unit tracking the vehicle in the target image using feature points of the vehicle extracted from the previous image according to the vehicle detection result according to the machine learning model.

Preferably, the search area is enlarged based on the vehicle area detected in the previous image.

It is preferable that the search area is enlarged according to the size of the detected vehicle.

It is preferable that the vehicle tracking unit tracks the vehicle by extracting feature points of the vehicle from a vehicle area detected in the previous image.

It is preferable that the vehicle tracking unit tracks the location of the vehicle in the target image using feature points of the extracted vehicle when vehicle detection according to the machine learning model fails or reliability of the detected vehicle is below a reference.

The vehicle tracking unit tracks the vehicle in parallel with the vehicle detection of the vehicle detection unit, but when the vehicle detection according to the machine learning model of the vehicle detection unit is successful or the reliability of the detected vehicle is higher than a reference, the vehicle tracking is terminated. It is preferred.

The vehicle detection apparatus further includes an output unit that displays the detected or tracked vehicle according to a predetermined user interface.

It is preferable that the output unit displays a notification related to forward collision according to the vehicle according to a predetermined user interface.

In the vehicle collision warning method according to an embodiment of the present invention for solving the above technical problem, Receiving a continuously photographed front image; Setting a search area of the vehicle in a target image based on a position of the vehicle or a vehicle area detected in a previous image among the front images; Detecting the vehicle in the search area according to a machine learning model; Tracking the vehicle in the target image using feature points of the vehicle extracted from the previous image according to a vehicle detection result according to the machine learning model; And determining a possibility of collision according to the distance and relative speed from the detected or tracked vehicle.

The solution means of the subject matter of the present application is not limited to the above-described solution means, and the solution means not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings. Will be able to.

According to the present invention, since the entire image is not used as the vehicle detection area, the processing speed is increased, and the front vehicle tracked in the augmented reality navigation can be continuously displayed continuously, thereby providing a stable service to the user.

In addition, the machine learning-based vehicle detection and tracking that detects the front vehicle using the learned vehicle information by applying it to the Forward Vehicle collision warning system (FVCWS) of the Advanced Driving Assistance System. It can improve the processing speed.

1 is a block diagram illustrating an electronic device according to an embodiment of the present invention.
2 is a flowchart illustrating a vehicle detection method according to an embodiment of the present invention.
3 to 4c are flowcharts showing in more detail a vehicle detection method according to an embodiment of the present invention.
5 to 7 are exemplary views showing the setting of an adaptive search area according to an embodiment of the present invention.
8 is an exemplary diagram illustrating vehicle detection according to a machine learning model and vehicle tracking according to feature point extraction in a vehicle detection method according to an embodiment of the present invention.
9 is a block diagram illustrating a system according to an embodiment of the present invention.
10 is a diagram illustrating a system network connected to a system according to an embodiment of the present invention.
11A and 11B are views illustrating a front collision warning guide screen of a system according to an embodiment of the present invention.
12 is a view showing an implementation form when the system according to an embodiment of the present invention does not include a photographing unit.
13 is a view showing an implementation form when the system according to an embodiment of the present invention includes a photographing unit.
14 is a view showing an implementation form using a head-up display (HUD) according to an embodiment of the present invention.
15 is a block diagram showing the configuration of an autonomous vehicle according to an embodiment of the present invention.
16 is a block diagram showing a detailed configuration of a control device according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art can implement various principles included in the concept and scope of the invention and implement the principles of the invention, although not explicitly described or illustrated in the specification. In addition, all the conditional terms and examples listed in this specification are intended to be understood in principle only for the purpose of understanding the concept of the invention, and should be understood as not limited to the examples and states specifically listed in this way. .

The above-described objects, features, and advantages will become more apparent through the following detailed description in connection with the accompanying drawings, and accordingly, those skilled in the art to which the invention pertains can easily implement the technical spirit of the invention. .

In addition, in the description of the invention, when it is determined that the detailed description of the known technology related to the invention may unnecessarily obscure the subject matter of the invention, the detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The above-described objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, the present invention can be modified in various ways and can have various embodiments. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail. Throughout the specification, the same reference numbers refer to the same components in principle. In addition, when it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Hereinafter, a configuration of an electronic device and a server related to the present invention will be described in more detail with reference to the drawings. The suffixes "modules" and "parts" for components used in the following description are given or mixed only in consideration of the ease of writing the specification, and do not have meanings or roles distinguished from each other in themselves.

Electronic devices described herein may include mobile phones, smart phones, notebook computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multi-media players (PMPs), and navigation terminals. have. Hereinafter, for convenience of description, it is assumed that the electronic device is a navigation terminal.

The traffic-related video is a traffic video collected from a user device and other devices (for example, CCTV, etc.), and includes still images and video including road congestion information, road surface status information, accident information, and height information. It can be data.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

An electronic device 10 according to an embodiment of the present invention will be described with reference to FIG. 1.

1 is a block diagram of an electronic device 10 according to an embodiment of the present invention.

In this embodiment, the electronic device 10 may include an image input unit 12, a region setting unit 14, a vehicle detection unit 16, and a vehicle tracking unit 18.

The image input unit 12 receives an image captured while driving the vehicle.

The image input unit 12 may receive a front image photographed directly by a camera module included in the electronic device 10. Alternatively, it is possible to directly receive an image related to driving of the vehicle from at least one external camera (not shown). For example, when the electronic device 10 operates as a navigation device of a vehicle, the image input unit 12 may receive and input an image captured by the vehicle black box device.

Specifically, in this embodiment, the image acquired by the image input unit 12 may be a continuously photographed video. That is, it is possible to receive front images at a predetermined frame rate per second. In addition, at this time, the frame rate of receiving the front image may be changed by various conditions such as the speed of the vehicle and the weather in the surrounding environment.

The region setting unit 14 sets a search region of the vehicle in a target image based on a location of the vehicle or a vehicle region detected in a previous image among the front images.

The region setting unit 14 may set a region in which a vehicle is expected to exist in advance, instead of searching the vehicle for the entire front image input by the vehicle detection unit 16 to be described later, and detect the vehicle within the set region. To make.

Specifically, the region setting unit 14 may use the information of the vehicle detected in the previous image.

In general, when an image is input at a rate of about 30 frames per second, even if the vehicle speed is considered, the front vehicle may still be included in a predetermined area based on the location existing in the previous image, unless the movement of the front vehicle is very large.

Accordingly, the region setting unit 14 may set a candidate region to detect the vehicle in the next image based on the location of the vehicle detected in the previous image.

It is also possible to set the candidate area using the detected area of the vehicle.

Specifically, in this embodiment, the size of the front vehicle can be determined through the detected vehicle area, and the search area may be set by expanding the width and height based on the size.

The vehicle detection unit 16 detects a vehicle in a set search area according to a machine learning model.

Specifically, as a method of generating an adaptive detection area to speed up the processing of vehicle detection, the Haar algorithm is used to detect the vehicle, and after detection, the area of the detected vehicle can be expanded to be used as the detection area of the next image. have.

The Haar algorithm used as a learning method in this embodiment is a method of finding a feature by using a difference in brightness between a region and an area in an image as a theory that a specific feature has a difference in contrast, and in this embodiment, a characteristic brightness difference of a vehicle By learning and finding features through this, it is possible to detect a vehicle in front.

In this embodiment, various image processing methods other than Haar may be used as an algorithm used for learning to detect a vehicle in front, and a method of extracting a feature point of Histogram of Oriented Gradient (HOG) and Local Bit Pattern (LBP) may be used. . In addition, Adaboost and SVM (Support Vector Machine) algorithms can be applied as learning algorithms.

However, in the detection method according to the present embodiment, in order to compensate for a problem that may occur when the vehicle is tracked only by detection by machine learning, vehicle detection may be continuously performed through the feature point-based vehicle tracking when not detected. To make.

Accordingly, the vehicle tracking unit 18 extracts a feature point from the previous front image or the vehicle area of the previous front image when the vehicle detection fails within the search area set in the target image, and uses the optical flow to extract the vehicle area. Can be traced.

At this time, in the present embodiment, referring to FIG. 8, the vehicle detection unit 16 may continuously track the vehicle area of the vehicle tracking unit 18 and continue to perform machine learning-based vehicle detection.

That is, the tracking process of the vehicle tracking unit 18 stops when the vehicle detection is successful in the vehicle detection unit, sets an adaptive search region as the detected vehicle region, and continuously performs vehicle detection.

In addition, the vehicle tracking unit 18 may track the movement of the vehicle in front using the optical flow.

Hereinafter, a vehicle detection method according to an embodiment of the present invention will be described in more detail with reference to FIGS. 2 to 4.

2 is a flowchart illustrating a vehicle detection method according to the present embodiment.

Referring to FIG. 2, the image input unit receives a continuously photographed front image (S10).

The image input unit 12 receives the front image photographed through the camera module according to the determined frame rate.

For example, a front image photographed at a rate of 30 frames per second may be continuously input, and when a frame rate for receiving an image is set, a front image photographed according to the set frame rate may be input.

The region setting unit 14 sets the search region of the vehicle in the target image based on the location of the vehicle or the vehicle region detected from the previous image among the continuously input front images (S20 ).

In this embodiment, the previous image may be at least one of the input front images, and may include a plurality of images as well as a single image.

Also, the target image may be an image that is an object of extraction of an object from among the input front images.

In this embodiment, when the vehicle is detected in the previous image, the region setting unit 14 may set the search region based on the location of the vehicle or the region recognized as the vehicle, and then apply the set search region to the target image.

Specifically, referring to FIG. 3, when the front image is input, the area setting unit 14 may first check whether an adaptive search area exists (S22).

The region setting unit 14 adaptively expands the search region according to the information of the vehicle detected in the previous front image when the vehicle detection for the previous front image is successful, generates a search region, and from the next target front image You can set the vehicle as a search area.

Accordingly, it is possible to check whether an adaptive search area exists (S22), and if so, it can be set as an area for detecting a vehicle (S24).

When the search area is set, the vehicle detection unit 16 detects the vehicle according to the machine learning model in the search area (S30).

Referring to FIG. 3, the step of detecting a vehicle (S30) detects the vehicle within the search area of the front image. According to the machine learning model, image information of a vehicle in front may be learned, an object corresponding thereto may be detected in a search area, and the object may be recognized as a vehicle in front.

Further, in this embodiment, when setting a candidate area for detecting a vehicle, the candidate area can be set using the detected lane (S23). That is, in the present embodiment, when setting a candidate area for detecting a vehicle from the front image, it is possible to efficiently and quickly detect the vehicle by setting the traveling lane along the lane detected as the range of the region as the detection target region. .

Since the lane can be defined as a simpler and more uniform shape than the vehicle, it can be more easily detected in the image, and the vehicle detector 16 can also detect the vehicle using the detected lane (S30).

The vehicle detected in this embodiment may target a front vehicle, so a lane may be used to detect a vehicle in a lane in which the current vehicle is running as a front vehicle.

Generally, since a vehicle is present in the detected lane, the area setting unit 14 may determine the setting of the search area into the progress lane.

Furthermore, in this embodiment, the adaptive search area can be adaptively set in consideration of the size of the vehicle, and thus can be used to consider the ratio of the vehicle to the lane to determine the size of the vehicle.

Since the width of the lane can be determined by being standardized in advance according to the standard of the general road, it is possible to determine the type of the vehicle through the ratio of the width of the vehicle to the determined lane width and use it to set the search area.

The vehicle detection unit 16 may detect the vehicle from the front image according to the machine learning model without setting the search region when the previously generated adaptive search region does not exist (S30 ).

For example, when the user activates the detection function of the front vehicle according to the present embodiment, since the previous search area has not been generated at the initial activation time, the front vehicle can be detected using the entire front image.

Alternatively, as described above, the search area may be partially limited by using the lane information in the front image to be used for the detection of the first front vehicle.

Alternatively, it is possible to recognize a road area based on the vanishing point of the front image and to detect a vehicle ahead in the area.

In addition, when the search area does not exist and the vehicle is detected from the first front image, the background area and the road area are distinguished from the entire image, and for the road area, the driving lane in which the vehicle travels is first detected and the object located in the lane It is also possible to detect as a vehicle in front.

When the vehicle detection unit 16 detects a vehicle from the entire first front image, the region setting unit 14 is a target image to detect a vehicle using an adaptive search region generated based on the vehicle detected in the previous image You can set the search area.

Furthermore, in this embodiment, the vehicle detection unit 16 may determine whether to detect the vehicle.

That is, even if the search area is present, a vehicle may not be detected due to other optical factors, changes in the surrounding environment, or other errors.

Therefore, in the present embodiment, the vehicle detection method may enable vehicle detection to be continued through vehicle tracking when a front vehicle is not detected.

The detection of the front vehicle may be performed as an auxiliary function for safe driving, such as in a front collision situation. If the front vehicle is present, and the detection of the front vehicle fails and the user recognizes that the front vehicle does not exist, incorrect driving information is generated. Since it can be provided to the user, the vehicle detection method according to the present embodiment proposes a method in which vehicle detection can be performed when the vehicle is not detected according to the machine learning model in order to increase the stability of the function.

Referring to FIG. 4A in more detail, the step of detecting a vehicle (S30) confirms whether or not the detection of the vehicle in front is successful (S32), and if detection is successful, an adaptive search area through information of the detected vehicle Can be generated (S34).

However, when the detection of the vehicle fails due to the machine learning model, the vehicle tracking unit 18 may additionally use the vehicle feature point to continuously track the vehicle.

Therefore, in the step of tracking (S40 ), when the vehicle detection by the machine learning model fails, the vehicle tracking unit 18 may first check whether a feature point exists from an image of the previously extracted vehicle (S42 ).

In this embodiment, the feature point may be an area composed of one pixel or a plurality of pixels of an optically characteristic image. For example, an area having a brightness higher than a reference value or a feature value that can be distinguished from surrounding objects from an image of a vehicle detected until recently can be used as a feature point.

Also, the feature point may be set to a value with little change in the vehicle area of the continuously input front image.

That is, the vehicle tracking unit 18 may be set as a feature point in a region that is distinguished from other surrounding objects and has a small change in feature values in the vehicle.

When a feature point that satisfies the above conditions is preset and exists, the vehicle area may be tracked using the optical flow of the feature point (S44).

If the preset feature point does not exist, the feature point is extracted from the adaptive search area of the previous front image (S46), and the feature point extracted from the previous image is tracked according to the optical flow in the subsequently input front image. It can be recognized (S44).

The above-described detection or tracking process of the vehicle may be repeatedly performed according to the continuous input S10 of the front image.

In this embodiment, the vehicle tracking unit 18 may compare the reliability of the detected vehicle with a reference value to determine a case where detection of the vehicle has failed. That is, even if the vehicle is detected by the machine learning model, it is possible to track the vehicle using the feature point when the reliability is low compared to the image of the previous vehicle.

In this embodiment, the reliability is a reference for determining whether the vehicle tracking unit 18 performs vehicle tracking or not, and the vehicle tracking unit 18 determines whether the object detected in the image by the vehicle detection unit 16 is a vehicle. You can decide whether to perform tracking.

For example, a criterion by comparing whether an object in the image detected by the vehicle detection unit 16 is a front vehicle with a predetermined vehicle determination criterion (for example, an object's size, shape, color, or consistency with an object in the previous image). The degree of coincidence with can be determined by the reliability of the recognized object. Therefore, the vehicle tracking unit 18 may determine that the detection of the vehicle has failed when the reliability of the detected object is less than 50%, for example, and may perform tracking of the vehicle.

In addition, in this embodiment, it is also possible for the vehicle detection unit 16 to directly determine the reliability of the detected object in the image. Therefore, the vehicle detection unit 16 determines the reliability of the detected object to the vehicle, and if it is below the standard, the vehicle tracking unit 18 transfers the determination result to the vehicle tracking unit 18 to determine that the vehicle has failed. It is also possible to have it performed.

However, in the vehicle detection method according to the present embodiment, even if the operation of tracking the vehicle using the feature point fails because the detection fails in step S30 of detecting the vehicle (S40), the operation of detecting the vehicle (S30) The detection of the vehicle by the machine learning model is continuously performed.

That is, in the present embodiment, the vehicle detection method is basically set to perform vehicle detection according to the machine learning model, but additionally an optical flow to continuously recognize the movement of the front vehicle even when vehicle detection according to the machine learning model fails. Vehicle tracking according to can be performed.

Accordingly, if vehicle detection fails while continuously performing vehicle detection according to the machine learning model, vehicle tracking according to the optical flow is additionally performed, and when vehicle detection according to the machine learning model is successful again, according to the optical flow It is possible to stop vehicle tracking and perform only the vehicle detection operation according to the machine learning model.

In addition, when the step of detecting a vehicle in this embodiment (S30) is performed together with the step of tracking a vehicle (S40), the vehicle tracking result may be further used.

That is, in the step of detecting the vehicle (S30 ), when setting the adaptive search area to increase the success rate of the vehicle detection, information on the feature points determined in the tracking process of the vehicle may be used.

Specifically, in the step of detecting a vehicle (S30 ), a motion vector of a feature point of the vehicle may be obtained from a plurality of previous images.

That is, the region setting unit 14 may calculate the optical flow as a motion vector and reflect the motion vector in the adaptive search region generated in the previous image.

Therefore, in the step of setting an area (S20 ), a corrected search area may be generated based on the motion vector and the search area. Specifically, an extended value of the search area may be determined according to the size and direction of the motion vector, and a modified search area may be generated.

Thereafter, the step of detecting the vehicle (S30) may perform vehicle detection in the modified search area according to the machine learning model and further increase the detection probability of the vehicle.

In addition, when detecting a vehicle according to a machine learning model in step S30 of detecting a vehicle in the present embodiment, it is also possible to use a classifier separately learned according to a size of a vehicle or a distance characteristic from a vehicle in front.

Specifically, in order to improve vehicle detection performance according to machine learning, classifiers created after learning a database storing an image of a vehicle into a general passenger vehicle and a large vehicle may be used.

In addition, it is also possible to use a classifier that learns only the image of the tail light side, not the entire vehicle image, in order to further increase the detection of a short-distance vehicle.

Referring to FIG. 4B, in this embodiment, three classifiers can be trained.

For example, the first classifier 16a may be trained using the first vehicle image 12a.

Specifically, the first vehicle image 12a is an image of a general passenger car, and the first classifier 16a can perform machine learning using the image of a general passenger car and improve classification performance for the general passenger car.

The second classifier 16b may be trained using the second vehicle image 12b.

At this time, the second vehicle image 12b is a large vehicle unlike the first vehicle image 12a and may be a rear image of a vehicle such as a bus or a truck.

Therefore, the second classifier 16b learned from the second vehicle image 12b may have high classification performance for a large vehicle.

Additionally, the third classifier 16c may be separately learned in consideration of characteristics according to the distance from the vehicle in front of the vehicle, not the size of the vehicle.

That is, the third vehicle image 12c obtained for a vehicle located at a very short distance includes only information on a partial area, not an image of the entire vehicle, such as the first vehicle image 12a and the second vehicle image 12b. can do.

Therefore, in this embodiment, the third classifier 16c may be trained by using a separate image including only a partial region as a learning image.

Through the above process, it is possible to more accurately detect a vehicle using a classifier individually learned according to image characteristics according to a size of each vehicle or a distance from a vehicle in front.

A specific vehicle detection method will be described in more detail with reference to FIG. 4C.

When the step (S30) of detecting the above-described vehicle performs detection using the machine-trained classifier, a plurality of individually trained classifiers may be sequentially used.

Therefore, first, the first classifier-based detection is performed (S1000).

When the detection result is checked (S1010), and the first vehicle is detected by the first classifier learned according to the first vehicle image, since the detection was successful, the step of detecting the vehicle (S30) may end (S5000).

However, when the first vehicle is not detected, the second classifier-based detection is performed (S2000).

When the detection result is checked (S2010), and when the second vehicle is detected by the second classifier learned according to the second vehicle image, since the detection was successful, the step of detecting the vehicle (S30) may end (S5000). However, when the second vehicle is not detected by the second classifier, detection is performed by the third classifier (S3000).

When the vehicle is finally detected by the third classifier (S3010), since the detection is successful, the step of detecting a third vehicle located at a short distance (S3010) may be terminated (S5000).

If the vehicle is not detected even by the third classifier, the detection has failed (S4000), so the vehicle tracking process based on the feature point may be performed in step S40 of tracking the vehicle.

Further, in the above embodiments, it has been described that the detection process according to each classifier is performed sequentially, but when parallel processing is possible, it is possible to input the classifiers at the same time, and collect the classification results to generate optimal detection results.

Alternatively, in the case of a vehicle adjacent to a short distance, such as a third classifier, it is necessary to preferentially detect according to a safety requirement, so it is possible to preferentially perform detection according to the third classifier.

Hereinafter, the setting of the adaptive search area according to this embodiment will be described in more detail with reference to FIGS. 5 to 8.

5 is a diagram illustrating an example of detecting a vehicle from an input front image 1000 according to an embodiment of the present invention.

Referring to FIG. 5, the vehicle detection unit 16 may detect the vehicle 1200 from the input front image 1000. In this embodiment, the vehicle is detected within the adaptive search area, but as described above, when the search area generated as the first front image does not exist, the vehicle can be detected from the entire front image.

For example, in the case of the first time when the vehicle's detection function is activated, since the adaptive search area is not set, an object determined to be a front vehicle can be detected from the entire image.

In addition, the vehicle detection unit 16 may more easily detect a front vehicle using the lane information 1105 in the road as described above.

When a vehicle is detected in the above process, the vehicle detector 16 may calculate width (w) and height (h) information of the object 1200 determined to be a vehicle. The region setting unit 14 may set an adaptive search region using the calculated width and height.

Referring to FIG. 6, in the present embodiment, the area setting unit expands the search area according to the height and width values of the vehicle in the area 1100 in which it is determined that the vehicle 1200 is present in the previous image, so that the area setting unit 1000. To adaptively, the search area 1300 may be generated.

In this embodiment, an area extended by four directions by half w/2 of width and half h/2 of height is set as an adaptive search area.

The region setting unit 14 assumes that the vehicle will be located in the extended adaptive search region unless there is a rapid change in the vehicle speed in the continuously input image in consideration of the frame rate, and accordingly sets the adaptive search region. Can be set.

For example, if the image input from the image input unit 12 is 30 frames per second, the movement of the vehicle in front is included in the range of the previous image unless the movement of the vehicle in front is very large.

Accordingly, when a vehicle is detected from the first image in which the front vehicle first appears, an enlarged search area is set according to the detected vehicle width or height, and this area is designated as a search area for vehicle detection in the next successive image. If this process is repeatedly performed, the search area adaptively changes according to the size or position of the detected vehicle.

The adaptive search area set in this embodiment is based on the size of the vehicle, but the size can be determined using the frame rate of the input image, and additionally considers information such as the speed of the vehicle and the driving direction of the vehicle. It is also possible to do.

For example, if the speed difference from the vehicle in front is large, the adaptive search area may be further enlarged, and the search area may be enlarged even when the vehicle is driving or rotating a curve.

Furthermore, in the present embodiment, the adaptive search area may recognize a vehicle in the next lane as a front vehicle when the curve is a curve that is steeper than a predetermined curvature, so the lane detection result may be used together with the adaptive search area. It is also possible to accurately recognize a vehicle located in the current driving lane as a front vehicle.

As described above, when the adaptive search area 1300 is set according to the area 1100 where the vehicle 1200 is present, the vehicle detection unit 16 detects a vehicle within a partial area of the input front image 1000. Is done.

When comparing the search area of the front image used for the first front vehicle detection according to FIG. 5 with the front image search area according to FIG. 7, FIG. 7 sets a candidate area 1300 expected to exist in the front vehicle and is within a set range Only vehicle detection is performed according to the machine learning model, so it is possible to detect vehicles ahead and more accurately.

On the other hand, as described above, if the vehicle detection fails within the search region set in the front image, the feature points are extracted from the previous vehicle region to track the vehicle region using optical flow, and at the same time, machine learning-based vehicle detection is also performed.

When the vehicle is detected again from the front image using the machine learning model, vehicle tracking using the feature point is stopped and vehicle detection is continuously performed in the search area.

Referring to FIG. 8, a vehicle is continuously detected by setting an adaptive search area according to a machine learning model, but when a vehicle is not detected in a front image, a vehicle tracking process through feature point extraction is simultaneously performed (82). ).

The vehicle detection method according to the present embodiment has the advantage that the processing speed is increased because the entire image is not used as the vehicle detection area, and the front vehicle tracked in the augmented reality navigation can be continuously displayed without interruption, thereby providing a stable service to the user. Can provide.

Meanwhile, the electronic device 10 is implemented as a module of an Advanced Driver Assistance System (ADAS) or a system 100 for autonomous driving to perform a route guidance and a forward vehicle collision warning system (FVCWS). Can be. This will be described in more detail with reference to FIGS. 9 to 10.

9 is a block diagram illustrating a system according to an embodiment of the present invention. Referring to FIG. 9, the system 100 includes a storage unit 110, an input unit 120, an output unit 130, a curve guide unit 140, an augmented reality providing unit 160, a control unit 170, and a communication unit ( 180), the sensing unit 190, all or part of the power supply unit 195.

Here, the system 100 is a smart phone, a tablet computer, a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a smart glass, a project glass, a navigation system, which can provide driving guidance to a driver of a vehicle ( navigation), a digital image recorder for a vehicle, a digital video recorder, a car dash cam or a car video recorder, and the like, and may be implemented in various devices.

Driving guidance includes a variety of ways to assist the driver of a vehicle, such as route guidance, lane departure guidance, lane maintenance guidance, front vehicle departure guidance, traffic light change guidance, front vehicle collision warning guidance, lane change guidance, lane guidance, curve guidance, etc. It may include guidance.

Here, the route guidance is augmented reality route guidance, 2D (2-Dimensional) or 3D (3-Dimensional), which performs route guidance by combining various information such as a user's location and direction with an image photographing the front of a driving vehicle. It may include 2D (2-Dimensional) or 3D (3-Dimensional) route guidance by combining various information such as the user's location and direction with the map data of.

In addition, the route guidance may include an aerial map route guidance that combines various information such as a user's location and direction in the aerial map data to perform route guidance. Here, the route guidance may be interpreted as a concept including not only the case where the user rides on the vehicle and drives, but also the route guidance when the user walks or jumps and moves.

In addition, the lane departure guidance may be to guide whether the vehicle being driven has left the lane.

In addition, the lane maintenance guidance may be to guide the vehicle to return to the original driving lane.

In addition, the front vehicle departure guidance may be to guide the departure of a vehicle located in front of the vehicle being stopped.

In addition, the traffic light change guide may be to guide whether the signal of the traffic light located in front of the vehicle being stopped is changed. For example, when the red traffic light indicating the stop signal is turned on and the blue traffic light indicating the start signal is changed, it may be guided.

In addition, the front vehicle collision warning guide may be to guide a vehicle in front of a vehicle that is stopped or running when the distance to the vehicle is within a predetermined distance to prevent collision with the vehicle in front.

Specifically, in the present embodiment, the distance between the front vehicle and the current vehicle recognized through the machine learning model or feature point extraction may be calculated, and collision avoidance guidance may be performed accordingly.

In addition, the lane change guide may be to guide a change from the lane where the vehicle is located to another lane to guide the route to the destination. In this embodiment, it is also possible to additionally perform whether or not the vehicle in front of the lane to be changed when the lane change guide is provided, and provide the change guide accordingly.

In addition, the lane guidance may be to guide the lane in which the current vehicle is located.

In addition, the curve guide may be to guide that the road on which the vehicle will travel after a predetermined time is a curve.

Such a driving-related image, such as a front image of a vehicle that enables providing various guidance, may be captured by a camera mounted on a vehicle or a camera of a smart phone. Here, the camera may be a camera that is integrally formed with the system 100 mounted on the vehicle and photographs the front of the vehicle.

As another example, the camera may be a camera mounted on a vehicle separately from the system 100 and photographing the front of the vehicle. In this case, the camera may be a separate vehicle image capturing device mounted toward the front of the vehicle, and the system 100 may receive a captured image through wired/wireless communication with the separately mounted vehicle image capturing device, or take a vehicle image When the storage medium for storing the captured image of the device is inserted into the system 100, the system 100 may receive the captured image.

Hereinafter, the system 100 according to an embodiment of the present invention will be described in more detail based on the above-described contents.

The storage unit 110 functions to store various data and applications required for the operation of the system 100. In particular, the storage unit 110 may store data necessary for the operation of the system 100, for example, an OS, a route search application, and map data. In addition, the storage unit 110 may store data generated by the operation of the system 100, for example, searched route data, received images, and the like.

The storage unit 110 includes random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), electronically erasable and programmable ROM (EEPROM), registers, hard disk, removable disk, memory It can be implemented as a built-in storage such as a card, a Universal Subscriber Identity Module (USIM), and a removable storage such as a USB memory.

The input unit 120 functions to convert a physical input from the outside of the system 100 into a specific electrical signal. Here, the input unit 120 may include all or part of the user input unit 121 and the microphone unit 123.

The user input unit 121 may receive a user input such as a touch or push operation. Here, the user input unit 121 may be implemented using at least one of various types of buttons, a touch sensor receiving a touch input, and a proximity sensor receiving an approaching motion.

The microphone unit 123 may receive a user's voice and sound generated inside and outside the vehicle.

The output unit 130 is a device that outputs data of the system 100 to a user through video and/or audio. Here, the output unit 130 may include all or part of the display unit 131 and the audio output unit 133.

The display unit 131 is a device that outputs data that can be visually recognized to a user. The display unit 131 may be implemented as a display unit provided on the front surface of the housing of the system 100. In addition, the display unit 131 may be formed integrally with the system 100 to output visual recognition data, and may be installed separately from the system 100, such as a head up display (HUD), to output visual recognition data. have.

The audio output unit 133 is a device that outputs data that the system 100 can perceive acoustically. The audio output unit 133 may be embodied as a speaker expressing sound of data to be informed to a user of the system 100.

The curve guide unit 140 may perform the functions of the above-described curve guide device 10. Specifically, the curve guide unit 140 obtains link information corresponding to the road on which the vehicle travels, determines a position on the link of the vehicle at a future point in time, and uses the determined position and the vehicle speed at a reference point to determine After the time, it is possible to determine the degree of danger of the curve section in which the vehicle will travel.

The augmented reality providing unit 160 may provide an augmented reality view mode. Here, augmented reality means a graphic element representing a point of interest (POI), a graphic element guiding a curve, and a driver's safe driving on a screen that contains the real world that the user is actually viewing. It may be a method of visually superimposing various supplemental information to help).

The augmented reality providing unit 160 may include all or part of a calibration unit, a 3D spatial generating unit, an object generating unit, and a mapping unit.

The calibration unit may perform calibration for estimating camera parameters corresponding to the camera from the captured image taken by the camera. Here, the camera parameter is a parameter constituting a camera matrix that is information representing a relationship that a photorealistic space is formed on a photo, and may include external camera parameters and intrinsic parameters.

The 3D space generating unit may generate a virtual 3D space based on the captured image captured by the camera. Specifically, the 3D space generator may generate a virtual 3D space by applying the camera parameters estimated by the calibration unit to the 2D captured image.

The object generating unit may generate an object for guiding in augmented reality, for example, a route guiding object, a forward collision warning guiding object, a lane changing guiding object, a lane departure guiding object, a curve guiding object, and the like.

The mapping unit may map the object generated by the object generator to the virtual 3D space created by the 3D space generator. Specifically, the mapping unit may determine the location of the object generated by the object creation unit in the virtual 3D space, and perform mapping of the object to the determined location.

Meanwhile, the communication unit 180 may be provided for the system 100 to communicate with other devices. The communication unit 180 may include all or part of the location data unit 181, the wireless Internet unit 183, the broadcast transmission/reception unit 185, the mobile communication unit 186, the local area communication unit 187, and the wired communication unit 189. Can be.

The location data unit 181 is a device that acquires location data through a Global Navigation Satellite System (GNSS). GNSS refers to a navigation system capable of calculating the location of a receiving terminal using a radio wave signal received from a satellite. Specific examples of GNSS include Global Positioning System (GPS), Galileo, Global Orbiting Navigational Satellite System (GLONASS), COMPASS, Indian Regional Navigational Satellite System (IRSSS), and Quasi-Zenith Satellite System (QZSS) depending on the operating entity. Can be. The location data unit 181 of the system 100 according to an embodiment of the present invention may obtain location data by receiving a GNSS signal serving in an area where the system 100 is used. Alternatively, the location data unit 181 may acquire location data through communication with a base station or an AP (Access Point) in addition to the GNSS.

The wireless Internet unit 183 is a device that connects to the wireless Internet to acquire or transmit data. The wireless internet unit 183 is provided through various communication protocols defined to perform wireless data transmission and reception of wireless LAN (WLAN), wireless broadband (Wibro), world interoperability for microwave access (Wimax), and high speed downlink packet access (HSDPA). You can connect to the Internet network.

The broadcast transmission/reception unit 185 is a device for transmitting and receiving broadcast signals through various broadcast systems. Broadcasting systems that can be transmitted and received through the broadcast transmitting and receiving unit 185 include Digital Multimedia Broadcasting Terrestrial (DMBT), Digital Multimedia Broadcasting Satellite (DMBS), Media Forward Link Only (MediaFLO), Digital Video Broadcast Handheld (DVBH), and Integrated SDTV (ISDBT) Services Digital Broadcast Terrestrial). The broadcast signal transmitted/received through the broadcast transmitting/receiving unit 185 may include traffic data, living data, and the like.

The mobile communication unit 186 may connect to a mobile communication network according to various mobile communication standards such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE) to communicate with voice and data.

The short-range communication unit 187 is a device for short-range communication. The short-range communication unit 187, as described above, Bluetooth (Bluetooth), Radio Frequency Identification (RFID), infrared communication (IrDA, Infrared Data Association), UWB (Ultra WideBand), ZigBee, NFC (Near Field Communication), Wi It can communicate through -Fi (Wireless-Fidelity).

The wired communication unit 189 is an interface device that can connect the system 100 to other devices by wire. The wired communication unit 189 may be a USB module that can communicate through a USB port.

The communication unit 180 includes at least one of a location data unit 181, a wireless Internet unit 183, a broadcast transmission/reception unit 185, a mobile communication unit 186, a local area communication unit 187, and a wired communication unit 189. Use to communicate with other devices.

As an example, when the system 100 does not include a camera function, the short-distance communication unit 187 and the wired communication unit 189 are used to record an image captured by a vehicle video recording device such as a digital video recorder, a car dash cam, or a car video recorder. It can be received using at least one.

As another example, when communicating with a plurality of devices, one may communicate with the local area communication unit 187 and the other may communicate with the wired communication unit 119.

The sensing unit 190 is a device capable of detecting the current state of the system 100. The sensing unit 190 may include all or part of the motion sensing unit 191 and the light sensing unit 193.

The motion sensing unit 191 may detect motion in the 3D space of the system 100. The motion sensing unit 191 may include a 3-axis geomagnetic sensor and a 3-axis acceleration sensor. By combining the motion data obtained through the motion sensing unit 191 with the position data obtained through the position data unit 181, the trajectory of the vehicle to which the system 100 is attached can be more accurately calculated.

The optical sensing unit 193 is a device for measuring the ambient illumination of the system 100. By using the illuminance data acquired through the light sensing unit 193, the brightness of the display unit 131 may be changed to correspond to the ambient brightness.

The power supply unit 195 is an apparatus that supplies power required for the operation of the system 100 or other devices connected to the system 100. The power supply unit 195 may be a device that receives power from an external power source such as a battery or a vehicle built into the system 100. In addition, the power supply unit 195 may be implemented as a wired communication module 119 or a wirelessly supplied device depending on the form of receiving power.

The control unit 170 controls the overall operation of the system 100. Specifically, the control unit 170 includes a storage unit 110, an input unit 120, an output unit 130, a curve guide unit 140, an augmented reality providing unit 160, a communication unit 180, a sensing unit 190, All or part of the power supply unit 195 can be controlled.

In particular, the control unit 170 may acquire link information corresponding to a road on which the vehicle will run later. Here, the link information may be obtained from route guidance data for route guidance to a destination.

For example, when destination information is input through the input unit 120, the control unit 170 may generate route guidance data to a destination using map data previously stored in the storage unit 110. Alternatively, when destination information is input through the input unit 120, the control unit 170 may transmit a route guidance request including at least one of current location information and destination information to the server. In addition, route guidance data may be received from a server according to the route guidance request. In this case, the control unit 170 may obtain link information corresponding to a road on which the vehicle travels from the route guidance data.

In addition, when the predicted route information of the vehicle is generated based on the real-time location information of the vehicle, the controller 170 may obtain link information based on this.

Meanwhile, the control unit 170 may provide the front vehicle collision warning guidance information according to an embodiment of the present invention. That is, it is possible to detect a front vehicle from the input front image and provide guidance information according to the detected distance of the front vehicle and the speed of the current vehicle and the front vehicle. In this case, the control unit 170 may further perform the step of calculating the distance and the relative speed in the determination process of FIGS. 1 to 4 described above.

That is, it is possible to calculate the relative speed of the front vehicle in consideration of the change in the distance to the vehicle in the input front image and the frame rate, and generate front collision warning guidance information by comparing with the speed of the current vehicle.

The control unit 170 may control the output unit 130 to output the deceleration required information according to the determination result. Also, the control unit 170 may calculate acceleration information required for specific deceleration.

Required acceleration for deceleration at the current time t in the present embodiment (A _req (t)) is the one person between the preceding vehicle acceleration in the (V _TV) (A _TV), and a preceding vehicle (V _TV) and the source vehicle (V _SV) It can be calculated using the speed (V _r (t), V _r (t) = V _TV (t)-V _SV (t)) and the distance between the vehicle in front and the driving vehicle (X _c (t)) (math Expression).

[Mathematics]

Furthermore, the distance from the vehicle in front may be calculated by further taking into consideration the driving distance X _r (t) during a time when the driver reacts to controlling the brake for deceleration.

In addition, in the present embodiment, the control unit warns of a front collision in consideration of the side distance between the driving vehicle and the front vehicle in the case of a cut-in vehicle that changes lanes to the driving lane other than the front vehicle in the traveling lane of the traveling vehicle. You can decide whether to calculate the required acceleration for

Specifically, when a front vehicle changing a lane is detected, it is possible to determine whether to calculate the required acceleration by comparing the vehicle width of the driving vehicle and the lateral distance between the center lines of both vehicles.

The control unit 170 may control the output unit 130 stepwise according to the required acceleration calculated through the above process.

If the required acceleration is the first level, the control unit 170 may control the output unit 130 to output the first deceleration guide. Here, the first level may be a value indicating that the user needs to decelerate preliminarily.

At this time, a numerical value indicating the necessity of deceleration may be calculated in consideration of the distance between the driving vehicle and the front vehicle, the speed of the vehicle, and the number of lanes.

If the required acceleration is the second level, the control unit 170 may control the output unit 130 to output the second deceleration guide. Here, the second level may be a value indicating that a higher degree of deceleration of the user is required.

If the required acceleration is lower than the first level or the speed of the driving vehicle is lower than the reference speed, the controller 170 may control the output unit 130 to not output the deceleration guidance.

In addition, the controller 170 may divide the required acceleration into three or more stages, and provide the user with a deceleration guide suitable for the situation in each stage.

In addition, the control unit 170 may control the output unit 130 so as not to output the deceleration guidance even if it exceeds the determined level by determining a condition that precedes the collision warning of the vehicle in front.

Specifically, the preferential condition may be considered as a priority when the vehicle in front changes the lane, or when the re-navigation of the route occurs, or when an event occurs in the driving lane after branching.

Meanwhile, such deceleration guidance may be performed within the augmented reality screen. Specifically, the augmented reality providing unit 160 may generate a front vehicle collision warning guide object and map it to a virtual 3D space to generate an augmented reality screen, and the controller 170 displays the generated augmented reality screen The display unit 131 can be controlled to do so.

10 is a view for explaining a system network connected to a system according to an embodiment of the present invention. Referring to FIG. 10, the system 100 according to an embodiment of the present invention may be implemented as various devices provided in a vehicle, such as a navigation device, an image capturing device for a vehicle, a smart phone, or other augmented reality interface providing device for a vehicle, It can be connected to various communication networks and other electronic devices 61 to 64.

In addition, the system 100 may calculate the current location and the current time zone by interlocking the GPS module according to the radio wave signal received from the satellite 20.

Each satellite 20 may transmit L band frequencies with different frequency bands. The system 100 may calculate the current position based on the time taken for the L-band frequency transmitted from each satellite 20 to reach the system 100.

Meanwhile, the system 100 may wirelessly access the network 30 through the control unit 180 through the control station 40, ACR, base station 50, RAS, and access point (AP). When the system 100 connects to the network 30, data can be exchanged by indirectly connecting to other electronic devices 61 and 62 connected to the network 30.

Meanwhile, the system 100 may indirectly connect to the network 30 through another device 63 having a communication function. For example, when a module that can connect to the network 30 is not provided in the system 100, it may communicate with another device 63 having a communication function through a short-range communication module.

11A and 11B are views illustrating a front vehicle collision warning screen of a system according to an embodiment of the present invention. Referring to FIGS. 11A and 11B, the system 100 may generate a guide object indicating a danger level of a vehicle collision in front and output the generated guide objects 1001 and 1003 through augmented reality.

Here, the guide objects 1001 and 1003 may be objects that guide a user in need of attention. That is, the front vehicle collision warning guide may be an attention guide for informing the vehicle that a front collision situation may occur. Also, referring to FIG. 11B, in this embodiment, the path guide object 1002 may be implemented as a texture image and expressed through augmented reality. Accordingly, the driver can easily recognize the road on which the host vehicle is driving.

In addition, the system 100 may output guide objects 1001 and 1003 through voice. Alternatively, it is possible to output through a haptic element.

12 is a view showing an implementation form when the system according to an embodiment of the present invention does not include a photographing unit. Referring to FIG. 12, a vehicle image photographing apparatus 200 provided separately from the vehicle system 100 may configure a system according to an embodiment of the present invention using a wired/wireless communication method.

The vehicle system 100 may include a display unit 131 provided on the front surface of the housing 191, a user input unit 121, and a microphone 123.

The vehicle image capturing apparatus 200 may include a camera 222, a microphone 224, and an attachment portion 281.

13 is a view showing an implementation form when the system according to an embodiment of the present invention includes a photographing unit. Referring to FIG. 13, when the system 100 includes the photographing unit 150, the user photographs the front of the vehicle by the photographing unit 150 of the system 100 and the user displays the display portion of the system 100. It may be a device that can be recognized. Accordingly, a system according to an embodiment of the present invention can be implemented.

14 is a view showing an implementation form using a head-up display (HUD) according to an embodiment of the present invention. Referring to FIG. 14, the HUD may display the augmented reality guide screen on the HUD through wired/wireless communication with other devices.

For example, augmented reality may be provided through a HUD using a windshield of a vehicle or an image overlay using a separate image output device, and the augmented reality providing unit 160 may interface with a real image or an overlay on the glass. Can generate Through this, an augmented reality navigation or a vehicle infotainment system may be implemented.

Meanwhile, the method for guiding a front vehicle collision warning according to various embodiments of the present invention may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing a server or device in which the program is stored.

On the other hand, in another embodiment, the front vehicle detection method or the front collision warning guidance method according to the present invention may be configured as a module in the control device 2100 of the autonomous vehicle 2000. That is, the memory 2122 and the processor 2124 of the control device 2100 can implement a software for implementing a vehicle detection method or a vehicle collision warning guidance method according to the present invention.

Hereinafter, it will be described in more detail with reference to FIG. 15.

15 is a block diagram showing the configuration of an autonomous vehicle 2000 according to an embodiment of the present invention.

15, the autonomous vehicle 2000 according to the present embodiment may include a control device 2100, a sensing module 2004a, 2004b, 2004c, 2004d, an engine 2006, and a user interface 2008. Can be.

In this embodiment, the control device 2100 includes a controller 2120, a sensor 2110, a wireless communication device 2130, a LIDAR 2140, and a camera module 2150 including a memory 2122 and a processor 2124. It can contain.

In this embodiment, the controller 2120 may be configured at the time of manufacture by the manufacturer of the vehicle, or additionally configured to perform the function of autonomous driving after manufacturing. Alternatively, a configuration for continuously performing additional functions may be included through an upgrade of the controller 2120 configured at the time of manufacture.

The controller 2120 includes a control signal 2110, an engine 2006, a user interface 2008, a wireless communication device 2130, a LIDAR 2140, and a camera module 2150, which include control signals in other components in the vehicle. Can be delivered to. In addition, although not shown, a control signal may be transmitted to an acceleration device, a braking system, a steering device, or a navigation device related to driving of a vehicle.

In this embodiment, the controller 2120 may control the engine 2006 and, for example, the engine 2006 such that the autonomous vehicle 2000 detects the speed limit of the road on which it is driving and does not exceed the speed limit. ), or may control the engine 2006 to accelerate the driving speed of the autonomous vehicle 2000 within a range not exceeding the speed limit. In addition, when the sensing module (2004a, 2004b, 2004c, 2004d) detects the environment outside the vehicle and transmits it to the sensor 2110, the controller 2120 receives it and receives an engine 2006 or a steering device (not shown). The driving of the vehicle can be controlled by generating a control signal.

The controller 2120 may control the engine 2006 or the braking system to decelerate the driving vehicle when another vehicle or an obstacle is in front of the vehicle, and may control trajectory, driving path, and steering angle in addition to the speed. . Alternatively, the controller 2120 may control driving of the vehicle by generating a necessary control signal according to recognition information of other external environments such as the driving lane of the vehicle, a driving signal, and the like.

In addition to the generation of its own control signal, the controller 2120 can also control the driving of the vehicle by performing communication with the surrounding vehicle or the central server and transmitting a command for controlling the peripheral devices through the received information.

In addition, since the controller 2120 may be difficult to recognize an accurate vehicle or lane according to the present embodiment when the position of the camera module 2150 is changed or the angle of view is changed, calibration of the camera module 2150 to prevent this ( It is also possible to generate control signals that control the calibration. Therefore, in this embodiment, the controller 2120 generates a calibration control signal to the camera module 2150, so that the mounting position of the camera module 2150 is caused by vibration or shock generated according to the movement of the autonomous vehicle 2000. Even if changed, it is possible to continuously maintain the normal mounting position, direction, angle of view, etc. of the camera module 2150. In the controller 2120, the initial mounting position, direction, and angle of view information of the pre-stored camera module 2120 and the initial mounting position, direction, and angle of view information of the camera module 2120 measured during driving of the autonomous vehicle 2000 are critical. If it is more than the value, a control signal may be generated to perform calibration of the camera module 2120.

In this embodiment, the controller 2120 may include a memory 2122 and a processor 2124. The processor 2124 may execute software stored in the memory 2122 according to a control signal of the controller 2120. Specifically, the controller 2120 stores data and instructions for performing the lane marking method or the lane departure guidance method according to the present invention in the memory 2122, and the instructions 2124 to implement one or more methods disclosed herein. ).

At this time, the memory 2122 may be stored in a recording medium executable on the nonvolatile processor 2124. The memory 2122 can store software and data through appropriate internal and external devices. The memory 2122 may include a random access memory (RAM), a read only memory (ROM), a hard disk, and a memory 2122 device connected to a dongle.

The memory 2122 may store at least operating system (OS), user application, and executable commands. The memory 2122 may also store application data and array data structures.

The processor 2124 may be a microprocessor or a suitable electronic processor, which may be a controller, microcontroller, or state machine.

The processor 2124 may be implemented as a combination of computing devices, and the computing device may be a digital signal processor, a microprocessor, or an appropriate combination thereof.

In addition, in the present embodiment, the control device 2100 may monitor characteristics of the inside and outside of the autonomous vehicle 2000 and sense a state using at least one sensor 2110.

The sensor 2110 may be composed of at least one sensing module 2004, and the sensing module 2004 may be implemented at a specific location of the autonomous vehicle 2000 according to the detection purpose. The autonomous vehicle 2000 may be located at a lower portion, a rear end, a front end, an upper end, or a side end of the autonomous vehicle, and may also be located in an internal part of a vehicle or a tire.

Through this, the sensing module 2004 can detect information related to driving, such as engine 2006, tires, steering angle, speed, and weight of the vehicle, as internal information of the vehicle. In addition, the at least one sensing module 2004 may be configured as an acceleration sensor 2110, a gyroscope, an image sensor 2110, a RADAR, an ultrasonic sensor, a LiDAR sensor, and the like, and display motion information of the autonomous vehicle 2000. Can be detected.

The sensing module 2004 may receive specific data about the external environment state, such as the state information of the road on which the autonomous vehicle 2000 is located, the surrounding vehicle information, the weather as external information, and detect the vehicle parameters accordingly. Do. The sensed information may be temporarily or long-term stored in the memory 2122 according to the purpose.

In this embodiment, the sensor 2110 may collect and collect information of sensing modules 2004 for collecting information generated from inside and outside of the autonomous vehicle 2000.

The control device 2100 may further include a wireless communication device 2130.

The wireless communication device 2130 is configured to implement wireless communication between the autonomous vehicles 2000. For example, the user's mobile phone or other wireless communication device 2130, another vehicle, a central device (traffic control device), a server, and the like, enables the autonomous vehicle 2000 to communicate. The wireless communication device 2130 may transmit and receive a wireless signal according to an access wireless protocol. The wireless communication protocol may be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM). It is not limited to this.

Also, in the present embodiment, the autonomous vehicle 2000 may implement communication between vehicles through the wireless communication device 2130. That is, the wireless communication device 2130 may perform communication with other vehicles and other vehicles on the road through vehicle-to-vehicle communication (V2V) communication. The autonomous vehicle 2000 may transmit and receive information such as driving warnings and traffic information through communication between vehicles, and may request or receive information from other vehicles. For example, the wireless communication device 2130 may perform V2V communication as a designated short-range communication (DSRC) device or a C-V2V (Celluar-V2V) device. In addition, communication between the vehicle and other objects (for example, electronic devices carried by pedestrians, etc.) in addition to communication between vehicles (V2X, Vehicle to Everything communication) may also be implemented through the wireless communication device 2130.

Also, the control device 2100 may include a LIDAR device 2140. The LIDAR device 2140 may detect an object around the autonomous vehicle 2000 during operation using data sensed through the LIDAR sensor. The LIDAR device 2140 may transmit the detected information to the controller 2120, and the controller 2120 may operate the autonomous vehicle 2000 according to the detection information. For example, the controller 2120 may command the vehicle to reduce the speed through the engine 2006 when there is a forward vehicle traveling at a low speed in the detection information. Alternatively, the vehicle may be commanded to reduce the entry speed according to the curvature of the entering curve.

The control device 2100 may further include a camera module 2150. The controller 2120 may extract object information from an external image photographed by the camera module 2150 and allow the controller 2120 to process the information.

In addition, the control device 2100 may further include imaging devices for recognizing an external environment. In addition to the LIDAR 2140, a RADAR, GPS device, odometer, and other computer vision devices can be used, and these devices can be selected or operated as needed to enable more precise detection.

The autonomous vehicle 2000 may further include a user interface 2008 for a user input to the above-described control device 2100. The user interface 2008 may allow the user to enter information through appropriate interaction. For example, it may be implemented with a touch screen, a keypad, and operation buttons. The user interface 2008 may transmit an input or command to the controller 2120, and the controller 2120 may perform a control operation of the vehicle in response to the input or command.

In addition, the user interface 2008 may communicate with the autonomous vehicle 2000 through a wireless communication device 2130 as a device external to the autonomous vehicle 2000. For example, the user interface 2008 may enable interworking with a mobile phone, tablet, or other computer device.

Furthermore, in this embodiment, the autonomous vehicle 2000 has been described as including the engine 2006, but it is also possible to include other types of propulsion systems. For example, the vehicle may be driven by electric energy, and may be driven by hydrogen energy or a hybrid system combining them. Accordingly, the controller 2120 may include a propulsion mechanism according to the propulsion system of the autonomous vehicle 2000, and may provide control signals to the components of each propulsion mechanism.

Hereinafter, with reference to FIG. 16, a detailed configuration of the control apparatus 2100 performing the front vehicle detection method or the front collision warning guidance method according to the present invention according to this embodiment will be described in more detail.

The control device 2100 includes a processor 2124. The processor 2124 may be a general purpose single or multi-chip microprocessor, dedicated microprocessor, microcontroller, programmable gate array, or the like. The processor may also be referred to as a central processing unit (CPU). Also, in the present embodiment, the processor 2124 may be used in combination of a plurality of processors.

The control device 2100 also includes a memory 2122. The memory 2122 may be any electronic component capable of storing electronic information. Memory 2122 may also include a combination of memories 2122 in addition to a single memory.

Data and instructions 2122a for performing the front vehicle detection method or the front collision warning guidance method according to the present invention may be stored in the memory 2122. When the processor 2124 executes the instructions 2122a, the instructions 2122a and all or part of the data 2122b necessary for the execution of the instructions may be loaded 2124a, 2124b onto the processor 2124.

The control device 2100 may include a transmitter 2130a, a receiver 2130b, or a transceiver 2130c to allow transmission and reception of signals. The one or more antennas 2132a and 2132b may be electrically connected to the transmitter 2130a, the receiver 2130b, or each transceiver 2130c, and may additionally include antennas.

The control device 2100 may include a digital signal processor (DSP) 2170. Through the DSP 2170, a digital signal can be quickly processed by the vehicle.

The control device 2100 may also include a communication interface 2180. The communication interface 2180 may include one or more ports and/or communication modules for connecting other devices to the control device 2100. The communication interface 2180 may enable the user and the control device 2100 to interact.

Various configurations of the control device 2100 may be connected together by one or more buses 2190, and the buses 2190 may include a power bus, a control signal bus, a status signal bus, a data bus, and the like. Under the control of the processor 2124, the components may transfer mutual information through the bus 2190 and perform a desired function.

The device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers.

The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The above description is merely illustrative of the technical spirit of the present invention, and those skilled in the art to which the present invention pertains may make various modifications, changes, and substitutions without departing from the essential characteristics of the present invention. It will be possible.

Therefore, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention, but to explain the scope of the technical spirit of the present invention. . The scope of protection of the present invention should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims (24)

In the vehicle detection method,
Receiving the continuously photographed front images;
Setting a search area of the vehicle in a target image based on a position of the vehicle or a vehicle area detected in a previous image among the front images;
Detecting the vehicle in the search area according to a machine learning model; And
And tracking the vehicle in the target image using feature points of a vehicle extracted from the previous image according to a vehicle detection result according to the machine learning model. According to claim 1,
The search area is a vehicle detection method characterized in that the enlargement is set based on the vehicle area detected in the previous image. According to claim 2,
The search area is enlarged according to the size of the detected vehicle Vehicle detection method. According to claim 1,
The tracking includes extracting a feature point of the vehicle from a vehicle area detected in the previous image, and tracking the vehicle. According to claim 1,
A vehicle detection method comprising tracking the position of the vehicle in the target image using feature points of the extracted vehicle when vehicle detection according to the machine learning model fails or reliability of the detected vehicle is below a reference. According to claim 1,
The tracking step tracks the vehicle in parallel with the detection of the vehicle in the detecting step, but when the vehicle detection according to the machine learning model of the detecting step is successful or the reliability of the detected vehicle is higher than the reference, the tracking of the vehicle Vehicle detection method characterized in that the end. According to claim 1,
The vehicle detection method,
And displaying the detected or tracked vehicle according to a predetermined user interface. The method of claim 7,
The displaying step is a vehicle detection method characterized in that the front collision related notification according to the vehicle is displayed according to a predetermined user interface. According to claim 1,
The detecting step further includes obtaining a motion vector of a feature point of the vehicle from a plurality of previous images and generating a modified search area based on the motion vector and the search area,
And detecting the vehicle in the modified search area according to a machine learning model. The method of claim 9,
The motion vector is a vehicle detection method characterized in that the feature point of the vehicle in each of the plurality of previous images is generated based on the relationship of the expressed positions. The method of claim 9,
The center position of the modified search area is determined based on the center position of the search area and the motion vector,
The width of the modified search area is determined based on the direction or size of the motion vector. An image input unit that receives continuously photographed front images;
A region setting unit configured to set a search region of the vehicle in a target image based on a location of the vehicle or a vehicle region detected from a previous image among the front images;
A vehicle detection unit detecting the vehicle in the search area according to a machine learning model; And
And a vehicle tracking unit tracking the vehicle in the target image using feature points of a vehicle extracted from the previous image according to a vehicle detection result according to the machine learning model. The method of claim 12,
The search area is a vehicle detection device characterized in that the enlargement is set based on the vehicle area detected in the previous image. The method of claim 13,
The search area is a vehicle detection device characterized in that the magnification is set according to the size of the detected vehicle. The method of claim 12,
The vehicle tracking unit tracks the vehicle by extracting feature points of the vehicle from the vehicle area detected in the previous image. The method of claim 12,
The vehicle tracking unit tracks the position of the vehicle in the target image using feature points of the extracted vehicle when vehicle detection according to the machine learning model fails or reliability of the detected vehicle is below a reference. Device. The method of claim 12,
The vehicle tracking unit tracks the vehicle in parallel with the vehicle detection of the vehicle detection unit, and when the vehicle detection according to the machine learning model of the vehicle detection unit is successful or the reliability of the detected vehicle is higher than a reference, the vehicle tracking is terminated. Vehicle detection device characterized in that. The method of claim 12,
The vehicle detection device,
And an output unit configured to display the detected or tracked vehicle according to a predetermined user interface. The method of claim 18,
The output unit is a vehicle detection device, characterized in that for displaying a front collision related notification according to the vehicle according to a predetermined user interface. In the vehicle collision warning method,
Receiving the continuously photographed front images;
Setting a search area of the vehicle in a target image based on a position of the vehicle or a vehicle area detected in a previous image among the front images;
Detecting the vehicle in the search area according to a machine learning model;
Tracking the vehicle in the target image using feature points of the vehicle extracted from the previous image according to a vehicle detection result according to the machine learning model; And
And determining a possibility of collision according to a distance and a relative speed with the detected or tracked vehicle. A computer-readable recording medium storing a program for performing the vehicle detection method according to any one of claims 1 to 11. A program stored in a computer-readable recording medium for performing the vehicle detection method according to any one of claims 1 to 11. A computer-readable recording medium storing a program for performing the vehicle collision warning method according to claim 20. A program stored in a computer-readable recording medium for performing the vehicle collision warning method according to claim 20.

Images