KR20240079755A - The system for lane detection and the method for lane detection - Google Patents

Abstract

The purpose of the present invention is to provide a lane recognition system and method that can minimize the increase in network computation while accurately extracting only key points and utilizing them in real time.
In order to achieve the above object, the present invention is characterized by including a process of converting the possibility of a lane existing into a probability density function through kernel density estimation and extracting extreme values of the probability density function.

Description

Lane recognition system and lane recognition method {THE SYSTEM FOR LANE DETECTION AND THE METHOD FOR LANE DETECTION}

The present invention relates to a lane recognition system and a lane recognition method, and more specifically, to a lane recognition system and method using kernel density estimation that can increase accuracy without increasing the network or computational amount.

As the era of self-driving cars approaches, technology development to realize unmanned self-driving cars is actively underway.

Accurate lane recognition technology is undoubtedly an essential technology for unmanned autonomous vehicles.

In the field of autonomous driving systems, lane recognition technology is well suited to the field of artificial intelligence, and the demand for lane recognition technology in the fields of autonomous driving systems and artificial intelligence is also quite high.

Lane recognition technology is a core technology of the autonomous driving system and is used in the lane departure warning system that assists in controlling the vehicle to keep it from leaving its lane.

Traditional lane recognition techniques require complex feature extraction procedures, cannot identify various lane shapes, and are vulnerable to various environments such as weather and illumination.

To compensate for these limitations, various lane recognition algorithms based on artificial intelligence have been developed.

Artificial intelligence-based lane recognition guarantees superior performance over existing methods and has the advantage of operating even in harsh environments such as rainy weather or aging lanes, but has the disadvantage of requiring a large amount of computation.

Most artificial intelligence-based lane recognition algorithms use segmentation as their technical basis, and segmentation technology requires so many calculations that real-time is not guaranteed even with modern computers.

To compensate for this, methods for expressing suboptimal shapes through parameter prediction methods rather than segmentation have recently been studied, but in terms of accuracy, they are significantly lower than existing segmentation-based methods.

Republic of Korea Patent Publication No. 10-2384797 (2022.04.05)

The purpose of the present invention to solve the conventional problems described above is to provide a lane recognition system and method with improved performance without expanding the network so that it can be installed on a vehicle.

In order to solve the above object, the lane recognition system according to the present invention includes an input unit where an image is input; a network unit that learns images input from the input unit; a normalization unit that normalizes information learned from the network unit; an extraction unit that analyzes the information received from the normalization unit; and an output unit that outputs analysis information from the extraction unit.

In addition, in order to achieve the above object, the lane recognition method according to the present invention includes a lane recognition system, normalizing a two-dimensional probability map row by row by a normalization unit of the lane recognition system; and an extraction step of extracting key points of the probability density function from the normalization result.

Additionally, in the lane recognition method according to the present invention, the normalization further includes a segmentation network that outputs the input image as a two-dimensional probability map.

Additionally, in the lane recognition method according to the present invention, the extraction step further includes setting a probability density function through kernel density estimation.

Additionally, in the lane recognition method according to the present invention, the extraction step is characterized by extracting extreme values of the probability density function as key points.

Specific details of other embodiments are included in “Specific Details for Carrying Out the Invention” and the attached “Drawings.”

The advantages and/or features of the present invention and methods for achieving them will become clear by referring to the various embodiments described in detail below along with the accompanying drawings.

However, the present invention is not limited to the configuration of each embodiment disclosed below, but may also be implemented in various different forms. However, each embodiment disclosed in this specification ensures that the disclosure of the present invention is complete, and the present invention It is provided to fully inform those skilled in the art of the present invention, and it should be noted that the present invention is only defined by the scope of each claim.

According to the present invention, there is an effect of increasing accuracy and precision in confirming lane recognition without increasing the amount of calculation.

When using the lane recognition system and lane recognition method according to the present invention, it has the effect of being able to operate in real time.

When using the lane recognition system and lane recognition method according to the present invention, network performance is improved without increasing the amount of calculation.

When using the lane recognition system and lane recognition method according to the present invention, accuracy is improved and complexity is reduced.

1 is a conceptual schematic diagram illustrating a lane recognition system and a lane recognition method from a captured image to recognizing a lane according to the present invention.
Figure 2 is a block diagram showing the configuration of a terminal in the lane recognition system of the present invention.
Figure 3 is a flow chart showing the overall flow of the lane recognition method of the present invention.
Figure 4 is a structural diagram of the segmentation network and the first step in the lane recognition system and method according to the present invention.
Figure 5 is a schematic diagram of a two-dimensional vector probability map when performing a segmentation network according to the present invention.
Figure 6 is a schematic diagram showing the second step of the lane recognition step according to the present invention.
Figure 7 shows a first example of the results of performing the first and second steps in the lane recognition step according to the present invention.
Figure 8 shows a second example of the results of performing the first and second steps in the lane recognition step according to the present invention.
Figure 9 is a graph showing the kernel density estimation result as a probability density function after performing the first and second steps according to the present invention of Figure 8.
Figure 10 shows the result of key point acquisition through kernel density estimation in the lane recognition system and lane recognition method according to the present invention.

Before explaining the present invention in detail, the terms or words used in this specification should not be construed as unconditionally limited to their ordinary or dictionary meanings, and the inventor of the present invention should not use the terms or words in order to explain his invention in the best way. It should be noted that the concepts of various terms can be appropriately defined and used, and further, that these terms and words should be interpreted with meanings and concepts consistent with the technical idea of the present invention.

That is, the terms used in this specification are only used to describe preferred embodiments of the present invention, and are not used with the intention of specifically limiting the content of the present invention, and these terms refer to various possibilities of the present invention. It is important to note that this is a term defined with consideration in mind.

In addition, it should be noted that in this specification, singular expressions may include plural expressions, unless the context clearly indicates a different meaning, and may include singular meanings even if similarly expressed in plural. .

Throughout this specification, when a component is described as “including” another component, it does not exclude any other component, but includes any other component, unless specifically stated to the contrary. It could mean that you can do it.

Furthermore, if a component is described as being "installed within or connected to" another component, it means that this component may be installed in direct connection or contact with the other component and may be installed in contact with the other component and may be installed in contact with the other component. It may be installed at a certain distance, and in the case where it is installed at a certain distance, there may be a third component or means for fixing or connecting the component to another component. It should be noted that the description of the components or means of 3 may be omitted.

On the other hand, when a component is described as being “directly connected” or “directly connected” to another component, it should be understood that no third component or means is present.

Likewise, other expressions that describe the relationship between each component, such as "between" and "immediately between", or "neighboring" and "directly neighboring", have the same meaning. It should be interpreted as

In addition, in this specification, terms such as "one side", "other side", "one side", "the other side", "first", "second", etc., if used, refer to one component. It is used to clearly distinguish it from other components, and it should be noted that the meaning of the component is not limited by this term.

In addition, in this specification, terms related to position such as "top", "bottom", "left", "right", etc., if used, should be understood as indicating the relative position of the corresponding component in the corresponding drawing. Unless the absolute location is specified, these location-related terms should not be understood as referring to the absolute location.

In addition, in this specification, when specifying the reference numeral for each component in each drawing, the same component has the same reference number even if the component is shown in different drawings, that is, the same reference is made throughout the specification. The symbols indicate the same component.

In the drawings attached to this specification, the size, position, connection relationship, etc. of each component constituting the present invention is exaggerated, reduced, or omitted in order to convey the idea of the present invention sufficiently clearly or for convenience of explanation. It may be described, and therefore its proportions or scale may not be exact.

In addition, hereinafter, in describing the present invention, detailed descriptions of configurations that are judged to unnecessarily obscure the gist of the present invention, for example, known technologies including prior art, may be omitted.

Hereinafter, embodiments of the present invention will be described in detail with reference to the related drawings.

Figure 1 is a conceptual schematic diagram for explaining a lane recognition system and a lane recognition method from a captured image to lane recognition according to the present invention.

Referring to FIG. 1, the lane recognition system 1000 according to the present invention includes a photographing unit 100 and a terminal 200.

The photographing unit 100 acquires all information about the image or video.

The photographing unit 100 can acquire either black-and-white or color images.

In addition, the photographing unit 100 may use an infrared sensor camera, a depth camera, a light system camera, a binocular stereoscopic camera, or a surround view monitor camera mounted on a vehicle, which can estimate the distance and depth of the lane. , but is not necessarily limited to this.

The terminal 200 may include a device that transmits the image received from the photographing unit 100.

The terminal 200 is described as a miniaturized device for easy attachment and detachment from a car, but is not limited thereto.

The terminal 200 can be used by inserting it into a black box or navigation.

The photographing unit 100 and the terminal 200 communicate through wireless communication.

In addition, wireless communications include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications) in addition to Wi-Fi modules and WiBro (Wireless broadband) modules. This can be achieved by a wireless communication module that supports various wireless communication methods such as (Time Division Multiple Access), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution).

With this configuration, images captured by the photographing unit 100 can be safely received by the terminal 200.

Additionally, the lane recognition system 1000 according to an embodiment of the present invention may further include a lane detection device that derives lane detection results in addition to the above-described configuration.

Figure 2 is a block diagram showing the configuration of a terminal in the lane recognition system of the present invention.

Referring to FIG. 2, the terminal 200 of the lane recognition system 1000 according to the present invention includes an input unit 210 where an image is input, a network unit 220, a normalization unit 230, and an extraction unit 240. ) and an output unit 250 where key points are output.

Such a lane recognition system 1000 includes an input unit 210 for inputting an image, a network unit 220 for learning the image input from the input unit 210, and information learned from the network unit 220. It may include a normalization unit 230 that normalizes, an extraction unit 240 that analyzes the information received from the normalization unit 230, and an output unit 250 that outputs analysis information from the extraction unit 240. .

As described above, the configuration of the terminal 200 according to the embodiment of the present invention is such that the image input through the photographing unit 100 passes through several steps from the input unit 210 to the output unit 250. It can be refined and output as key points.

Terminal 200 of the lane recognition system 1000 according to the present invention including the above input unit 210, network unit 220, normalization unit 230, extraction unit 240, and output unit 250. ) may be implemented in the form of a hardware module or a software module, or may be implemented in a combination of a hardware module and a software module.

Here, a software module can be understood as, for example, an instruction executed by a processor that controls operations within the terminal 200, and these instructions may be mounted in a memory within the terminal 200.

In addition, it is clarified that the division of each component is merely a classification according to the main function each component is responsible for.

That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions.

This will be described in more detail later with reference to FIG. 3.

Figure 3 is a flow chart showing the overall flow of the lane recognition system and method of the present invention.

Referring to FIG. 3, the lane recognition system 1000 and lane recognition method according to the present invention may include three steps.

This lane recognition system 1000 and lane recognition method are implemented by the terminal 200 of the lane recognition system 1000, which constitutes the network unit 220, the normalization unit 230, and the extraction unit 240. .

In the first step (S100) of the lane recognition method according to the present invention, the image input to the network unit 220 is formed into a two-dimensional probability map of the same size through the segmentation network (S100).

The second step (S200) of the lane recognition method according to the present invention includes normalization (S200) in which the variance of the output of the two-dimensional probability map input to the normalization unit 230 is reduced according to the visibility of the image.

The third step (S300) of the lane recognition method according to the present invention includes an extraction step (S200) in which key points of the probability density function are extracted from the result of normalization (S200).

Each step of the lane recognition method is described by way of an example to correspond to or match the overall configuration within the terminal 200, but is not limited thereto and can be configured to correspond to the lane detection device described above.

This will be described in more detail with reference to FIGS. 4 to 7.

Figure 4 is a diagram showing the structure of a segmentation network, which is an example of a neural network according to the lane recognition system and lane recognition method according to the present invention.

Referring to FIG. 4, the lane recognition method according to the present invention is implemented by an input unit 210 and a network unit 220 that input an image from the terminal 200.

Lane recognition methods have recently been using neural networks to obtain highly accurate judgment results through image processing.

A neural network is a network structure similar to the human neural network, and is also called a deep neural network.

Neural networks may include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks, Restricted Boltzman Machines, Generative Adversarial Network (GAN), recurrent deep learning networks, etc., but are not limited to the examples above.

According to one embodiment of the present invention, the terminal 200 using an artificial intelligence (AI) system can perform a segmentation network (S100), which is one of the functions of a neural network, through the network unit 220.

Here, the network unit 220 may include a segmentation network (S100).

Segmentation network (S100) is an image detection method based on a regression deep learning model. It involves inputting an image containing an object and a background into the regression deep learning model and outputting objects separated from the image using the regression deep learning model. It may include the step of generating an image.

The regression deep learning model structure of the segmentation network (S100) can use a convolutional encoder decoder deep learning network.

The encoder structure of the segmentation network (S100) explains how RGB images are input using Resnet, but is not limited to the above example.

The segmentation network (S100) is a pixel-level regression method among regression deep learning models, and can distinguish objects and backgrounds and output a two-dimensional vector probability map of the same size.

Figure 5 is a schematic diagram of a two-dimensional vector probability map when performing a segmentation network according to the present invention.

Image information is classified from captured image information through a separate camera, and the segmentation network (S100) stores network parameter information generated by learning the classified image information using a deep learning network algorithm.

At this time, it is preferable that the classification of image information by type is provided by the user, but it may also be provided automatically through an image analysis algorithm.

As shown in FIG. 4, the segmentation network (S100) consists of an encoder and a decoder.

The encoder consists of repeated convolution, normalization, and activation layers while reducing the image size, and passes through each layer to extract features (edge, pattern, texture) representing the input image. Includes.

The structure of the encoder of the segmentation network (S100) adopts a method in which RGB images are input using Resnet, but is not limited to this.

Then, the decoder uses the features extracted through the encoder to classify the pixel (lane, road surface, sky, bush, vehicle, etc.).

Additionally, because the decoder reduces the size of the video input to the encoder, an up-sampling layer may be included to provide the same size as the original video.

The decoder can provide features with a low level of abstraction but detailed pixel change information to supplement detailed localization features when the size of the video coming from the encoder is reduced.

In other words, when extracting information from the encoder in the decoder, the decoder information requires both context information to classify pixels at a specific location and local information to reconstruct the correlation between each pixel. It is not necessarily limited.

Figure 6 is a diagram showing the second step of the lane recognition step according to the present invention.

The second step (S200) of the lane recognition method according to the present invention includes normalization of the two-dimensional probability map input to the normalization unit 230 in which the variance of the output is reduced according to the visibility of the image.

In other words, the two-dimensional probability map of the same size undergoes row-level normalization, which reduces the variance of the output, depending on the visibility of the image, so that the suboptimal areas are clearly emphasized.

The visibility of the image means setting different probability distributions depending on lanes that are not visible because they are obscured by other objects, differences in light intensity, or lanes in the form of dotted lines, but is not limited to this.

The second step (S200) described above may perform normalization by dividing the image by row, rather than normalizing the entire image.

When performing normalization on the entire image, there is a problem in that different results are performed for suboptimal visibility and environments.

However, when normalization (S200) is performed by dividing the image by row, there is high consistency with respect to suboptimal visibility and environment (light intensity, location, occlusion) within the same row in the image, thereby improving accuracy and precision. It can be raised.

This row-level normalization (S200) process can use Softmax to represent the result of the activation function for each state.

The output value of the normalization (S200) can be obtained by applying the Softmax function to obtain the probability of each state in each row.

In order to efficiently learn each state in each row, a cross-entropy measure and a back-propagation algorithm may be additionally included, and a drop-out technique may be further included and applied.

For example, the dropout technique sets random nodes in the hidden layer to a specific probability value, and is known to have a similar effect to bagging and reduce overfitting.

In the above normalization (S200) process, the softmax function is explained as an example, but it is not limited to this.

Figure 7 is a first example of the results of performing the first and second steps in the lane recognition step according to the present invention.

Figure 8 is a second example of the results of performing the first and second steps in the lane recognition step according to the present invention.

Referring to Figures 7 and 8, as a result of performing the second step in the lane recognition step according to the present invention, the result according to this embodiment of row-level normalization from observation data is explained using the softmax function as an example, You can see that the lane area is clearly emphasized.

Accordingly, the present invention has been described using examples, but these examples are illustrative and not limiting.

Anyone skilled in the art to which the present invention pertains can understand that various changes and modifications can be made without departing from the spirit of the present invention and the scope of rights set forth in the appended patent claims.

Accordingly, Figure 9 is a graph showing the results of kernel density estimation as a probability density function after performing the first and second steps of the lane recognition method according to the present invention.

Referring to FIG. 9, the row-level normalization result is input into the extraction unit 240, and an extraction step (S300) of setting a probability density function through kernel density estimation can be performed.

That is, the extraction step (S300) of the lane recognition method according to the present invention may further include the step of setting a probability density function through kernel density estimation.

Figure 10 is a schematic diagram showing the result of key point acquisition through kernel density estimation in the lane recognition system and lane recognition method according to the present invention.

Referring to FIG. 10, the actual result obtained through kernel density estimation as described above is the result of the segmentation network (S100), but the suboptimal area is clearly emphasized compared to the result of row-level normalization (S200). You can see .

Accordingly, according to the present invention, accuracy and precision are increased in confirming lane recognition without increasing the amount of calculation.

When using the lane recognition system and lane recognition method according to the present invention, it has the effect of being able to operate in real time.

When using the lane recognition system and lane recognition method according to the present invention, network performance is improved without increasing the amount of computation.

When using the lane recognition system and lane recognition method according to the present invention, accuracy is improved and complexity is reduced.

Above, various preferred embodiments of the present invention have been described by giving some examples, but the description of the various embodiments described in the "Detailed Contents for Carrying out the Invention" section is merely illustrative and the present invention Those skilled in the art will understand from the above description that the present invention can be implemented with various modifications or equivalent implementations of the present invention.

In addition, since the present invention can be implemented in various other forms, the present invention is not limited by the above description, and the above description is intended to make the disclosure of the present invention complete and is commonly used in the technical field to which the present invention pertains. It is provided only to fully inform those with knowledge of the scope of the present invention, and it should be noted that the present invention is only defined by each claim in the claims.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that it does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

1000: Lane recognition system
100: Filming Department
200: terminal
210: input unit
220: Network unit
230: Normalization unit
240: extraction unit
250: output unit

Claims (5)

An input unit where an image is input;
a network unit that learns images input from the input unit;
a normalization unit that normalizes information learned from the network unit;
an extraction unit that analyzes the information received from the normalization unit; and
Characterized in that it includes an output unit that outputs analysis information from the extraction unit,
Lane recognition system.
Including a lane recognition system according to paragraph 1,
Normalization of the two-dimensional probability map row by row by the normalization unit of the lane recognition system; and
An extraction step of extracting key points of the probability density function from the normalization result,
Lane recognition method.
According to claim 2,
The normalization is,
Characterized in that it further includes a segmentation network that outputs the input image as a two-dimensional probability map.
Lane recognition method.
According to claim 2,
The extraction step is,
Characterized in that it further comprises the step of setting a probability density function through kernel density estimation,
Lane recognition method.
According to claim 2,
The extraction step is,
Characterized by extracting extreme values of the probability density function as key points,
Lane recognition method.

Images