KR20190049147A - Lane departure detecting device and controlling method thereof - Google Patents

Abstract

A lane departure detecting device and a control method thereof are disclosed. The control method of the lane departure detecting device comprises the following steps of: receiving an image including a lane; applying a random deformation value to an input image to generate a composite image; and labeling the generated composite image to a departure image when an absolute value of the random deformation value is larger than a predetermined threshold value, and labeling the generated composite image to a normal image when the absolute value of the random deformation value is smaller than the predetermined threshold value.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a lane departure detection device,

The present disclosure relates to a lane departure detection apparatus and a control method, and more particularly, to a lane departure detection apparatus and a control method for automatically labeling learning data.

Research on techniques for recognizing objects from photographed images is underway. Object recognition techniques are widely used in security devices, automobile safety devices, and the like. In the past, technology was widely used by developers to set criteria for recognizing objects, to recognize objects based on established criteria, and to judge the situation. However, there is a problem that a method of setting a standard by a developer causes an error of object recognition error or a situation judgment when an unexpected event occurs.

However, as researches on artificial intelligence have been actively developed recently, artificial intelligence is learned by using a large amount of data in advance, and artificial intelligence is being developed for recognizing objects and determining the situation. The learned artificial intelligence can appropriately cope with the occurrence of an unexpected event by appropriately recognizing an object or judging a situation using existing learning data. A lot of data is needed to learn AI to cope properly with unexpected events. Traditionally, developers have learned artificial intelligence by labeling their own learning data. However, since the developer has to directly label tens of thousands of pieces of learning data, there has been a problem that a lot of time and cost are incurred. Thus, there is a need for an efficient technique for learning artificial intelligence.

DISCLOSURE OF THE INVENTION The present disclosure is intended to solve the above-mentioned problems, and an object of the present disclosure is to provide a lane departure detection apparatus and a control method which automatically label image data.

According to an aspect of the present invention, there is provided a control method for a lane departure detection apparatus, comprising: receiving an image including a lane; generating a composite image by applying a random transformation value to the input image; Labeling the generated composite image as an outgoing image if the absolute value of the random deformation value is greater than a preset threshold value and labeling the generated composite image as a normal image if the absolute value of the random deformation value is smaller than the predetermined threshold value .

The random deformation value may be at least one of a rotational deformation value and a shift deformation value.

The control method of the lane departure detection apparatus may further include learning the neural network using the labeled composite image.

The learning of the neural network may include generating a first matrix map by convoluting the labeled composite image, comparing the generated first matrix map with 0 to generate a second matrix map in which a large value is returned, The generated second matrix map may be resized to generate a property map, the generated property map may be concatenated, and the class of the labeled synthetic image may be classified based on the connected property map.

Meanwhile, the class may be at least one of normal driving, left lane departure and right lane departure.

The control method of the lane departure detection apparatus may include receiving a determination target image including a lane, sensing a lane from the determination target image, and determining a normal operation, a lane departure, and a lane departure based on the learned neural network and the detected lane, A left lane departure or a right lane departure.

According to an embodiment of the present disclosure for achieving the above object, a lane departure detection apparatus includes an input unit for receiving an image including a lane, and an automatic annotation unit for deforming and labeling the input image, Wherein the generating unit generates a composite image by applying a random deformation value to the input image, and labels the generated composite image as an output image if the absolute value of the random deformation value is greater than a preset threshold value, The generated composite image is labeled as a normal image.

As described above, according to various embodiments of the present disclosure, the lane departure detection apparatus and the control method can automatically label the image data.

The lane departure detection apparatus and the control method can label tens of thousands of image data within a few minutes.

In addition, the lane departure detection apparatus and control method can learn more accurate lane departure by learning artificial intelligence with more image data in a short time.

1 is a block diagram of a lane departure detection apparatus according to an embodiment of the present disclosure;
2 is a block diagram of a lane departure detection apparatus according to another embodiment of the present disclosure;
3 is a view for explaining a process of labeling image data according to an embodiment of the present disclosure.
4 is a diagram for explaining a process of learning lane departure according to an embodiment of the present disclosure.
5 is a diagram illustrating the output of a lane departure network in accordance with one embodiment of the present disclosure;
6 is a flowchart of a method of controlling a lane departure detection apparatus according to an embodiment of the present disclosure.

Various embodiments will now be described in detail with reference to the accompanying drawings. The embodiments described herein can be variously modified. Specific embodiments are described in the drawings and may be described in detail in the detailed description. It should be understood, however, that the specific embodiments disclosed in the accompanying drawings are intended only to facilitate understanding of various embodiments. Accordingly, it is to be understood that the technical idea is not limited by the specific embodiments disclosed in the accompanying drawings, but includes all equivalents or alternatives falling within the spirit and scope of the invention.

 Terms including ordinals, such as first, second, etc., may be used to describe various elements, but such elements are not limited to the above terms. The above terms are used only for the purpose of distinguishing one component from another.

In this specification, the terms " comprises " or " having ", and the like, are intended to specify the presence of stated features, integers, steps, operations, elements, parts, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof. It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

In the meantime, " module " or " part " for components used in the present specification performs at least one function or operation. Also, " module " or " part " may perform functions or operations by hardware, software, or a combination of hardware and software. Also, a plurality of " modules " or a plurality of " parts ", other than a " module " or " part ", to be performed in a specific hardware or performed in at least one processor may be integrated into at least one module. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In addition, in the description of the present invention, when it is judged that the detailed description of known functions or constructions related thereto may unnecessarily obscure the gist of the present invention, the detailed description thereof will be abbreviated or omitted.

1 is a block diagram of a lane departure detection apparatus according to an embodiment of the present disclosure;

Referring to FIG. 1, the lane departure detection apparatus 100 includes a first input unit 110 and an auto annotator 120. The first input unit 110 receives an image including a lane. An image containing a lane may be a normal image without lane departure. For example, the first input unit 110 may include a communication module connected to the communication interface, an input terminal connected to the input interface, and the like. That is, when the first input unit 110 is implemented as a communication module, the lane departure detection apparatus 100 can receive an image including a lane using a wire / wireless communication method. Alternatively, when the first input unit 110 is implemented as an input terminal, the lane departure detection apparatus 100 may receive an image including the lane from the internal and external storage devices.

The automatic annotation unit 120 receives the image including the lane inputted through the first input unit 110. [ The automatic annotation unit 120 generates a random deformation value and then applies an arbitrary deformation value to the input image to generate a composite image. For example, an arbitrary deformation value may include at least one of a rotational deformation value and a shift deformation value. In one embodiment, the automatic annotation unit 120 may apply an arbitrary rotational deformation value to the input image. When the automatic annotation unit 120 applies an arbitrary rotation deformation value to the input image, the input image is transformed into an image rotated by the rotational deformation value to the left or right. That is, the automatic annotation unit 120 can generate a composite image to which an arbitrary rotational deformation value is applied from the input image. Alternatively, the automatic annotation unit 120 may apply an arbitrary shift deformation value to the input image. When the automatic annotation section 120 applies an arbitrary shift deformation value to the input image, the input image is transformed to the image shifted by the shift deformation value to the left or right. That is, the automatic annotation unit 120 can generate a composite image to which an arbitrary shift deformation value is applied from the input image. The automatic annotation unit 120 may generate a composite image in which a certain rotational deformation value and an arbitrary shift deformation value are applied together.

The automatic annotation unit 120 can generate a composite image by applying a different arbitrary deformation value to the input image. Accordingly, the automatic annotation unit 120 may generate a plurality of composite images having different arbitrary deformation values applied to one input image.

The automatic annotation unit 120 compares the absolute value of an arbitrary deformation value with a predetermined threshold value. For example, the predetermined threshold may be set to an absolute value or a relative value according to the input image. In one embodiment, when the scale or the lane width of the input image is constant, the predetermined threshold may be set to an absolute value such as 20 degrees or 2 cm. If the scale or lane width of the input image is varied, the predetermined threshold value may be set to a relative value such as 20% of the lane width of the input image. When the predetermined threshold value is set to a relative value according to the input image, the automatic annotation unit 120 can label the image including various scales or various lane widths with a normal image or a lane-free image .

The automatic annotation unit 120 compares the absolute value of an arbitrary deformation value with a preset threshold value, and labels the generated composite image as a lane departure image when any deformation value is larger than a preset threshold value. Then, the automatic annotation unit 120 compares the absolute value of an arbitrary deformation value with a preset threshold value, and labels the generated composite image as a normal image when any deformation value is smaller than a predetermined threshold value. That is, the automatic annotation unit 120 may generate an image in which the lane is virtually deviated and label it as a lane departure image.

As described above, the automatic annotation unit 120 can generate a plurality of composite images by applying different arbitrary deformation values to one input image. Then, the automatic annotation unit 120 can generate a plurality of labeling images for a plurality of composite images. Accordingly, although the existing technique can generate only one labeling image for one input image, the lane departure detection apparatus 100 according to the present disclosure may generate a plurality of labeling images for one input image.

Also, the automatic annotation unit 120 may receive a plurality of images and generate a plurality of labeled images. On the other hand, the lane departure detection apparatus 100 may further include an additional configuration.

2 is a block diagram of a lane departure detection apparatus according to another embodiment of the present disclosure;

2, the lane departure detection apparatus 100a includes a first input unit 110, an automatic annotation unit 120, a CNN (Convolution Neural Network) 130, a second input unit 140, and an output unit 150, . ≪ / RTI > The first input unit 110 and the automatic annotation unit 120 are the same as those shown in FIG.

CNN 130 learns about lane departure using the labeled composite image. The specific process by which the CNN 130 learns about lane departure will be described later. When the CNN 130 completes the learning of the lane departure using the composite image, the lane departure detection apparatus 100a can receive the actual image and sense the lane departure.

The second input unit 140 may receive a determination target image including a lane. For example, the second input unit 140 may include a communication module connected to the first input unit 110 through a communication interface, an input terminal connected to the input interface, and the like. That is, when the first input unit 110 is implemented as a communication module, the lane departure detection apparatus 100a can receive the determination target image including the lane using the wire / wireless communication method. Alternatively, when the first input unit 110 is implemented as an input terminal, the lane departure detection apparatus 100a may receive the determination target image including the lane from the internal and external storage devices. Also, the second input unit 140 may include a camera. That is, when the second input unit 140 is implemented by a camera, the lane departure detection apparatus 100a may receive an actual driving image including a lane photographed by the camera. When the second input unit 140 is implemented by a camera, the actual driving image including the lane may be the determination target image.

The CNN 130 may sense the lane from the determination target image. Then, the CNN 130 can determine whether or not the lane departure is based on the sensed lane and the learned lane departure data. That is, the CNN 130 can judge whether the normal operation, the left lane departure, or the right lane departure from the determination target image.

The output unit 150 may output the lane departure warning based on the lane departure detection result. For example, the output 150 may include a buzzer, a speaker, a motor, a haptic module, an LED, a display, and the like. When the output unit 150 is implemented as a buzzer, the output unit 150 may output various types of warning sounds. When the output unit 150 is implemented as a speaker, the output unit 150 can output the left lane departure or the right lane departure voice. When the output unit 150 is implemented as a motor or a haptic module, the output unit 150 can output vibrations of various patterns. When the output unit 150 is implemented by an LED, the output unit 150 may output light of various patterns. When the output unit 150 is implemented as a display, the output unit 150 may display a message such as a left lane departure or a right lane departure. The output unit 150 may be implemented as one unit or as a plurality of units. For example, when the output unit 150 is implemented with a buzzer, a motor, and an LED, the output unit 150 may output a warning sound, vibration, and light at the same time.

On the other hand, the CNN 130 may determine not only the left lane departure or the right lane departure. In one embodiment, the CNN 130 may determine that the left lane departure is 10%, 20%, or 30%. When the CNN 130 determines the degree of departure of the lane, the output unit 150 outputs a predetermined type of warning sound, vibration pattern, light pattern, voice output of a predetermined period, Output.

As described above, the lane departure detection apparatus 100a may include a first input unit 110, an automatic annotation unit 120, a CNN 130, a second input unit 140, and an output unit 150. On the other hand, the lane departure detection apparatus 100a may be implemented as a separate apparatus from the learning apparatus 100a-1 and the application apparatus 100a-2. That is, the learning apparatus 100a-1 of the lane departure detection apparatus 100a may include only the first input unit 110 and the automatic annotation unit 120 to generate the labeling image and learn the CNN 130. [ The application apparatus 100a-2 may determine whether or not the lane departure is possible by using the learned CNN 130 including the CNN 130, the second input unit 140, and the output unit 150 have.

3 is a view for explaining a process of labeling image data according to an embodiment of the present disclosure.

Referring to FIG. 3, the structure of the automatic annotation portion is shown. First, the automatic annotation unit receives the image through the input unit (S11). The video image is a normal image that does not deviate from the lane, and may be a video image of one frame. The automatic annotation unit may generate a composite image using an arbitrary deformation value (S12). For example, any deformation value may be any rotational deformation value or any shift deformation value. The automatic annotation unit can generate a plurality of composite images by applying various arbitrary deformation values from one video image.

The automatic annotation unit may compare an absolute value of an arbitrary deformation value with a predetermined threshold value for each of the generated composite images (S13). For example, the predetermined threshold value may be a rotation threshold value, and may be a shift threshold value. The automatic annotation section compares the rotation threshold value with any rotation deformation value when any deformation value is any rotation deformation value, and compares it with the shift threshold value when any deformation value is any shift deformation value.

The automatic annotation unit compares the absolute value of an arbitrary deformation value with a predetermined threshold value, and can label the normal image if the absolute value of an arbitrary deformation value is smaller than a predetermined threshold value (S14-1). Then, the automatic annotation section can label the image as an escape image when the absolute value of an arbitrary deformation value is larger than a preset threshold value (S14-2). It is not shown in FIG. 3 when a certain distortion value is equal to a predetermined threshold value. However, if the arbitrary deformation value is equal to the predetermined threshold value, the automatic annotation section may be set to label the image with one of the normal image or the escape image.

As described above, since the lane departure detection apparatus can automatically generate a plurality of labeling images from a single image, a labeling image for CNN learning can be generated easily and quickly. The lane departure detection apparatus can learn the deep neural network using the generated labeling image.

4 is a diagram for explaining a process of learning lane departure according to an embodiment of the present disclosure.

Referring to FIG. 4, the structure of the CNN 22 is shown. In FIG. 4, CNNs 22 each including three convolution layers, a ReLU (Rectified Linear Unit) layer, and a pulling layer are shown, but the number of layers can be variously set. The CNN 22 can receive the labeling image 21. For example, the labeled image may be a 360 x 120 size and may be a color image containing RGB data. The labeling image input to the CNN 22 may be an image obtained by subtracting the average of all training images before being input to the CNN 22.

The labeling image input to the CNN 22 may be generated as a first matrix map by a predetermined filter in the first convolution layer. The generated first matrix map is passed to a first ReLU layer using ReLU as an activation function. The activation function is a function that is activated when the action potential signal transmitted to the human synapse exceeds the minimum stimulus value and is transmitted to the next neuron. That is, the ReLU layer compares the generated first matrix map with 0 to generate a second matrix map in which a large value is returned.

The second matrix map processed in the first ReLU layer is transferred to the first pooling layer. In one embodiment, the pooling layer may perform average pooling. The pooling layer serves to reduce the size of the image, and the average pooling means a method of taking an average of the pixels within the pooling window. The map (or image) processed in the first pooling layer can be resized to half the size of the original image. Therefore, the image processed in the CNN 22 shown in Fig. 4 can be output as a feature map of 45 x 15 size.

The CNN 22 shown in FIG. 4 includes two FCs (Fully connected layers), each of which can have 1024 nodes. The number of FCs and the number of nodes described above are one embodiment, and the CNN 22 may include various numbers of FCs and nodes depending on the implementation. FC can connect the characteristics of the characteristic map and the characteristic map to which the characteristics are connected can be classified into one of three classes through the process of probability of soft max. For example, the three classes may include normal driving, left lane departure, and right lane departure.

In the embodiment shown in FIG. 4, three classes of lane departure are classified according to the implementation method. However, according to the implementation method, the CNN 22 is classified into the left 10% lane departure, left 20% lane departure, left 30% lane departure, Departure, right 20% lane departure and right 30% lane departure.

5 is a diagram illustrating the output of a lane departure network in accordance with one embodiment of the present disclosure;

Referring to FIG. 5, various images determined by CNN are shown. As shown in FIG. 5, the CNN can classify images into classes of left lane departure, right lane departure, and normal operation through a learning process.

Various embodiments of the lane departure detection apparatus have been described so far. A flowchart of the lane departure detection device control method will be described below.

6 is a flowchart of a method of controlling a lane departure detection apparatus according to an embodiment of the present disclosure.

Referring to FIG. 6, the lane departure detection apparatus receives an image including a lane (S610). For example, an image containing a lane may be a normal driving image without lane departure. The lane departure detection apparatus generates a composite image by applying an arbitrary deformation value to the input image (S620). The arbitrary deformation value may be a rotational deformation value for rotating the input image or a shift deformation value for shifting to the right or left. The lane departure detection apparatus can generate a plurality of composite images by applying various arbitrary deformation values to one input image.

The lane departure detection apparatus compares the absolute value of an arbitrary deformation value with a preset threshold value (S630). The predetermined threshold value may be a predetermined threshold value and may be a rotation threshold value, and may be a shift threshold value.

The lane departure detection apparatus labels a normal image if the absolute value of an arbitrary deformation value is smaller than a preset threshold (S640). If the absolute value of any deformation value is greater than a preset threshold value, the lane departure detection apparatus labels the image as an escape image (S650). The lane departure detection device can learn CNN using the labeling image.

The method according to the various embodiments described above may also be provided as a computer program product. The computer program product may include a software program itself or a non-transitory computer readable medium in which the software program is stored.

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

100, 100a: lane departure detection device 110: first input section
120: Automatic comment unit 130: CNN
140: second input unit 150: output unit

Claims (7)

Receiving an image including a lane;
Generating a composite image by applying a random deformation value to the input image; And
And labeling the generated composite image as an escape image if the absolute value of the random deformation value is greater than a predetermined threshold value and labeling the generated composite image as a normal image if the absolute value is smaller than the predetermined threshold value Of the lane departure detection device. The method according to claim 1,
The random deformation value may be determined by:
A rotational deformation value, and a shift deformation value. The method according to claim 1,
And learning the neural network using the labeled composite image. The method of claim 3,
Wherein learning the neural network comprises:
Generating a first matrix map by convoluting the labeled composite image, generating a second matrix map in which a large value is returned by comparing the generated first matrix map with 0, and generating the second matrix map by resizing generating a characteristic map by linking the generated characteristic map, and classifying the class of the labeled synthetic image based on the connected characteristic map. 5. The method of claim 4,
The class comprises:
A left lane departure, and a right lane departure of the lane departure detection apparatus. 6. The method of claim 5,
Receiving a determination target image including a lane;
Detecting a lane from the image to be judged; And
Determining a normal driving, a left lane departure, or a right lane departure based on the learned neural network and the position of the sensed lane. An input unit for receiving an image including a lane; And
And an automatic annotation unit for deforming and labeling the input image,
Wherein the automatic annotation unit comprises:
And generating a composite image by applying a random deformation value to the input image, and if the absolute value of the random deformation value is greater than a preset threshold value, labeling the generated composite image as an outgoing image, And if it is small, labels the generated composite image as a normal image.

Images