KR102384797B1 - Learning method based on segmentation labeling data and autonomous driving device using the same - Google Patents

Abstract

The present invention relates to an autonomous driving technology by automatically performing segmentation labeling related to an unpaved road, by projecting location information, which is obtained through a navigation sensor, onto an image. In this case, a learning method for autonomous driving includes: obtaining data including image information and location information, which are obtained, while a vehicle including a camera and a navigation sensor drives on the unpaved road; performing the segmentation labeling related to the unpaved road, based on the location information obtained through the navigation sensor and the image information obtained through the camera; learning a first network related to a driving situation of the vehicle, based on data to which the segmentation labeling is reflected; and learning a second network related to controlling the vehicle, based on the driving situation of the vehicle.

Description

A learning method based on segmentation labeling data and an autonomous driving device using the same

Embodiments of the present specification relate to a learning method related to autonomous driving using automated segmentation labeled data based on location information acquired through a navigation sensor and image information acquired through a camera, and an autonomous driving apparatus using the same.

Autonomous driving technology is a promising technology in the future, and autonomous driving technology is also rapidly developing according to the development of neural network related technologies. In autonomous driving, it is important to distinguish between road and non-road and control the vehicle accordingly. In particular, it is more important to classify a vehicle on an unstandardized unpaved road than a standardized paved road, but there was a problem in that it was more difficult to handle than a paved road to distinguish a road from a non-road on an unpaved road. In the past, the neural network was supervised and learned based on the labeling data manually classified by a person on the road or off-road for each pixel of the images acquired while driving. In addition, autonomous driving of military vehicles on unpaved roads is a more important factor in performing missions, and there is a problem in that it is difficult to apply autonomous driving technology on unpaved roads to unpaved roads.

Therefore, there is a need for a technology capable of automatically performing segmentation labeling for classifying a road and a non-road on an unpaved road in order to efficiently use resources.

An embodiment of the present specification is proposed to solve the above-described problem, and based on location information obtained through a navigation sensor and image information obtained through a camera, it is a learning method related to autonomous driving using automated segmentation labeled data. It's about technology. The technical problem to be achieved by this embodiment is not limited to the technical problems as described above, and other technical problems may be inferred from the following embodiments.

In order to achieve the above object, in a learning method related to autonomous driving according to an embodiment of the present specification, related data including image information and location information acquired while a vehicle equipped with a camera and a navigation sensor is driving on an unpaved road to confirm; performing segmentation labeling related to the unpaved road based on the location information obtained through the navigation sensor and the image information obtained through the camera; learning a first network related to the driving situation of the vehicle based on the data to which the segmentation labeling is reflected; and learning a second network related to the control of the vehicle based on the driving condition of the vehicle.

According to an embodiment, the performing of the segmentation labeling includes: m (m is an integer greater than or equal to 1) obtained through the navigation sensor corresponding to the i-th (i is an integer greater than or equal to 1) image information obtained through the camera. identifying location information; identifying a boundary line of the unpaved road corresponding to a direction perpendicular to the driving direction of the vehicle based on the m pieces of location information; and performing segmentation labeling related to the unpaved road based on the boundary line.

According to an embodiment, the step of identifying the boundary line of the unpaved road includes at least one piece of location information L corresponding to the left boundary line of the unpaved road and at least one corresponding to the right boundary line of the unpaved road based on the m pieces of location information. identifying one piece of location information R; and identifying a boundary line of the unpaved road based on the at least one piece of location information L and the at least one piece of location information R.

According to an embodiment, the learning of the first network may include learning according to supervised learning based on data to which the segmentation labeling is reflected.

According to an embodiment, the learning of the second network may include: checking control data of the vehicle according to the driving condition of the vehicle from related data obtained while driving the unpaved road; and learning the second network based on the control data.

According to an embodiment, the acquiring of the related data may include transforming the coordinates of the location information acquired through the navigation sensor using a preset matrix.

According to an embodiment, the obtaining of the related data may include obtaining coordinates on the image information from the position information converted using the preset matrix based on the focal length of the camera and the size information for the image information. It may include further steps.

According to an embodiment, the preset matrix may be determined based on a preset vector indicating a direction in which the camera faces, and the preset vector may be determined based on location information of the vehicle.

In order to achieve the above object, a non-transitory computer-readable storage medium according to an embodiment of the present specification includes a medium configured to store computer-readable instructions, wherein the computer-readable instructions are executed by a processor. In this case, the processor: checking related data including image information and location information acquired while driving a vehicle equipped with a camera and a navigation sensor while driving on an unpaved road; performing segmentation labeling related to the unpaved road based on the location information obtained through the navigation sensor and the image information obtained through the camera; learning a first network related to the driving situation of the vehicle based on the data to which the segmentation labeling is reflected; and learning a second network related to control of the vehicle based on the driving condition of the vehicle.

In order to achieve the above object, an autonomous driving apparatus according to an embodiment of the present specification includes: a communication unit; Memory; and confirming related data including image information and location information obtained while a vehicle equipped with a camera and a navigation sensor is driving on an unpaved road, and adding the location information obtained from the navigation sensor and the image information obtained from the camera Perform segmentation labeling related to the unpaved road based on the segmentation labeling, learn a first network related to the driving situation of the vehicle based on the data reflected by the segmentation labeling, and control the vehicle based on the driving situation of the vehicle It may include a controller for learning the second network.

According to an embodiment of the present specification, there are one or more of the following effects.

First, segmentation labeling data is automatically generated based on location information obtained through a navigation sensor and image information obtained through a camera, and a lot of time and money can be saved by supervising a neural network based on this.

Second, using the supervised neural network, it is possible to show more specialized performance in relation to the classification of roads and non-roads on non-standardized unpaved roads.

Third, a vehicle can be controlled based on a specialized autonomous driving model on an unpaved road using a neural network learned in relation to the driving situation of the vehicle and a neural network learned in relation to vehicle control.

Fourth, it may be more useful in performing missions by autonomous driving of military vehicles on unpaved roads.

Effects of the embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram illustrating image information acquired through a camera while driving a vehicle according to an embodiment, and FIG. 2 is a diagram illustrating location information acquired through a navigation sensor while driving a vehicle according to an embodiment.
3 is a diagram for describing a vector corresponding to a direction in which a camera is facing, according to an exemplary embodiment.
4 is a diagram according to an embodiment of visualizing m pieces of location information corresponding to i-th image information.
5 is a diagram illustrating a process of identifying a boundary line of an unpaved road for segmentation labeling, FIG. 6 is a diagram according to an embodiment showing image information in which segmentation labeling is reflected, and FIG. 7 is an unpaved road It is a diagram according to an embodiment for explaining a specific process of identifying a boundary line of
8 is a diagram for explaining learning of a first network according to an embodiment.
9 is a diagram for describing a learning method related to autonomous driving according to an exemplary embodiment.
10 is a block diagram of an autonomous driving apparatus according to an exemplary embodiment.

Terms used in the embodiments are selected as currently widely used general terms as possible while considering functions in the present disclosure, but may vary according to intentions or precedents of those of ordinary skill in the art, emergence of new technologies, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In the entire specification, when a part “includes” a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “~ unit” and “~ module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

The expression “at least one of a, b, and c” described throughout the specification is, 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c ', or 'all of a, b, and c'.

The "terminal" referred to below may be implemented as a computer or a portable terminal capable of accessing a server or other terminal through a network. Here, the computer includes, for example, a laptop, a desktop, and a laptop equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution) and other communication-based terminals, smartphones, tablet PCs, etc. It may include a handheld-based wireless communication device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

1 is a diagram illustrating image information acquired through a camera while driving a vehicle according to an exemplary embodiment, and FIG. 2 is a diagram illustrating location information obtained through a navigation sensor while driving a vehicle according to an exemplary embodiment.

Vehicles equipped with cameras and navigation sensors can acquire relevant data while driving on unpaved roads. For example, a vehicle equipped with a camera as shown in FIG. 1 may acquire image information while driving on an unpaved road, and a vehicle equipped with a navigation sensor as shown in FIG. 2 may acquire location information while driving on an unpaved road. In this case, the navigation sensor may include a global positioning system (GPS) and an inertial navigation unit (IMU), and when both the GPS and the IMU are used complementary to each other, location information with a relatively small error can be obtained.

When a vehicle equipped with a camera and a navigation sensor is driving, image information obtained through the camera and location information obtained through the navigation sensor are matched and stored for every frame. In this case, the location information may include, for example, (longitude, latitude, altitude) as information for identifying the location of the vehicle.

Here, the unpaved road is a non-standardized road, and may correspond to a road that cannot be regarded as a general road. For example, as shown in FIG. 1 , a road such as a road and a rough area where a road and a place other than the road are not clearly distinguished may be unpaved roads. Such unpaved roads are roads to which standards applied to general roads are not applied, and the learning method related to autonomous driving described in the present specification exhibits more specialized performance for autonomous driving on unpaved roads than general roads.

In order to project the position information acquired while driving the vehicle on the image information, it is necessary to convert position information expressed in the existing coordinate system into a new coordinate system using a preset matrix V. For example, it is necessary to convert coordinates of location information from the existing global coordinate system (GCS) to the camera coordinate system. This is because, in order to project the location information onto the image information, the navigation sensor needs to convert the location information acquired in the global coordinate system into a coordinate system based on the direction the camera sees. For example, the global coordinate system may be an existing coordinate system expressed as (X, Y, Z), and the camera coordinate system may be a coordinate system defined based on the position of the camera and the direction the camera sees.

라고 할 때 카메라 좌표계로 변환된 위치 정보

는 다음의 수학식 1을 통해 변환될 수 있다.Specifically, location information (eg, longitude, latitude, altitude) of the i-th frame

position information converted to the camera coordinate system when

can be transformed through Equation 1 below.

[Equation 1]

The preset matrix V may be determined based on a preset vector indicating a direction in which the camera faces. In this case, the preset vector may be determined based on location information of the vehicle, which will be described in detail below with reference to FIG. 3 .

3 is a diagram for describing a vector corresponding to a direction in which a camera is facing, according to an exemplary embodiment.

Referring to FIG. 3 , when the location information is obtained, the direction is not obtained together, so that the direction the camera is facing may be determined. The direction the camera is facing may be determined by a difference between previous location information and next location information. For example, the direction the camera is facing in the i-th frame may be determined by the difference between the position information of the i-1th frame and the position information of the i+1th frame, and the direction the camera is facing in the i+1th frame is It can be determined by the difference between the position information of the i-th frame and the position information of the i+2th frame, and the direction the camera is facing in the i+2th frame is the positional information of the i+1th frame and the position of the i+3th frame. It can be determined by the difference in information. This is compared to the case of using the position information of the i-th frame and the position information of the i+1th frame when determining the direction the camera is facing in the i-th frame, the position information of the i-1th frame and the position information of the i+1th frame This is because it can be determined with relatively accurate probability when using .

4 is a diagram illustrating m pieces of location information corresponding to i-th image information visualized according to an embodiment.

들을 시각화한 이미지를 확인할 수 있다. 카메라 좌표계 상의 위치 정보

는 이미지 좌표계 상의 위치 정보

로 다음의 수학식 2를 통해 변환될 수 있다. 4 , m pieces of location information corresponding to the i-th image information

You can check the visualized images. Location information in the camera coordinate system

is the location information on the image coordinate system

can be converted through Equation 2 below.

[Equation 2]

는 카메라의 초점 거리, W와 H는 이미지 정보에 대한 크기 정보로서 W는 이미지에 대한 폭, H는 이미지에 대한 높이에 대응할 수 있다. Here, the image coordinate system is position information on the image, and in Equation 2

is a focal length of the camera, W and H are size information for image information, W may correspond to a width for an image, and H may correspond to a height for an image.

5 is a diagram illustrating a process of identifying a boundary line of an unpaved road for segmentation labeling, FIG. 6 is a diagram according to an embodiment showing image information in which segmentation labeling is reflected, and FIG. 7 is an unpaved road It is a diagram according to an embodiment for explaining a specific process of identifying a boundary line of

Referring to FIG. 5 , image information obtained by visualizing dots on both sides of the location information shown in FIG. 4 for segmentation labeling may be checked. Here, the points on both sides of the location information displayed in FIG. 4 may be location information corresponding to the boundary line of the unpaved road. The location information corresponding to the boundary line of the unpaved road may be location information corresponding to a direction perpendicular to the driving direction of the vehicle based on the m pieces of location information of FIG. 4 . At this time, the location information corresponding to the direction perpendicular to the driving direction of the vehicle is based on the m pieces of location information of FIG. 4 , at least one piece of location information L corresponding to the left boundary line of the unpaved road and at least one corresponding to the right boundary line of the unpaved road It may correspond to one piece of location information R. A boundary line of the unpaved road may be identified based on the at least one piece of location information L and the at least one piece of location information R. At this time, at least one piece of location information corresponding to the boundary line of the unpaved road may be displayed on the image information as shown in FIG. 5 .

If the area made up of the dots on both sides visualized in FIG. 5 is filled with a specific color, image information to which segmentation labeling as shown in FIG. 6 is reflected can be acquired. Referring to FIG. 6 , an area filled with a specific color may correspond to an unpaved road, and an area not filled with a specific color may correspond to a non-paved road. Here, the unpaved road filled with a specific color is a road that can be driven when the vehicle autonomously drives, and an area not filled with a specific color can correspond to a road that is not drivable when the vehicle autonomously drives.

의 오른쪽 위치 정보 R와 왼쪽 위치 정보 L을 결정할 수 있다. 구체적으로, heading 벡터들 사이의 각

를 계산하고, 도로 폭을

라고 할 경우, 위치 정보 R은

이고 위치 정보 L은

이다. 여기서, heading 벡터

는 도 3에서 전술한 방법과 마찬가지로, 이전 위치 정보와 다음 위치 정보 간의 차이로 다음 수학식 3과 같이 결정될 수 있다. Referring to FIG. 7 , a detailed process of determining the location information L and the location information R corresponding to the boundary line of the unpaved road can be confirmed. i+kth position information in the camera coordinate system based on the i-th position information

It is possible to determine the right position information R and the left position information L of . Specifically, the angle between the heading vectors

Calculate the road width

If , the location information R is

and the location information L is

am. where, heading vector

Like the method described above with reference to FIG. 3 , as a difference between the previous location information and the next location information, ?

[Equation 3]

Here, the heading vector is a vector corresponding to the moving direction of the vehicle, and the position information L and the position information R corresponding to the boundary lines on both sides of the unpaved road may be displayed at a position perpendicular to the moving direction of the vehicle.

Based on at least one piece of location information L and at least one piece of location information R corresponding to the boundary lines on both sides of the unpaved road at a location perpendicular to the moving direction of the vehicle, as shown in FIG. 6 , image information in which segmentation labeling is reflected can be generated. .

The first network related to the driving situation of the vehicle may be learned based on the data to which the segmentation labeling is reflected. Details of the learning of the first network will be described with reference to FIG. 8 .

In addition, a second network related to the control of the vehicle corresponding to the driving situation of the vehicle may be learned. For example, a second network necessary for controlling a vehicle such as steering and throttle of the vehicle when driving on a curved road in an unpaved road may be learned.

As a vehicle equipped with a camera and navigation sensor drives on an unpaved road, not only image and location information but also steering and throttle values corresponding to the driving situation of the vehicle may be acquired for every frame. The second network may be learned based on the steering and throttle values for vehicle control corresponding to the determination result of the driving condition of the vehicle output through the first network. Accordingly, an autonomous driving model in which a control command for vehicle control is automatically performed in response to an image including an unpaved road may be generated using the first network and the second network.

8 is a diagram for explaining learning of a first network according to an embodiment.

Referring to FIG. 8 , a first network including an encoder 810 and a decoder 820 may be supervised. In this case, the first network may correspond to, for example, a Fully Convolutional Network (FCN) or Deeplabv1 to Deeplabv3+ as a neural network. At this time, the difference loss (850) derived by comparing the prediction (830) of the input image and the data (840) to which the segmentation labeling is reflected can be used for learning of the first network through backpropagation (860). there is.

The first network sufficiently learned through such a learning process may distinguish an unpaved road and a non-paved road area in the image with respect to the input image.

9 is a diagram for explaining a learning method related to autonomous driving according to an embodiment.

Referring to FIG. 9 , in step S910 , related data including image information and location information obtained by a vehicle equipped with a camera and a navigation sensor while driving on an unpaved road may be checked. In this case, image information may be obtained through a camera, and location information may be obtained through a navigation sensor. In addition, various information related to driving on an unpaved road may be acquired. Here, the coordinates of the position information obtained through the navigation sensor may be converted from the global coordinate system to the camera coordinate system using a preset matrix, and may also be converted into the image coordinate system based on Equation (2).

In step S920 , segmentation labeling related to the unpaved road may be performed based on the location information obtained through the navigation sensor and the image information obtained through the camera.

Specifically, it is possible to identify m pieces of location information corresponding to the i-th image information, and based on the m pieces of location information, identify the boundary line of the unpaved road corresponding to the direction perpendicular to the driving direction of the vehicle, and based on the boundary line, the unpaved road Segmentation labeling related to roads may be performed. More specifically, based on the m pieces of location information, it is possible to identify at least one piece of location information L corresponding to the left boundary line of the unpaved road and at least one piece of location information R corresponding to the right boundary line, and at least one piece of location information L and A boundary line of the unpaved road may be identified based on at least one piece of location information R. Segmentation labeling may be performed based on the identified boundary line of the unpaved road.

In operation S930, the first network related to the driving situation of the vehicle may be learned based on the data to which the segmentation labeling is reflected. Specifically, the first network may be learned according to supervised learning based on data to which segmentation labeling is reflected.

In operation S940 , a second network related to vehicle control may be learned based on the driving condition of the vehicle. Control data of the vehicle according to the driving condition of the vehicle can be checked from related data obtained while driving on an unpaved road. For example, while driving on an unpaved road, steering and throttle values according to the driving situation of the vehicle may be checked as control data. A second network for vehicle control for each driving situation of the vehicle may be learned based on the checked control data.

According to an embodiment, an autonomous driving model in which control data for a vehicle is automatically generated in response to an image including an unpaved road may be generated using the learned first network and the second network. Such an autonomous driving model can exhibit specialized performance on unpaved roads rather than standardized roads.

10 is a block diagram of an autonomous driving apparatus according to an exemplary embodiment.

According to an embodiment, the autonomous driving apparatus 1000 may include a communication unit 1010 , a memory 1020 , and a control unit 1030 . In the autonomous driving apparatus 1000 illustrated in FIG. 10 , only components related to the present embodiment are illustrated. Accordingly, it can be understood by those of ordinary skill in the art related to the present embodiment that other general-purpose components may be further included in addition to the components shown in FIG. 10 . Since the autonomous driving apparatus 1000 may perform the above-described learning method related to autonomous driving, a description of overlapping content will be omitted.

The communication unit 1010 may transmit/receive related information using wired/wireless communication. The controller 1030 may control the overall operation of the autonomous driving apparatus 1000 and process data and signals. The controller 1030 may be configured with at least one hardware unit. Also, the controller 1030 may operate by one or more software modules generated by executing program codes stored in the memory 1020 . The controller 1030 may execute a program code stored in the memory 1020 to control the overall operation of the autonomous driving apparatus 1000 and process data and signals. The control unit 1030 performs segmentation labeling related to the unpaved road based on the image information obtained from the camera and the location information obtained from the navigation sensor, and based on the data to which the segmentation labeling is reflected, a first network related to the driving situation of the vehicle It is possible to learn the second network related to the control of the vehicle based on the driving condition of the vehicle.

The electronic device or terminal according to the above-described embodiments includes a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a touch panel, and a key (key) , a user interface device such as a button, and the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes a magnetic storage medium (eg, read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and an optically readable medium (eg, CD-ROM). ) and DVD (Digital Versatile Disc)). The computer-readable recording medium may be distributed among network-connected computer systems, so that the computer-readable code may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed on a processor.

This embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented in any number of hardware and/or software configurations that perform specific functions. For example, an embodiment may be an integrated circuit configuration, such as memory, processing, logic, look-up table, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. can be hired Similar to how components may be implemented as software programming or software components, this embodiment includes various algorithms implemented in a combination of data structures, processes, routines, or other programming constructs, including C, C++, Java ( It can be implemented in a programming or scripting language such as Java), assembler, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, the present embodiment may employ the prior art for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means” and “configuration” may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in association with a processor or the like.

The above-described embodiments are merely examples, and other embodiments may be implemented within the scope of the claims to be described later.

Claims (10)

In a learning method related to autonomous driving performed by an autonomous driving device,
checking related data including image information and location information obtained while driving a vehicle equipped with a camera and a navigation sensor on an unstandardized unpaved road;
performing segmentation labeling related to the unpaved road based on the location information obtained through the navigation sensor and the image information obtained through the camera;
learning a first network related to the driving situation of the vehicle based on the data to which the segmentation labeling is reflected; and
and learning a second network related to the control of the vehicle based on the driving condition of the vehicle,
The step of performing the segmentation labeling comprises:
identifying m pieces of (m is an integer greater than or equal to 2) position information obtained through the navigation sensor corresponding to i-th (i is an integer greater than or equal to 2) image information obtained through the camera;
determining a direction vector of the camera corresponding to the i-th image information based on the position information corresponding to the i+1th image information and the position information corresponding to the i-1th image information;
determining an angle between a direction vector of the camera corresponding to the i-th image information and a direction vector of the camera corresponding to the i+1-th image information;
determining at least one piece of location information L corresponding to the left boundary of the unpaved road and at least one piece of location information R corresponding to the right boundary of the unpaved road by using the determined angle and the m pieces of location information;
identifying a boundary line of the unpaved road based on the at least one piece of location information L and the at least one piece of location information R; and
Comprising the step of performing segmentation labeling related to the unpaved road based on the boundary line,
How to learn. delete delete According to claim 1,
Learning the first network comprises:
Including the step of learning according to supervised learning based on the data to which the segmentation labeling is reflected,
How to learn. According to claim 1,
Learning the second network comprises:
checking control data of the vehicle according to the driving condition of the vehicle from the related data obtained while driving the unpaved road; and
learning the second network based on the control data;
How to learn. According to claim 1,
The step of confirming the relevant data is,
Containing the step of using a predetermined matrix to check the related data, the coordinates of the location information obtained through the navigation sensor are transformed,
How to learn. 7. The method of claim 6,
The step of confirming the relevant data is,
Further comprising the step of confirming the coordinates on the image information from the position information transformed using the preset matrix based on the focal length of the camera and the size information for the image information,
How to learn. 7. The method of claim 6,
The preset matrix is determined based on a preset vector indicating a direction in which the camera faces, and the preset vector is determined based on location information of the vehicle,
How to learn. A non-transitory computer-readable storage medium comprising:
A medium configured to store computer readable instructions, comprising:
The computer readable instructions, when executed by a processor, cause the processor to:
checking related data including image information and location information obtained while driving a vehicle equipped with a camera and a navigation sensor on an unstandardized unpaved road;
performing segmentation labeling related to the unpaved road based on the location information obtained through the navigation sensor and the image information obtained through the camera;
learning a first network related to the driving situation of the vehicle based on the data to which the segmentation labeling is reflected; and
and learning a second network related to the control of the vehicle based on the driving condition of the vehicle,
The step of performing the segmentation labeling comprises:
identifying m pieces of (m is an integer greater than or equal to 2) position information obtained through the navigation sensor corresponding to i-th (i is an integer greater than or equal to 2) image information obtained through the camera;
determining a direction vector of the camera corresponding to the i-th image information based on the position information corresponding to the i+1th image information and the position information corresponding to the i-1th image information;
determining an angle between a direction vector of the camera corresponding to the i-th image information and a direction vector of the camera corresponding to the i+1-th image information;
determining at least one piece of location information L corresponding to the left boundary of the unpaved road and at least one piece of location information R corresponding to the right boundary of the unpaved road by using the determined angle and the m pieces of location information;
identifying a boundary line of the unpaved road based on the at least one piece of location information L and the at least one piece of location information R; and
and performing a segmentation labeling associated with the unpaved road based on the boundary line to perform a learning method. communication department;
Memory; and
A vehicle equipped with a camera and a navigation sensor checks related data including image information and location information acquired while driving on a non-standardized unpaved road, and the location information acquired from the navigation sensor and the image acquired from the camera Segmentation labeling related to the unpaved road is performed based on the information, a first network related to the driving condition of the vehicle is learned based on the data reflected by the segmentation labeling, and the vehicle is controlled based on the driving condition of the vehicle Includes a controller (controller) for learning the second network related to,
The control unit is
Identifies m pieces of (m is an integer greater than or equal to 2) position information obtained through the navigation sensor corresponding to the i-th (i is an integer greater than or equal to 2) image information obtained through the camera, and corresponds to the i+1-th image information determining the direction vector of the camera corresponding to the i-th image information based on the position information corresponding to the i-th image information and the position information corresponding to the i-th image information, and the direction vector of the camera corresponding to the i-th image information and the Determine an angle between the direction vector of the camera corresponding to the i+1th image information, and use the determined angle and the m pieces of location information to determine at least one piece of location information L corresponding to the left boundary of the unpaved road and the at least one piece of location information R corresponding to the right boundary line of the unpaved road is determined, and the boundary line of the unpaved road is identified based on the at least one location information L and the at least one piece of location information R, and based on the boundary line performing segmentation labeling related to the unpaved road,
autonomous driving device.

Images