KR20210098027A - Method and Apparatus for Reading Road Friction Coefficient - Google Patents

Abstract

An embodiment of the present invention relates to an apparatus and a method for decoding a road friction coefficient, the apparatus which learns labeling data labeling road friction coefficient data calculated from ESC using road image data obtained from a camera to accurately estimate road friction coefficient information using only an image of the camera even during normal driving and then provides the information to a requiring control device as driving control information to improve stability of an autonomous vehicle.

Description

Road surface friction coefficient reading device and method {Method and Apparatus for Reading Road Friction Coefficient}

This embodiment relates to a road surface friction coefficient reading apparatus and method. More specifically, it relates to a deep learning-based road friction coefficient reading apparatus and method using vehicle camera and ESC data.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

As the intelligence of vehicles progresses, research on autonomous vehicles is being actively conducted.

Among them, many research achievements have already been made on the development of an active safety device that helps the vehicle avoid danger by intervening on its own in a situation where an accident may occur due to an unexpected situation. A typical example is the 'Anti Lock Brake System (ABS)' function that prevents the wheels from locking by using an electric device that controls the brake hydraulic pressure. The anti-lock braking system is a technology for preventing or reducing accidents when there is a risk of collision while driving, and is a basic technology for the safety of autonomous vehicles.

Although the operation method and configuration of such an anti-locking wheel braking system are different for each manufacturer, they are gradually being commercialized. However, since there are various uncertainties on the road, it is very difficult for the anti-lock braking system to maintain the same performance in any environment.

One of the representative measures of road uncertainty is the friction coefficient of the road surface. Since the friction coefficient of the road directly affects the braking distance of the vehicle, it has a very large effect on the performance of the anti-lock braking system.

In order to improve the performance of the anti-lock braking system, a technology for reading the road surface condition using a sensor such as a camera while driving has been proposed. There is a problem that the road surface friction coefficient information cannot be known. That is, according to the prior art, it is possible to calculate the road friction coefficient only in specific cases that utilize the tire grip to the maximum, such as ABS braking and ESC avoidance maneuver.

In this embodiment, by learning the labeling data that labels the road surface friction coefficient data calculated by ESC on the road surface image data obtained from the camera, it is possible to accurately estimate the road surface friction coefficient information only with the image of the camera even during normal driving, and this The main purpose is to improve driving stability for autonomous vehicles by providing necessary driving control information as driving control information.

The present embodiment includes: a data collection unit for collecting road surface image data obtained by photographing a road on which the vehicle is traveling from a camera mounted on the vehicle; a road friction coefficient estimating unit including a pre-trained image learning model and estimating road surface friction coefficient information corresponding to the road surface by inputting the road surface image data to the image learning model; and an output unit for providing the information on the coefficient of road friction as control information for driving control of the vehicle.

In addition, according to another aspect of the present embodiment, the process of collecting road surface image data obtained by photographing a road on which the vehicle is traveling from a camera mounted on the vehicle; estimating road friction coefficient information corresponding to the road surface by inputting an image learning model trained in advance on the road surface image data; and providing the road friction coefficient information as control information for driving control of the vehicle.

In addition, according to another aspect of the present embodiment, the data acquisition unit for acquiring the road surface image data and road friction coefficient information collected for the road surface on which the vehicle is driving, respectively, from the camera and the ESC; a labeling data generator generating labeling data by labeling the road surface friction coefficient information on the road surface image data; and a training unit for training a deep learning-based image learning model using the labeling data.

In addition, according to another aspect of the present embodiment, the process of acquiring road surface image data and road friction coefficient information collected for the road surface on which the vehicle is driving, respectively, from the camera and the ESC; generating labeling data by labeling the road surface friction coefficient information on the road surface image data; and training a deep learning-based image learning model using the labeling data.

As described above, according to this embodiment, by learning the labeling data that labels the road surface friction coefficient data calculated by the ESC on the road surface image data secured from the camera, the road surface friction coefficient information is accurately estimated even with the camera image even during general driving. This can be done, and by providing it to a necessary control device as driving control information, there is an effect of improving driving stability for an autonomous vehicle.

1 is an exemplary diagram for explaining a related configuration for a method of reading a road surface friction coefficient according to the present embodiment.
2 is a block diagram schematically showing a road friction coefficient reading apparatus according to the present embodiment.
3 is a block diagram schematically showing a learning apparatus for reading a road friction coefficient according to the present embodiment.
4 is a flowchart for explaining a learning method for reading a road friction coefficient according to the present embodiment.
5 is a flowchart for explaining a method of reading a road surface friction coefficient according to the present embodiment.
6 is an exemplary view illustrating a method of utilizing road image data according to the present embodiment.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated in different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

Recently, deep learning techniques have been applied to image recognition algorithms, showing improved performance in image recognition. In the case of this embodiment, based on such deep learning, a new method of estimating road friction coefficient information from a camera image is proposed.

In more detail, by learning the labeling data obtained by labeling the road surface friction coefficient data calculated by ESC (electronic stability control) on the road surface image data obtained from the camera, it is possible to accurately estimate the road surface friction coefficient information only with the camera image even during normal driving. We suggest a method for making it possible and how to use it.

On the other hand, this embodiment describes a method of reading the road friction coefficient in an autonomous or semi-autonomous driving vehicle equipped with a camera and ESC, but this is for convenience of explanation, and is used for various moving means equipped with a camera and ESC can be applied.

1 is an exemplary diagram for explaining a related configuration for a method of reading a road surface friction coefficient according to the present embodiment.

The camera 110 is mounted on one side of the vehicle, and provides an image of a surrounding area of the vehicle 100 . In the present embodiment, the camera 110 may be a photographing device that acquires road surface image data obtained by photographing a road on which the vehicle 100 is traveling.

The camera 110 provides the obtained road surface image data to the road friction coefficient reading device 130 . Meanwhile, in the present embodiment, the road friction coefficient reading device 130 is a kind of local electronic control device that performs a function of storing and processing image data photographed from the camera 110 in connection with the camera 110 . (ECU: electronic control unit) may be. In another embodiment, the road friction coefficient reading device 130 may be implemented as one component on the local electronic control device.

The ESC 120 shows the most effective braking performance by identifying the state of the vehicle through interworking with various sensors, and refers to a device that controls the driving force of each wheel during driving.

The ESC 120 assists the vehicle to respond to a dangerous situation before the driver's judgment or by judging a dangerous situation that the driver is easy to miss, for example, a function that helps the driver to safely drive by controlling the vehicle posture and reducing concentration carry out

The ESC 120 according to this embodiment calculates road friction coefficient information for the road surface on which the vehicle 100 is traveling based on data collected in the vehicle control process, and reads the calculated road friction coefficient information. It is provided as learning data for

The ESC 120 synthesizes longitudinal acceleration (ax) and lateral acceleration (ay) based on sensor data collected using an acceleration sensor, and calculates road friction coefficient information based on the synthesized acceleration. In another embodiment, the ESC 120 calculates road friction coefficient information based on a force applied to each wheel of the vehicle 100 , for example, vertical load information, longitudinal force information, and lateral force information applied to one wheel. can do.

In the present embodiment, the ESC 120 preferably provides, but not necessarily limited to, only the information on the road friction coefficient calculated in a specific situation as learning data for reading the road friction coefficient.

The ESC 120 may provide, as learning data, only the road friction coefficient information calculated when a preset event situation occurs in relation to the learning of the road friction coefficient information. In this case, the preset event situation may be determined according to the prediction accuracy of the road friction coefficient information calculated by the ESC 120 when the corresponding event situation occurs. For example, the ESC 120 may set a situation when the prediction accuracy for the calculated road friction coefficient information is equal to or greater than a preset threshold as an event situation.

In this embodiment, an anti-lock brake system (ABS), a vehicle dynamic control (VDC), and a traction control system (TCS) are driven in response to a preset event situation. , that is, a situation in which the tire grip is maximally utilized is exemplified and described, but is not necessarily limited thereto.

The ESC 120 generates event generation information when it is determined that a preset event situation has occurred in relation to learning the road friction coefficient information, and transmits it to the road friction coefficient reading device 130 together with the calculated road friction coefficient information. . This has the effect of allowing the road friction coefficient reading device 130 to utilize only meaningful data as learning data.

The road friction coefficient reading apparatus 130 calculates road surface friction coefficient information corresponding to the road surface on which the vehicle 100 is currently driving from the road surface image data secured from the camera 110 based on deep learning.

In this embodiment, the road friction coefficient reading device 130 includes a pre-trained image learning model, and inputs road surface image data collected from the camera 110 to the image learning model to calculate road surface friction coefficient information .

To this end, the road friction coefficient reading apparatus 130 may first perform training on the image learning model based on the road surface image data secured from the camera 110 and the road surface friction coefficient information secured from the ESC 120 . .

The road friction coefficient reading device 130 generates labeling data based on the road surface image data collected from the camera 110 and the road surface friction coefficient information collected from the ESC, and uses the generated labeling data to learn the image learning model. It is used as data to perform training on the image learning model.

On the other hand, in the present embodiment, it is described as an example that training for the image learning model is directly performed by the road friction coefficient reading device 130 , but is not necessarily limited thereto. For example, in the present embodiment, a separate device for learning may be provided, and in this case, the road friction coefficient reading device 130 may be implemented in the form of receiving a trained image learning model from the corresponding device.

In this embodiment, the road friction coefficient reading device 130 is implemented to estimate the road surface friction coefficient information only with the data of the camera 110, thereby overcoming the limitation that conventionally could not estimate the road surface friction coefficient during general driving. there is an effect

The road friction coefficient reading device 130 transmits the calculated road friction coefficient information to the peripheral device using an in-vehicle communication method such as CAN communication (controller area network). For example, the road friction coefficient reading device 130 may provide the road surface friction coefficient information as one of the driving control information to a necessary controller. This has the effect of allowing the vehicle to avoid danger by intervening on its own in a situation where an accident may occur due to an unexpected situation.

2 is a block diagram schematically showing a road friction coefficient reading apparatus according to the present embodiment.

As shown in FIG. 2 , the road friction coefficient reading apparatus 130 according to the present embodiment includes a data collection unit 200 , a learning unit 210 , a road friction coefficient estimation unit 220 , and an output unit 230 . include Here, the components included in the road friction coefficient reading device 130 are not necessarily limited thereto. For example, in this embodiment, the learning unit 210 may be implemented as a separate device that is interlocked with the road friction coefficient reading device 130 rather than a component of the road friction coefficient reading device 130 .

The data collection unit 200 performs a function of acquiring data for reading a road friction coefficient in connection with various sensors mounted on the vehicle 100 .

The data collection unit 200 is linked with the camera 110 to collect road surface image data obtained by photographing the road surface on which the vehicle 100 is driving by the camera 110 .

The data collection unit 200 is linked with the ESC 120 to collect information on the road friction coefficient calculated by the ESC 12 with respect to the road surface on which the vehicle 100 is traveling. In this embodiment, the data collection unit 200 preferably collects the above-mentioned road friction coefficient information only when training on the image learning model is performed, but is not necessarily limited thereto.

The learning unit 210 performs a function of training an image learning model for estimating information on the road surface friction coefficient.

In this embodiment, the learning unit 210 may be implemented as a learning device 300 which is a separate independent device that is interlocked with the road friction coefficient reading device 130 . 3 is a block diagram schematically showing a learning apparatus for reading a road friction coefficient according to the present embodiment. The functional modules of the learning apparatus 300 shown in FIG. 3 are on the learning unit 210 shown in FIG. can be implemented in the same way.

Hereinafter, the operation of the learning unit 210 according to the present embodiment will be described with reference to FIG. 3 together.

The learning unit 210 according to the present embodiment includes a data acquisition unit 310 , a labeling data generation unit 320 , and a training unit 330 .

The data acquisition unit 310 performs a function of collecting learning data for training an image learning model.

The data acquisition unit 310 acquires, from the ESC 120 , the road friction coefficient information collected with respect to the road surface on which the vehicle 100 is traveling. When the learning unit 210 is implemented as a component of the road friction coefficient reading device 130 , the data acquisition unit 310 may acquire the information on the road surface friction coefficient by using the data collection unit 200 .

The data acquisition unit 310 collects road surface image data of the camera 110 corresponding to the acquired road surface friction coefficient information.

On the other hand, the road surface image data from the camera 110 is an image of a road surface area located in front of the vehicle 100 , and there is actually a certain distance difference between the road surface area in the image and the vehicle 100 .

Due to this, the data acquisition unit 310 according to the present embodiment collects the road surface image data of the camera 110, the distance between the road surface area in the image and the vehicle 100, for example, the tire of the vehicle 100 In consideration of the difference, accurate road surface image data corresponding to the corresponding road surface friction coefficient information can be collected.

Meanwhile, in this embodiment, the data acquisition unit 310 selectively collects road image data corresponding to the corresponding road friction coefficient information only when event occurrence information is collected from the ESC 120 together with the road friction coefficient information. can do.

That is, when event occurrence information in the data transmitted from the ESC 120 is detected together, the data acquisition unit 310 determines that the reliability of the road friction coefficient information is high, extracts the road friction coefficient information, and extracts the extracted road friction coefficient information. It is possible to collect road image data corresponding to the coefficient information. In this case, the data acquisition unit 310 may specify and acquire a value between 20 ms and 1 s in the data as the road friction coefficient information in order to accurately extract the road friction coefficient information. In addition, data of the initial 1 s and the last 1 s that may include event occurrence information may be omitted.

Meanwhile, in the present embodiment, the event occurrence information is generated by the ESC 120 only when a preset event situation occurs in relation to the learning of the road surface friction coefficient information. In this case, the preset event situation may be determined according to the prediction accuracy of the road friction coefficient information calculated by the ESC 120 when the corresponding event situation occurs. For example, the ESC 120 may set a situation when the prediction accuracy for the calculated road friction coefficient information is equal to or greater than a preset threshold as an event situation.

The data acquisition unit 310 may perform pre-processing on the collected road image data. For example, referring to FIG. 6 , the data acquisition unit 310 may detect a road surface area in the collected road surface image data, and may extract and provide only image data corresponding to the detected road surface area. This has the effect of simplifying the calculation in the training process performed later by reducing the pixel area to be calculated.

The labeling data generating unit 320 acquires the collected road image data and the road friction coefficient information using the data acquisition unit 310 and generates learning data for training the image learning model based on this.

In this embodiment, the labeling data generation unit 320 generates labeling data based on the road surface image data and road friction coefficient information collected using the data acquisition unit 310, and uses the generated labeling data as learning data. to provide.

For example, the labeling data generator 320 may generate the labeling data by labeling the road friction coefficient information on the collected road surface image data.

The training unit 330 trains a deep learning-based image learning model by using the labeling data generated using the labeling data generation unit 320 as training data. On the other hand, the specific method for the training unit 330 to train the learning model based on the learning data is common in the relevant field, so a detailed description will be omitted.

According to the training of the training unit 330 according to the present embodiment, the image learning model is learned to estimate the road friction coefficient information only from the image of the camera 110 . That is, the image learning model learns and stores road friction coefficient information measured by the ESC 120 for each road image data having different characteristic information.

The road friction coefficient estimating unit 220 includes an image learning model trained in advance through the learning unit 210, and through a learning process using the same, the road surface friction coefficient information corresponding to the road surface on which the vehicle 100 is currently driving. perform the function of estimating

The road friction coefficient estimating unit 220 estimates road surface friction coefficient information corresponding to the road surface by applying the road surface image data of the camera 110 collected using the data collection unit 200 to the image learning model.

The road friction coefficient estimator 220 applies the road image data to the image learning model to extract matching learning data with respect to the road image data having the same or similar characteristic information as the road image data, and based on the extracted learning data to estimate the road friction coefficient information. That is, the road friction coefficient estimator 220 estimates the previous road surface friction coefficient information labeled with respect to the previous road surface image data having the same or similar characteristic information as the road surface image data collected at the current time point as the current road surface friction coefficient information. can

The output unit 230 provides the road friction coefficient information estimated by using the road friction coefficient estimator 220 as control information for driving control of the vehicle 100 .

The output unit 230 transmits road friction coefficient information to a necessary controller using CAN communication, and through this, vehicle control based on the road surface friction coefficient information can be performed even during general driving. By doing so, there is an effect that the control performance of the autonomous vehicle can be improved.

4 is a flowchart for explaining a learning method for reading a road friction coefficient according to the present embodiment.

The road surface friction coefficient reading apparatus 130 obtains the road surface friction coefficient information collected with respect to the road surface on which the vehicle 100 is traveling from the ESC 120 (S402).

The road friction coefficient reading device 130 checks whether event occurrence information is collected together with the road friction coefficient information (S404). In step S404 , the road friction coefficient reading device 130 may receive the event occurrence information calculated from the ESC 120 only when a preset event situation occurs in relation to learning the road friction coefficient information.

Such event occurrence information indicates that the collected road surface friction coefficient information is meaningful data for learning.

When the collection of event occurrence information is confirmed in step S404, the road friction coefficient reading device 130 collects road image data of the camera 110 corresponding to the road friction coefficient information obtained in step S402 (S406). In step S406 , the road friction coefficient reading apparatus 130 considers the actual distance difference between the road surface area in the image and the vehicle 100 so that accurate road surface image data corresponding to the actual corresponding road surface friction coefficient information can be collected.

The road friction coefficient reading apparatus 130 may detect a road surface area in the collected road surface image data, and may extract and provide only image data corresponding to the detected road surface area.

The road friction coefficient reading device 130 labels the road surface friction coefficient information of step S402 on the road image data collected in step S406 to generate labeling data (S408).

The road friction coefficient reading device 130 trains a deep learning-based image learning model by using the labeling data generated in step S408 as learning data (S410).

5 is a flowchart for explaining a method of reading a road surface friction coefficient according to the present embodiment.

The road friction coefficient reading device 130 is linked with the camera 110 to collect road surface image data obtained by photographing the road surface on which the vehicle 100 is traveling by the camera 110 ( S502 ).

The road friction coefficient reading apparatus 130 estimates the road surface friction coefficient information corresponding to the road surface by inputting the road surface image data collected in step S502 to the image learning model trained in advance (S504). In step S504, the road friction coefficient reading device 130 applies the road image data to the image learning model to extract matching learning data for the road image data having the same or similar characteristic information as the road image data, and extracts the extracted learning data. Estimate the road friction coefficient information based on the data.

The road friction coefficient reading device 130 provides the road friction coefficient information estimated in step S504 as control information for driving control of the vehicle 100 ( S506 ). In step S506, the road friction coefficient reading device 130 transmits the road surface friction coefficient information to the necessary controller using CAN communication, so that vehicle control based on the road surface friction coefficient information can be performed even during general driving. .

Here, since steps S402 to S410 and steps S502 to S506 correspond to the operations of the respective components of the road friction coefficient reading apparatus 130 described above, further detailed descriptions are omitted.

Meanwhile, although it is described that each process is sequentially executed in FIGS. 4 and 5 , the present invention is not limited thereto. In other words, since the process described in FIGS. 4 and 5 may be changed and implemented or one or more processes may be executed in parallel, FIGS. 4 and 5 are not limited to a time-series sequence.

As described above, the learning method and the road friction coefficient reading method of the road friction coefficient reading apparatus 130 described in FIGS. 4 and 5 are implemented as a program and a readable recording medium (CD-ROM, RAM) using computer software. , ROM, memory card, hard disk, magneto-optical disk, storage device, etc.).

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: vehicle 110: camera
120: ESC 130: road friction coefficient reading device
200: data collection unit 210: learning unit
220: road friction coefficient estimation unit 230: output unit
300: learning device 310: data acquisition unit
320: labeling data generation unit 330: training unit

Claims (14)

a data collection unit for collecting road surface image data obtained by photographing a road on which the vehicle is traveling from a camera mounted on the vehicle;
a road friction coefficient estimator including a pre-trained image learning model and estimating information on the road surface friction coefficient corresponding to the road surface by inputting the road surface image data to the image learning model; and
An output unit that provides the road surface friction coefficient information as control information for driving control of the vehicle
Road friction coefficient reading device comprising a. The method of claim 1,
The image learning model is a deep learning model, and it reads the friction coefficient of the road, characterized in that it learns and stores the information on the friction coefficient measured by the ESC (electronic stability control) for each road image data having different characteristic information. Device. 3. The method of claim 2,
The road friction coefficient estimating unit,
The road surface image data is applied to the image learning model to extract matching learning data with respect to the road image data having the same or similar characteristic information as the road image data, and the road friction coefficient information based on the extracted learning data Road friction coefficient reading device, characterized in that for estimating. The method of claim 1,
A learning unit that generates labeling data based on the road surface image data collected from the camera and the road surface friction coefficient information collected from the ESC of the vehicle, and uses the labeling data as learning data of the image learning model. Road friction coefficient reading device, characterized in that. 5. The method of claim 4,
The learning unit,
Collecting the road surface image data of the camera corresponding to the road surface friction coefficient information collected from the ESC, and labeling the road surface friction coefficient information collected from the ESC on the collected road surface image data to generate the labeling data road friction coefficient reading device. 6. The method of claim 5,
The learning unit,
The road friction coefficient reading device, characterized in that collecting the road surface image data of the camera when the event occurrence information is collected together with the road surface friction coefficient information from the ESC of the vehicle. 7. The method of claim 6,
The event occurrence information is
The road friction coefficient reading apparatus, characterized in that generated by the ESC when a preset event situation occurs in relation to learning the road friction coefficient information. 8. The method of claim 7,
The preset event situation is,
The road friction coefficient reading device, characterized in that it is determined according to the prediction accuracy of the road friction coefficient information calculated by the ESC when the event situation occurs. 5. The method of claim 4,
The learning unit,
A road friction coefficient reading apparatus, characterized in that the labeling data is generated by detecting a road surface area in the road surface image data collected from the camera, and extracting the image data of the detected road surface area. 5. The method of claim 4,
The learning unit,
and generating the labeling data in consideration of actual distance information between a road surface area on the road surface image data collected from the camera and the vehicle. a process of collecting road surface image data obtained by photographing a road on which the vehicle is traveling from a camera mounted on the vehicle;
estimating road friction coefficient information corresponding to the road surface by inputting an image learning model trained in advance on the road surface image data; and
A process of providing the road surface friction coefficient information as control information for driving control of the vehicle
Road friction coefficient reading method comprising a. 12. The method of claim 11,
Generating labeling data based on the road surface image data collected from the camera and the road surface friction coefficient information collected from the ESC of the vehicle, and using the labeling data as learning data to train the image learning model A method of reading the friction coefficient of the road, characterized in that. a data acquisition unit that acquires road surface image data and road friction coefficient information collected from the camera and the ESC for the road on which the vehicle is driving, respectively;
a labeling data generator generating labeling data by labeling the road surface friction coefficient information on the road surface image data; and
Training unit for training a deep learning-based image learning model using the labeling data
A learning apparatus for reading an image-based friction coefficient of a road surface, characterized in that it comprises a. acquiring road surface image data and road friction coefficient information collected for the road surface on which the vehicle is driving, respectively, from the camera and the ESC;
generating labeling data by labeling the road surface friction coefficient information on the road surface image data; and
The process of training a deep learning-based image learning model using the labeling data
A learning method for reading an image-based friction coefficient of a road surface, comprising:

Images