KR102260574B1 - Deep learning-based autonomous vehicle auditory system for the visually impaired and method thereof - Google Patents

Abstract

The deep learning-based autonomous driving vehicle auditory system and method for the visually impaired according to the embodiment enable the visually impaired to use the autonomous driving vehicle, and intuitively provide the autonomous driving vehicle driving information and state information as an auditory signal to general users. to improve the speed of delivering information to the driver, information recognition speed, and driver reaction speed. In addition, the autonomous vehicle can be used without visual fatigue, and vehicle state information can be recognized only with an auditory signal. It is possible to inform the occupants of various information than the conventional instrument panel without compromising the real-time of the conventional cluster system. It enhances the accessibility of autonomous vehicles for the visually impaired by promptly alerting the visually impaired by visualizing situations that need to be confirmed visually, such as vehicle status and accident conditions. Although the conventional instrument panel is designed for quick access while driving, the system presented in the embodiment is designed for passengers of a fully autonomous vehicle, thereby improving user convenience of the autonomous vehicle.

Description

DEEP LEARNING-BASED AUTONOMOUS VEHICLE AUDITORY SYSTEM FOR THE VISUALLY IMPAIRED AND METHOD THEREOF

The present invention relates to a system and method for auralization of an autonomous driving vehicle, and more specifically, to a deep learning-based auditory system and method for converting various driving information of an autonomous driving vehicle into auditory information for the visually impaired.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not an admission that it is prior art.

Many global companies such as Google, Tesla, and Mercedes-Benz are actively conducting research and development of autonomous vehicles. However, Google's self-driving car crashed into a bus, Tesla's self-driving car crashed into a bicycle, and caused a fire. An accident occurred during the test stage of the autonomous vehicle, and thus far, the driving safety of the autonomous vehicle is not perfect. In particular, since the occupant of a fully autonomous vehicle does not drive directly, it is difficult to recognize the situation before and after a vehicle accident, and it is difficult to recognize detailed state information of the vehicle, so there is a possibility of neglecting vehicle inspection. In an autonomous vehicle, it is necessary to inform the occupants of vehicle status information and driving information in more detail and frequently. The conventional autonomous vehicle converts most driving information, detailed state information, and self-diagnosis data of the vehicle into visual data and provides it to the driver.

However, if all information collected from the vehicle is converted into visual data and transmitted, the complexity of visual information is maximized, which may cause problems in data identification. In addition, if only visual information is provided when a blind person rides a fully autonomous vehicle, the user cannot recognize intuitive and accurate vehicle information, thereby limiting the use of the vehicle by the visually impaired. Since most of the information of the conventional autonomous driving vehicle is converted into visual information and provided, the visually impaired cannot actively use the autonomous driving vehicle. In addition, there is a problem in that visual fatigue increases after driving because general users also need to understand most of the detailed vehicle information as visual information.

1. Korean Patent Publication No. 10-1933150 (2018.12.20) 2. Korean Patent Publication No. 10-1702612 (2017.01.26)

The system and method for auralization of an autonomous vehicle according to an embodiment stores sensor data and self-diagnosis data of the vehicle inside the vehicle, receives a voice command signal from the visually impaired, converts the information into text of the vehicle, and then auralizes it. Make fully autonomous vehicles accessible to the visually impaired.

In addition, by converting vehicle driving information and vehicle status information such as fuel shortage, charging amount notification, and emergency situations into audible signals, users in autonomous vehicles can intuitively recognize vehicle driving information and status information through auditory signals. make it possible

The deep learning-based autonomous driving vehicle auditory system for the visually impaired according to the embodiment collects vehicle status information including fuel and battery remaining amount, and self-diagnosis data of each autonomous driving vehicle operation system including the vehicle internal communication system and driving system. a sensor module for sensing and detecting driving data including RPM, speed, and speed change amount; A data storage management module that collects and stores vehicle status information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and stores a message received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle status ; a first conversion module that converts vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information; and a second conversion module that converts the autonomous driving visualization information generated in the first conversion module into voice information to generate autonomous driving audio information; includes

In the method for generating auditory information of a deep learning-based autonomous driving vehicle auditory system for the visually impaired according to another embodiment, the sensor module includes vehicle state information including fuel and battery remaining amount, and vehicle internal communication system and vehicle including a driving system. A first step of sensing self-diagnosis data and sensing driving data including RPM, speed, and speed change amount; The data storage management module includes: a second step of collecting and storing vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and storing a message received from an external server communicating with the vehicle and the diagnosed vehicle state; The first conversion module includes: a third step of converting vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information; And the second conversion module converts the autonomous driving visualization information generated in the first conversion module into voice information a fourth step of generating autonomous driving audio information; includes

The deep learning-based autonomous driving vehicle auditory system and method for the visually impaired according to the embodiment enable the visually impaired to use the autonomous driving vehicle, and provide the autonomous driving vehicle driving information and state information as an auditory signal to general users. By enabling intuitive recognition, the speed of delivering information to the driver, information recognition speed, and driver reaction speed can be improved. In addition, it is possible to use an autonomous vehicle without visual fatigue and to recognize vehicle state information only with an auditory signal.

It is possible to inform the occupants of various information than the conventional instrument panel without compromising the real-time of the conventional cluster system. It enhances the accessibility of autonomous vehicles for the visually impaired by promptly alerting the visually impaired by visualizing situations that need to be checked visually, such as vehicle status and accident situations. Although the conventional instrument panel is designed for quick access while driving, the system presented in the embodiment is designed for passengers of a fully autonomous vehicle, thereby improving user convenience of an autonomous vehicle.

It should be understood that the effects of the present invention are not limited to the above-described effects, and include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

1 is a view showing a deep learning-based self-driving vehicle auditory system for the visually impaired according to an embodiment;
2A is a diagram showing the configuration of a data storage management module according to an embodiment;
2B is a diagram for explaining a data processing flow in the data storage management module;
Figure 3a is a view showing the data processing configuration of the first conversion module according to the embodiment
Figure 3b is a diagram showing a configuration diagram of HMM (hidden Markov Model) used in the first conversion module according to the embodiment;
Figure 4a is a view showing a second conversion module (TWM, Text to Wave Module400) according to the embodiment
Figure 4b is a view showing a data processing flow of the second conversion module for generating auditory information according to an embodiment;
5 is a diagram illustrating a data processing flow of an autonomous driving vehicle auditory system according to an embodiment;

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 is a diagram illustrating a deep learning-based self-driving vehicle auditory system for the visually impaired according to an embodiment.

Referring to FIG. 1 , the autonomous driving vehicle auditory system according to the embodiment is configured to include a sensor module 100 , a data storage management module 200 , a first conversion module 300 and a second conversion module 400 . can be As used herein, the term 'module' should be construed to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, the software may be machine language, firmware, embedded code, and application software. As another example, the hardware may be a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a sensor, a Micro-Electro-Mechanical System (MEMS), a passive device, or a combination thereof.

The sensor module (Sensor Module, 100) detects vehicle state information and driving data. In the embodiment, the vehicle state information includes vehicle state information that a passenger should recognize for safe driving of the vehicle, such as a battery, a remaining amount of washer fluid, a window, and a door open state. The self-diagnosis data of the vehicle is a hardware and software system installed for driving and driving the vehicle, such as a CAN communication system installed inside the vehicle, a driving system including wheels, brakes, motors, batteries, and accelerator control devices, and a cluster system that displays vehicle information. It is the data that diagnoses the abnormality of

The driving data is data representing driving information of the vehicle, and may include RMP, gear state, moving direction, speed change, and the like.

The data storage and management module (DCMM, Data Collection and Management Module, 200) stores and manages all sensor data, self-diagnosis data, and graphic library collected from the vehicle in the interior space of the vehicle. For example, the data storage management module 200 collects and stores vehicle state information detected by the sensor module 100, self-diagnosis data of the vehicle, and driving data, and a message received from an external server communicating with the vehicle and the diagnosed vehicle save the state

The first conversion module (STM, Speech to Text module, 300) converts vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information. In addition, in the embodiment, when the user inputs a voice signal, the first conversion module 300 converts the user's voice signal into a text signal to be used as a vehicle control signal. In addition, the first conversion module 300 visualizes vehicle data according to user control or automatic control such as diagnostic information and control signals of the vehicle, extracts a visual library of information to be output, and adaptively suits the display of the vehicle. It can be placed to inform the status of the vehicle.

The second conversion module (TWM, Text to wave Module, 400) converts the autonomous driving visualization information generated by the first conversion module 300 into voice information to generate autonomous driving audio information. In an embodiment, the second conversion module 400 extracts dangerous state notification information indicating a dangerous state of the vehicle from among vehicle state information, self-diagnosis data, and driving data, and converts the extracted dangerous state notification information into voice information for autonomous driving Generates auditory information. In an embodiment, the dangerous state notification information may include information about the vehicle's surrounding environment and object data and driving state information sensed to prevent accidents, such as whether a dangerous object exists near the vehicle, whether a speed limit is maintained, and a traffic situation. In addition, the second conversion module 400 receives the user's voice data including the visually impaired to output the vehicle's state information as a voice or an auditory signal.

2A is a diagram showing the configuration of a data storage management module according to an embodiment, and FIG. 2B is a diagram for explaining a data processing flow in the data storage management module.

Referring to FIG. 2A , the data storage management module 200 according to the embodiment may include a sensor data storage unit 210 , a data visualization library 230 , and a wave format library 250 .

The storage unit 210 stores vehicle state information collected from the vehicle, vehicle self-diagnosis data, driving data, and message information transmitted from other communication objects to the vehicle. The data visualization library 230 stores graphics, images, and texts for visualizing the stored information. The wave format library 230 stores a wave format that auralizes vehicle state information, vehicle self-diagnosis data, driving data, and text information collected from the vehicle.

The function of the data storage management module 200 according to the embodiment will be described in more detail with reference to FIG. 2B . The data storage management module 200 according to the embodiment may further include a receiver for receiving sensor messages inside the vehicle, and the sensor data storage unit 210 stores the received messages. The receiver receives a message by CAN communication, and receives a MOST message, a FLEXRAY message, a LIN message, and the like. In an embodiment, each message received by the receiver is stored in a storage unit in a table format. The receiver may convert data differently according to the type of the sensor data message, classify it in the sensor data storage unit, and store it. In the embodiment, when the data storage management module receives the data to be auralized or the data to be visualized, the first conversion module 300 ) or it can be transmitted to the second conversion module 400 .

In addition, the data storage management module 200 according to the embodiment stores the state of the vehicle diagnosed by using the self-diagnosis system of the vehicle, and transmits the stored vehicle state information to the first conversion module or the second conversion module to Make sure that the self-diagnosis results are visualized or auralized.

Figure 3a is a diagram showing the data processing configuration of the first transformation module according to the embodiment, Figure 3b is a diagram showing the configuration diagram of the HMM (hidden Markov Model) used in the first transformation module according to the embodiment.

Referring to FIG. 3A , the first conversion module 300 according to the embodiment may include a learning unit 310 , an operation unit 330 , and a conversion unit 350 . In an embodiment, the first conversion module 300 visualizes various information and data or converts them into text.

The learning unit 310 learns the vehicle state displayed according to the vehicle state information sensed by the vehicle, the vehicle self-diagnosis data, and the driving data through deep learning. The learning unit 310 according to the embodiment learns the HMM (Hidden Markov Model) independently of the cloud using the point-to-point learning method and NVIDIA DRIVER PX2, receives the voice from the vehicle's microphone, and outputs a result corresponding to the voice signal make it possible To this end, a solid state drive (SSD) inside the vehicle transmits learning data to the learning unit 310 .

The calculating unit 330 calculates vehicle state information, vehicle self-diagnosis data, and driving data sensed by the vehicle, and calculates a probability for each vehicle state according to the calculation result. For example, since the operation unit 330 recognizes data and voice to be visualized in units of short words, it is possible to calculate a result value in a state in which an input and an output are given using the Markov chain. In the embodiment, since the result should be calculated only from the user's input value, the HMM assumes that the result is hidden and performs a probability operation using the Markov chain. In general, the HMM appears in the form of a graph as shown in FIG. 3B. Table 1 describes the components and meanings of the HMM.

Components of HMM collective name Set contents meaning
Q
{q ₁ , q ₂ , q _{3 … .} q _N }

set of hidden states
Y

{y _{1 ,} y ₂ … … … y _M }
A collection of observations that can occur in a hidden state

{

,

… … ..

| R ^N }
A set of initial probabilities p(qi) representing the probability that the initial state is qi
T

{T ₁₂ , T ₂₁ … … … T _NM, T _MN | R ^NxN
A set of transition probabilities p(qiqj) representing the probability of moving from qi to qj
E

{E ₁₁ , E ₁₂ … .. E _NM | R ^NxM }
emission probability representing the probability that yj occurs in qi

{

, T ,E}
HMM parameters

가 주어졌을 때 다이나믹 프로그래밍(dynamic programming) 기반의 포워드 알고리즘(forward algorithm)을 사용하여 주어진 관측 값이 나타난 확률을 계산하고 알고리즘(예컨대, Viterbi algorithm)을 이용하여 그 확률이 가장 높은 상태의 나열을 획득한다. 상태의 나열이 계산되면 학습부는 이용한 알고리즘(Viterbi algorithm)을 산출하고 바움 웰치 알고리즘(Baum-Welch algorithm)과 학습데이터를 이용하여 HMM의

를 학습 시킨다. 즉, 연산부는 초기

를 지정하여 관측 값들이 나타날 확률과 상태나열을 계산하고 학습부는 산출된 상태나열 값을 기반으로 HMM을 학습한다. In the embodiment, the operation unit 330 is the HMM

Given a dynamic programming-based forward algorithm, calculate the probability that a given observation value appears, and use the algorithm (e.g., Viterbi algorithm) to obtain a list of states with the highest probability do. When the list of states is calculated, the learning unit calculates the algorithm used (Viterbi algorithm), and uses the Baum-Welch algorithm and learning data to

to learn That is, the arithmetic unit is initially

By designating , it calculates the probability that the observed values appear and the state sequence, and the learning unit learns the HMM based on the calculated state sequence value.

The conversion unit 350 converts the vehicle state representing the highest probability among the calculated probabilities into text information.

Figure 4a is a view showing a second conversion module (TWM, Text to Wave Module 400) according to the embodiment, Figure 4b is a view showing the data processing flow of the second conversion module for generating auditory information according to the embodiment.

Referring to FIG. 4A , the second conversion module 400 according to the embodiment may include a data input unit 410 , an extraction unit 430 , a determination unit 450 , and a conversion unit 470 . The data input unit 410 receives vehicle state information, vehicle self-diagnosis data, driving data, and converted text. The extraction unit 430 extracts important data and text features to be converted into a waveform from among the input data. The determination unit 450 determines the importance and complexity of each part from the extracted important data. The conversion unit 470 converts the input text, vehicle state information, vehicle self-diagnosis data, and driving data into an auditory signal according to the determination result.

Referring to Figure 4b, the second conversion module (TWM, Text to Wave Module 400) according to the embodiment is the audio input data and the first conversion module (STM, Speech to Text module 300) of data input by voice do. In an embodiment, the second conversion module auralizes text by performing TTS (Text to Speech) using a specific conversion algorithm (eg, Google Tacotron 2). Since Tacotron 2 used in the second conversion module operates based on the RNN encoder-decoder model, in the embodiment, Tacotron 2 is learned using a speech database, and the sensor data state of the vehicle is accurately output as voice.

The second transformation module (TWM, 400) according to the embodiment receives a sentence, divides it into word units, and inputs the divided units to an encoder. When a vector of word units is input to the encoder, the pre-net of the transformation algorithm one hot encodes the word. At this time, the performed encoding (one hot encoding) is a process of converting a text vector into an array of 0 and 1 for easy recognition by the encoder. When the pre-net encodes the text vector (one hot encoding), the second transformation module (TWM, 400) transmits the encoding result to the CBHG network of the transformation algorithm (eg, Tacontron). CBHG is a neural network model consisting of a convolution bank, a highway net, and a bidirectional GRU (Gated Recurrent Unit). The convolution bank extracts text features, the highway net creates a deep neural network model, and the bidirectional gate circulation unit creates text embeddings by considering the pre-netted vector before and after information. The generated text embedding is sent to the Attention Model. The attention model uses the attention RNN to determine which parts are more complex and which parts are more important in the transmitted text embedding. This week's model allows the Tacotron to be trained end-to-end, transforming the untrained text vector into speech.

Hereinafter, a method for making an autonomous vehicle auditory will be described in turn. Since the operation (function) of the autonomous driving vehicle auditory method according to the embodiment is essentially the same as the function on the autonomous driving vehicle auditory system, the description overlapping with FIGS. 1 to 4B will be omitted.

5 is a diagram illustrating a data processing flow of an autonomous driving vehicle auditory system according to an embodiment.

In step S10, the sensor module senses vehicle status information including fuel and battery remaining amount and self-diagnosis data of the vehicle including the vehicle internal communication system and driving system, and detects driving data including RPM, speed, and speed change amount. .

In step S20, the data storage management module collects and stores vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and stores a message received from an external server communicating with the vehicle and the diagnosed vehicle state. In step S20, vehicle state information collected from the vehicle, vehicle self-diagnosis data, driving data, and message information transmitted to the vehicle are stored, and graphics, images, and texts for visualizing the stored information are stored. In addition, it stores the vehicle state information, vehicle self-diagnosis data, driving data, and a wave format that auralizes text information collected from the vehicle.

In step S30, data to be output as an audio signal or a visualization signal among self-diagnosis data and driving data is extracted from the data storage management module.

In step S40, the first conversion module converts vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information. In step S40, the vehicle state information, vehicle self-diagnosis data, and driving data sensed by the vehicle are learned through deep learning, and the vehicle state information sensed by the vehicle, vehicle self-diagnosis data, and driving data are calculated and calculated. According to the result, a probability for each vehicle state is calculated. Thereafter, the vehicle state representing the highest probability among the calculated probabilities is converted into text information.

In step S50, the second conversion module 400 converts the autonomous driving visualization information generated by the first conversion module 300 into voice information to generate autonomous driving audio information.

In the embodiment, in step S50, the second conversion module 400 extracts dangerous state recognition information indicating the dangerous state of the vehicle among the vehicle state information, self-diagnosis data, and driving data, and converts the extracted dangerous state recognition information into voice information. to generate autonomous driving audio information.

In addition, in step S50, vehicle state information, vehicle self-diagnosis data, driving data, and converted text are input to generate audio information of the autonomous vehicle, and important data and text features to be converted into a waveform are extracted from the input data. do. Thereafter, the importance and complexity of each part are determined from the extracted important data, and the input text, vehicle state information, vehicle self-diagnosis data, and driving data are converted into an auditory signal according to the determination result.

In step S60, the auditory information is output from the autonomous vehicle.

The deep learning-based autonomous driving vehicle auditory system and method for the visually impaired according to the embodiment enable the visually impaired to use the autonomous driving vehicle, and provide the autonomous driving vehicle driving information and state information as an auditory signal to general users. By enabling intuitive recognition, the speed of delivering information to the driver, information recognition speed, and driver reaction speed can be improved. In addition, it is possible to use an autonomous vehicle without visual fatigue and to recognize vehicle state information only with an auditory signal.

In addition, it is possible to inform the passenger of more information than the conventional instrument panel without impairing the real-time of the conventional cluster system. It enhances the accessibility of autonomous vehicles for the visually impaired by promptly alerting the visually impaired by visualizing situations that need to be checked visually, such as vehicle status and accident situations. Although the conventional instrument panel is designed for quick access while driving, the system presented in the embodiment is designed for passengers of a fully autonomous vehicle, thereby improving user convenience of an autonomous vehicle.

The disclosed content is merely an example, and can be variously changed and implemented by those skilled in the art without departing from the gist of the claims claimed in the claims, so the protection scope of the disclosed content is limited to the specific It is not limited to an Example.

Claims (10)

In a deep learning-based self-driving vehicle auditory system for the visually impaired,
A sensor that senses vehicle status information including fuel and battery remaining amount and self-diagnostic data of each autonomous vehicle operating system including the vehicle internal communication system and driving system, and detects driving data including RPM, speed, and speed change amount module;
Data storage management that collects and stores vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and stores a message received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle state module;
a first conversion module that converts vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information;
a second conversion module that converts the autonomous driving visualization information generated by the first conversion module into voice information to generate autonomous driving audio information; including,
the first conversion module; silver
a learning unit for learning through deep learning the vehicle state displayed according to the vehicle state information sensed by the vehicle, the vehicle self-diagnosis data, and the driving data;
a calculation unit for calculating vehicle state information, vehicle self-diagnosis data, and driving data sensed by the vehicle and calculating a probability for each vehicle state according to the calculation result;
a conversion unit converting a vehicle state representing the highest probability among the calculated probabilities into text information; includes
the second conversion module; silver
a data input unit for receiving vehicle state information, vehicle self-diagnosis data, driving data, and converted text;
an extraction unit for extracting important data and text features to be converted into a waveform from among the input data;
a determination unit for determining the importance and complexity of each part from the extracted important data; and
a conversion unit that converts the input text, vehicle state information, vehicle self-diagnosis data, and driving data into an auditory signal according to the determination result; A self-driving vehicle auditory system comprising a.
According to claim 1, wherein the second conversion module; silver
Autonomy, characterized in that by extracting dangerous state notification information indicating the dangerous state of the vehicle from among vehicle state information, self-diagnosis data, and driving data, and converting the extracted dangerous state notification information into voice information to generate autonomous driving audio information Driving Vehicle Hearing System.
delete According to claim 1, wherein the data storage management module; silver
a storage unit for storing vehicle state information collected from the vehicle, vehicle self-diagnosis data, driving data, and message information transmitted to the vehicle;
a data visualization library for storing graphics, images, and texts for visualizing the stored information;
a wave format library for storing a wave format for auralizing vehicle state information, vehicle self-diagnosis data, driving data, and text information collected from the vehicle; A self-driving vehicle auditory system comprising a.
delete In a method for generating auditory information of a deep learning-based autonomous driving vehicle auditory system for the visually impaired,
The sensor module senses vehicle state information including fuel and battery remaining amount and self-diagnosis data of the vehicle including the vehicle internal communication system and the driving system, and the first step of detecting driving data including RPM, speed, and speed change amount ;
The data storage management module collects and stores the vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and stores a message received from an external server communicating with the vehicle and the diagnosed vehicle state. ;
The first conversion module includes: a third step of converting vehicle state information, vehicle self-diagnosis data, and driving data into text to generate autonomous driving visualization information; and
The second conversion module converts the autonomous driving visualization information generated by the first conversion module into voice information to generate autonomous driving audio information; including,
the third step; is
learning the vehicle state displayed according to the vehicle state information sensed by the vehicle, the vehicle self-diagnosis data, and the driving data through deep learning;
calculating vehicle state information, vehicle self-diagnosis data, and driving data sensed by the vehicle, and calculating a probability for each vehicle state according to the calculation result;
converting a vehicle state representing the highest probability among the calculated probabilities into text information; including,
the fourth step; is
receiving vehicle state information, vehicle self-diagnosis data, driving data, and converted text;
extracting important data and text features to be converted into a waveform from among the input data;
determining the importance and complexity of each part from the extracted important data; and
and converting the input text, vehicle state information, vehicle self-diagnosis data, and driving data into an auditory signal according to the determination result.
7. The method of claim 6, further comprising: the third step; is
extracting dangerous state notification information indicating a dangerous state of the vehicle from among vehicle state information, self-diagnosis data, and driving data; and
generating autonomous driving audio information by converting the extracted dangerous state notification information into voice information; A method for generating auditory information, comprising:
delete The method of claim 6, wherein the second step;
storing the vehicle state information collected from the vehicle, vehicle self-diagnosis data, driving data, and message information transmitted to the vehicle;
storing graphics, images, and texts for visualizing the stored information;
and storing the vehicle state information, vehicle self-diagnosis data, driving data, and text information collected from the vehicle in a wave format for auditory information.





delete

Images