KR102270011B1 - Deep learning-based autonomous vehicle visualize system for the visually impaired and method thereof - Google Patents

Abstract

Initiating a deep learning-based autonomous vehicle visualization system for the visually impaired. The deep learning-based autonomous driving vehicle visualization system for the blind according to the embodiment converts the information collected from the vehicle into a recognizable visualization object according to the visual characteristics of the low-vision blind person and provides it, so that even the low-vision visually impaired can use the autonomous driving vehicle. Make it available without hesitation. In addition, by classifying and outputting the vehicle driving information into visualization data, auditory data or audiovisual data according to the type of information, the output information collected from the vehicle can be recognized more intuitively and easily by the vehicle occupant, and the reaction speed of the occupant Accordingly, the optimal output form of vehicle state information can be identified, and information output and transmission efficiency can be improved.

Description

{DEEP LEARNING-BASED AUTONOMOUS VEHICLE VISUALIZE SYSTEM FOR THE VISUALLY IMPAIRED AND METHOD THEREOF}

The present invention relates to a visualization system and method for an autonomous vehicle, and more particularly, to a deep learning-based visualization system and method for converting various driving information of an autonomous vehicle into visual information recognizable by a visually impaired person.

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and inclusion in this section is not an admission that it is prior art.

A self-driving car refers to a car that detects the situation with various sensors of the vehicle, such as precise maps and GPS, and finds a destination by itself, without the driver operating the steering wheel, accelerator pedal, or brake. In a strict sense, it is different from driverless cars that move without a human being, but in reality, they are used interchangeably. Dozens of technologies are needed to make autonomous vehicles a reality. For example, HDA technology that automatically maintains the distance between vehicles, Lane Departure Warning System (LDWS), Lane Keeping Assist System (LKAS), Blind Spot Warning System (BSD), Advanced Smart Cruise Control (ASCC), Automatic Emergency Braking System (AEB) is an essential system for autonomous vehicles. The technological stage of autonomous vehicles is divided into five stages: from stage 1, which supports driving assistance functions such as automatic braking and automatic speed control, to fully autonomous driving and unmanned driving vehicles that move without a human being.

Since the occupant of a fully autonomous vehicle does not drive directly, it is difficult to recognize the situation before and after a vehicle accident, and it is difficult to recognize detailed state information of the vehicle, so there is a possibility of neglecting the vehicle inspection. In an autonomous vehicle, it is necessary to inform the occupants of vehicle status information and driving information in more detail and frequently. A conventional autonomous vehicle converts driving information, detailed state information, and self-diagnosis data of the vehicle into visual data and provides it to the driver.

However, most of the visualization information of the conventional autonomous vehicle is difficult to check easily for the visually impaired or a user with low vision. This can act as a decisive factor preventing the blind or the elderly with low vision from actively using autonomous vehicles.

1. Korean Patent Publication No. 10-2018-0066618 (2018.06.19) 2. Korean Patent Publication No. 10-2019-0117419 (2019.10.16)

A visualization system and method for an autonomous vehicle according to an embodiment stores sensor data and self-diagnosis data of the vehicle inside the vehicle, and visualizes driving information and vehicle state information of the autonomous vehicle to be recognized by a blind or low-vision user By outputting it as an object, even the visually impaired can use the fully autonomous vehicle.

In addition, data to be converted into audible information and data to be converted into visualization information among the vehicle driving information or status information are divided and output according to the vehicle information, the driving situation and the body information of the occupant, so that users with visual or hearing impairments Make autonomous vehicles available.

The deep learning-based autonomous driving vehicle visualization system for the visually impaired according to the embodiment grasps the visual characteristics of the low-vision blind person riding in the autonomous driving vehicle, and adjusts the illuminance, color, size, and contrast of the autonomous driving vehicle according to the vision characteristic information. a setting module for setting detailed information of a visualization information object including; A sensor that senses vehicle status information including fuel and battery remaining amount, self-diagnosis data of each autonomous vehicle operation system including vehicle internal communication system and driving system, and detects driving data including RPM, speed, and speed change amount module; The vehicle status information detected by the sensor module, self-diagnosis data of the vehicle, and driving data are collected and stored, and a message received from a communication object including an external server that communicates with the vehicle, the diagnosed vehicle status, and the format of the information object to be converted a data storage management module for storing; and an information object generating module for generating autonomous driving visualization information by converting vehicle state information, vehicle self-diagnosis data, and driving data into a visualization information object recognizable by the visually impaired. includes

In the information generation method of the deep learning-based autonomous driving vehicle visualization system for the visually impaired according to another embodiment, the setting module grasps the vision characteristics of the low-vision blind person riding in the autonomous driving vehicle, and according to the vision characteristic information, A first step of setting detailed information of the visualization information object including illumination, color, size, and contrast; The sensor module senses vehicle status information including fuel and battery remaining amount and self-diagnosis data of each autonomous vehicle operation system including the vehicle internal communication system and driving system, and receives driving data including RPM, speed, and speed change. a second step of sensing; The data storage management module collects and stores vehicle status information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and messages received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle status and conversion a third step of storing the format of the information object to be used; and a fourth step of generating the autonomous driving visualization information by converting the information object generation module into a visualization information object recognizable by the low-vision blind person from vehicle state information, vehicle self-diagnosis data, and driving data; includes

The deep learning-based autonomous vehicle visualization system for the blind as described above converts the information collected from the vehicle into a recognizable visualization object according to the visual characteristics of the low-vision blind person and provides it, so that even the low-vision visually impaired can use the autonomous driving vehicle. Make it available without hesitation. In addition, by classifying and outputting the vehicle driving information into visualization data, auditory data or audiovisual data according to the type of information, the output information collected from the vehicle can be recognized more intuitively and easily by the vehicle occupant, and the reaction speed of the occupant Accordingly, the optimal output form of vehicle state information can be identified, and information output and transmission efficiency can be improved.

In addition, since it is possible to provide both a visualized information object and an auralized information object in the embodiment, a user with weak eyesight or a weak hearing sets the form of output information that he or she mainly uses and the proportion of output in the set information form This will allow the elderly and the weak with low hearing or vision to actively use autonomous vehicles.

It should be understood that the effects of the present invention are not limited to the above-described effects, and include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

1 is a diagram showing a data processing configuration of a deep learning-based autonomous vehicle visualization system for the visually impaired according to an embodiment;
2 is an example of a visualization object of a deep learning-based autonomous driving vehicle visualization system for the visually impaired according to the embodiment;
3 is a data processing configuration of the determination module 400 according to the embodiment.
4 is a data processing block of the information object generating module 500 according to the embodiment.
5 is a diagram illustrating a data processing process of a deep learning-based autonomous driving vehicle visualization system for the visually impaired according to an embodiment;

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 is a diagram illustrating a data processing configuration of a deep learning-based autonomous driving vehicle visualization system for the visually impaired according to an embodiment.

Referring to FIG. 1 , a deep learning-based autonomous driving vehicle visualization system for the visually impaired according to an embodiment includes a setting module 100 , a sensor module 200 , a storage management module 300 , a determination module 400 , and an information object. It may be configured to include a generating module 500 . As used herein, the term 'module' should be construed to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, the software may be machine language, firmware, embedded code, and application software. As another example, the hardware may be a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a sensor, a Micro-Electro-Mechanical System (MEMS), a passive device, or a combination thereof.

The setting module 100 identifies the visual characteristics of the visually impaired, including the low-vision visually impaired, riding in the autonomous driving vehicle, and according to the vision information, the illuminance and color of the autonomous vehicle, the size, the contrast, and the details of the information object including Set the information.

Blind people are divided into blind and low-vision blind people, and low-vision blind people account for about 80% of the total blind people. Because low-vision blind people are not blind, they can recognize the rough outline of objects and the presence or absence of objects through sight. However, low-vision blind people have different types of recognizable visualization objects according to individual visual characteristics. Specifically, color blindness is also included in the low-vision visual impairment, and depending on the characteristics of the low-vision impairment, there are cases in which the visualization recognition rate of an object varies depending on the illuminance. Specifically, some visually impaired people can recognize a visualization object only when the illuminance is high, and there are also visually impaired people who can recognize a visualization object only when the illuminance is low.

As described above, the setting module 100 identifies the visual characteristics of each of the visually impaired riding in the autonomous driving vehicle, and sets the type of visualization object that the visually impaired can recognize according to the visual characteristics. For example, as shown in FIG. 2 showing an example of a visualization object according to the embodiment, when the visually impaired can recognize all objects only in black and white, in the embodiment, the contrast ratio of the visualization object generated by the autonomous vehicle is set to be very high Alternatively, a black-and-white processed visualization information object is created and provided to the passenger. In addition, the setting module 100 sets the size, color, format, etc. of the output visualization information object and the auditory information object based on the biometric information including the visual characteristics of the passenger.

The sensor module 200 senses vehicle state information including fuel and battery remaining amount, and self-diagnosis data of each autonomous vehicle operating system including the vehicle internal communication system and driving system, and includes RPM, speed, and speed change. Detect driving data.

The storage management module 300 collects and stores vehicle state information detected by the sensor module 200, self-diagnosis data and driving data of the vehicle, and a message received from a communication object including an external server that communicates with the vehicle, the vehicle It stores the state and format of the information object to be converted. The visualization information object according to the embodiment may be generated in various formats as a visualization object that can be grasped according to the visual characteristics of the visually impaired. For example, a first information object composed of black and white and border lines perceptible by the visually impaired, a second information object composed only of colors perceptible by the visually impaired; and a third information object configured to configure light intensity and brightness differently for each area. and the like. In addition, information objects and three-dimensional information objects that are visualized only by differences in brightness between white, gray, and black may be included.

The determination module 400 separates data to be converted into a visualization information object among vehicle state information, self-diagnosis data, and driving data, data to be converted into an auditory information object, and data to be converted into an audiovisual information object. For example, the determination module 400 determines an optimal information object format according to the type of collected information in order to easily recognize driving information or vehicle state information to an autonomous vehicle occupant.

Specifically, the determination module 400 may convert the dangerous state notification information indicating the dangerous state of the vehicle among the vehicle state information, self-diagnosis data, and driving data into audio-visual data. In addition, it is easy for the user to recognize the vehicle status information, such as door open and turn signal on, into audible data, and converting object information existing around the vehicle into visualization data allows the user to recognize the information. Since it is more helpful, the determination module 400 may determine the conversion format of the collected vehicle information based on the correlation between the type of information collected from the vehicle and the ease of recognition of the output format.

The information object generation module 500 generates autonomous driving visualization information by converting vehicle state information, vehicle self-diagnosis data, and driving data into information objects recognizable by the visually impaired with low vision. In addition, in an embodiment, the information object generation module 500 generates an audiovisual information object and an audio information object corresponding thereto when it is determined that vehicle information including driving data is converted into an audiovisual information object or an audiovisual information object. print out

3 is a diagram showing the data processing configuration of the determination module 400 according to the embodiment.

Referring to FIG. 3 , the determination module 400 according to the embodiment may include a driving situation analysis unit 410 , a classification unit 430 , and a visualization information object format matching unit 450 .

The driving situation analysis unit 410 analyzes the driving situation according to the location information of the autonomous vehicle and the surrounding object information. For example, the driving situation analysis unit 410 may determine the degree of congestion on the road on which the vehicle is congested and the density and number of obstacles around the vehicle.

The classification unit 450 identifies information to be output among the collected vehicle state information and driving information, and classifies data to be visualized, data to be auralized, and data to be audiovised among the information to be output. For example, the classification unit 450 may classify a dangerous situation message such as when the fuel and battery are below a certain level or when a moving object suddenly appears around the vehicle as data to be audio-visualized, and the vehicle such as door open, window open, etc. State information can be classified as information to be auralized.

In an embodiment, the classification unit 430 feeds back the optimal output form of information collected from the vehicle by identifying the user reaction speed and recognition speed when visualized or auralized information is output after classifying the vehicle information output format through deep learning. can do. For example, if output information about a moving object suddenly appearing around the vehicle is provided as an audiovisual information object, if it is calculated that the vehicle occupant's reaction speed or control speed is faster, the output information about the object around the vehicle is converted into audiovisual information You can give feedback for output.

The visualization information object format matching unit 450 extracts information to be visualized according to the analyzed driving situation and vehicle state information, and identifies a visualization format into which the extracted information object is to be converted. For example, the visualization information object format matching unit 430 matches the information to be output in the format of the visualization information object in consideration of the visual characteristics information of the blind person riding in the vehicle. Specifically, when a visually impaired person riding in a vehicle is red-green color blind, a black-and-white visualization information object format is matched, and when a specific visually impaired person can recognize only a three-dimensional object, the three-dimensional information object format is added to the information to be output as a visualization. match

4 shows a data processing block of the information object generating module 500 according to an embodiment.

Referring to FIG. 4 , the visualization information object generation unit 510 according to the embodiment creates a visualization information object according to a format of a visualization information object matched according to the visual characteristics of a visually impaired person riding in a vehicle. For example, the visualization information object generating unit 510 converts the information collected from the vehicle into a visualization object recognizable by a low-vision blind person riding in the vehicle and outputs it. In an embodiment, a visualization information object may be created in various formats, such as a black-and-white information object, a three-dimensional information object, and an information object composed of only a specific color.

The audiovisual information object generation unit 520 converts the information collected from the vehicle into an audiovisual information object and outputs it. In an embodiment, the audiovisual information object outputs the visualized output information and the auditory output information at the same time, or outputs the two information with a time difference so that the user in the vehicle can more intuitively recognize the information collected from the vehicle. . In an embodiment, the audiovisual information object generation unit 250 adjusts the output weight of the visualization information to a high level when the hearing impaired rides, and adjusts the output weight of the audiovisual information to a high level when the passenger, who is mainly dependent on hearing, rides. You can create an audiovisual information object and output it. In addition, the optimal output timing of the visualization information object and the auditory information object may be calculated differently according to vehicle information and information may be output.

Hereinafter, the method will be described in turn. The operation (function) of the deep learning-based autonomous driving vehicle visualization system for the visually impaired according to the embodiment is essentially the same as the function of the deep learning-based autonomous driving vehicle visualization method for the visually impaired, so the description overlaps with FIGS. 1 to 4 should be omitted.

5 is a diagram illustrating a data processing process of a deep learning-based autonomous driving vehicle visualization system for the visually impaired according to an embodiment.

Referring to FIG. 5 , in step S510, the setting module identifies the visual characteristics of the visually impaired, including the low-vision blind, who ride the autonomous driving vehicle, and adjusts the illuminance, color, size, contrast, and brightness of the autonomous driving vehicle according to the vision information. Set detailed information of the included information object.

In step S520, the sensor module senses vehicle status information including fuel and battery remaining amount, and self-diagnosis data of each autonomous driving vehicle operation system including the vehicle internal communication system and driving system, and includes RPM, speed, and speed change. Detect driving data.

In step S530, the data storage management module collects and stores vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and a message received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle It stores the state and format of the information object to be converted.

In step S540, the autonomous driving visualization information is generated by converting the vehicle state information, vehicle self-diagnosis data, and driving data into an information object recognizable by the visually impaired with low vision, and in step S550, the generated object is output.

The deep learning-based autonomous vehicle visualization system for the blind as described above converts the information collected from the vehicle into a recognizable visualization object according to the visual characteristics of the low-vision blind person and provides it, so that even the low-vision visually impaired can use the autonomous driving vehicle. Make it available without hesitation. In addition, by classifying and outputting the vehicle driving information into visualization data, auditory data or audiovisual data according to the type of information, the output information is more intuitively and easily recognized by the vehicle occupant, and the vehicle state according to the occupant's reaction speed By understanding the optimal output form of information, information output efficiency can be improved.

In addition, since it is possible to provide both a visualized information object and an auralized information object in the embodiment, a user with weak eyesight or a weak hearing sets the form of output information that he or she mainly uses and the proportion of output in the set information form This will allow the elderly and the weak with low hearing or vision to actively use autonomous vehicles.

The disclosed content is merely an example, and can be variously changed and implemented by those of ordinary skill in the art without departing from the gist of the claims claimed in the claims, so the protection scope of the disclosed content is limited to the specific It is not limited to an Example.

Claims (10)

In a deep learning-based autonomous vehicle visualization system for the visually impaired,
a setting module for identifying the visual characteristics of the low-vision blind person riding in the autonomous driving vehicle, and setting detailed information of the visualization information object including illuminance, color, size, and contrast of the autonomous driving vehicle according to the visual characteristic information;
A sensor that senses vehicle status information including fuel and battery remaining amount, self-diagnosis data of each autonomous vehicle operation system including vehicle internal communication system and driving system, and detects driving data including RPM, speed, and speed change amount module;
The vehicle status information detected by the sensor module, self-diagnosis data of the vehicle, and driving data are collected and stored, and messages received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle status and information object to be converted. a data storage management module for storing the format; and
an information object generating module that converts vehicle state information, vehicle self-diagnosis data, and driving data into a visualization information object recognizable by a low-vision blind person to generate autonomous driving visualization information; including,
The deep learning-based autonomous vehicle visualization system for the visually impaired is
Further comprising a determination module for separating data to be converted into a visualization information object from among vehicle state information, self-diagnosis data, and driving data, data to be converted into an auditory information object, and data to be converted into an audiovisual information object,
The determination module;
Deep learning-based autonomous driving for the visually impaired, characterized in that the optimal output information object format of the information collected from the vehicle is set according to the output timing of the visual information object, the auditory information object, and the audiovisual information object and the reaction speed of the occupant Vehicle visualization system.
According to claim 1, wherein the visualization information object
As a visualization object that can be grasped according to the visual characteristics of the visually impaired, a first information object consisting of black and white and a boundary line;
a second information object composed only of colors perceptible to the visually impaired riding in the vehicle; and
a third information object configured to differently configure light intensity and brightness for each area; Deep learning-based autonomous vehicle visualization system for the visually impaired, characterized in that it comprises a.
delete The method of claim 1, further comprising: the determination module; silver
A deep learning-based autonomous vehicle visualization system, characterized in that the dangerous state notification information indicating the dangerous state of the vehicle among the vehicle state information, self-diagnosis data, and driving data is converted into audiovisual data.
The method of claim 4, wherein the audiovisual data is
Among the vehicle status information, self-diagnosis data, and driving data, there is a visualization information object that can recognize dangerous state notification information indicating the dangerous state of the vehicle, and an audible information object that notifies the dangerous state notification information by sound. A deep learning-based autonomous vehicle visualization system, characterized in that the visualization information object and the audio information object included in the audiovisual data are output with a time difference or output simultaneously.
In an information generation method of a deep learning-based autonomous vehicle visualization system for the visually impaired,
The setting module determines the visual characteristics of the low-vision blind person riding in the autonomous driving vehicle, and sets detailed information of the visualization information object including the illuminance, color, size, and contrast of the autonomous driving vehicle according to the visual characteristic information. step;
The sensor module senses vehicle status information including fuel and battery remaining amount and self-diagnosis data of each autonomous vehicle operation system including the vehicle internal communication system and driving system, and receives driving data including RPM, speed, and speed change. a second step of sensing;
The data storage management module collects and stores vehicle state information detected by the sensor module, self-diagnosis data of the vehicle, and driving data, and a message received from a communication object including an external server that communicates with the vehicle and the diagnosed vehicle state and a third step of storing the format of the information object to be converted; and
The information object generating module includes: a fourth step of converting vehicle state information, vehicle self-diagnosis data, and driving data into a visualization information object recognizable by a low-vision blind person to generate autonomous driving visualization information; including,
the fourth step; is
4-1 step of separating the data to be converted into a visualization information object from among the vehicle state information, self-diagnosis data, and driving data in the determination module, data to be converted into an auditory information object, and data to be converted into an audiovisual information object,
the step 4-1; is
An information generating method, characterized in that the optimal output information object format of the information collected from the vehicle is set according to the output timing of the visual information object, the auditory information object, and the audiovisual information object and the reaction speed of the occupant.
delete delete The method of claim 6, wherein the step 4-1;
An information generating method, characterized in that the ratio of the visualization data and the auditory data included in the audiovisual output data is adjusted according to the visual characteristics and the hearing characteristics of the autonomous vehicle occupant.
The method of claim 6, wherein the step 4-1; is
When outputting the generated audiovisual information object, the information generating method, characterized in that for calculating the output time of the visual information object and the output time of the auditory information object according to the type and driving information of the collected vehicle information.

Images