KR20230115353A - Autonomous Driving Multimedia System - Google Patents

Abstract

An object of the present invention is to provide a self-driving multimedia system that provides a new user experience by loading a virtual environment when multimedia contents are loaded while the self-driving vehicle is driving.
Accordingly, an autonomous driving multimedia system according to an aspect of the present invention includes a sensing unit for detecting surrounding objects while driving and an output unit for outputting multimedia contents, and the output unit includes the detected object in the multimedia contents and outputs the multimedia contents. do.

Description

Autonomous Driving Multimedia System}

The present invention relates to a multimedia system, and more particularly, to an autonomous driving multimedia system combined with driving of an autonomous vehicle.

Vehicles may be classified into internal combustion engine vehicles, external combustion engine vehicles, gas turbine vehicles, electric vehicles, and the like, depending on the type of prime mover used.

Autonomous vehicles are vehicles that can drive automatically without human intervention. Driverless cars drive autonomously by recognizing the surrounding environment with radar, LIDAR (light detection and ranging), GPS, and camera and designating a destination. Unmanned vehicles that are already in practical use include unmanned vehicles operated by the Israeli military that patrol preset routes and unmanned operation systems such as dump trucks that are operated at overseas mines or construction sites.

The first core technologies of these self-driving vehicles are the unmanned car system and the actual system. It is a technology that builds an unmanned car system in reality as well as simulation in the laboratory. It implements driving devices such as accelerators, decelerators, and steering devices to suit unmanned operation, and enables control using computers, software, and hardware installed in unmanned cars. do.

The second key technology is to receive and process visual information using vision and sensors. It is the basis of autonomous driving for unmanned operation, and it is a technology that accepts image information and extracts necessary information from the image. This uses not only a CCD (charge-coupled device) camera, but also sensors such as ultrasonic sensors and range filters to fuse information necessary for distance and driving, allowing obstacle avoidance and unexpected situations to be dealt with through analysis and processing.

The third core technology is the integrated control system and operation monitoring fault diagnosis system technology. This technology establishes an operation monitoring system that monitors vehicle operation and gives appropriate commands according to frequently changing situations, analyzes various situations generated by individual processors and sensors, diagnoses system failures, and provides appropriate information to the operator. or to perform the function of notifying an alarm.

The fourth key technology is intelligent control and intelligent driving devices. This technology generates control commands based on mathematical analysis using actual vehicle models as an unmanned driving technique. Intelligent forward control is a system based on radar guide technology that maintains the distance from the vehicle in front or obstacles by adjusting the speed on its own without the driver manipulating the pedals. When the driver inputs the distance to the vehicle in front, the long-range radar attached to the front of the vehicle detects the location of the vehicle in front, maintains a constant speed, decelerates or accelerates, and comes to a complete stop if necessary, which is useful in poor visibility weather.

The fifth applied technology is the lane departure prevention system. This is a technology in which a camera installed inside detects the lane and informs the driver of an unintended deviation situation.

The sixth applied technology is a parking assist system. When the driver searches for the assist button, puts the reverse gear in and presses the brake pedal, the car adjusts the steering to assist in parallel parking in reverse. It automatically guides the vehicle from the starting point to the parking space based on infrastructure as well as vehicle-mounted sensors to save unnecessary time and energy when parking, thereby minimizing cost and environmental pollution.

The seventh applied technology is an automatic parking system. This is a technology in which the driver stops the car in front of the parking lot, turns off the engine, gets off, and presses the remote control lock switch twice in a row. am.

The eighth applied technology is a blind spot information guidance system. Sensors mounted on both sides of the car determine if there is another vehicle in the blind spot that is not visible through the side mirror and warn the driver. .

The biggest advantage of autonomous driving is that driving speed and traffic management data match, so it helps fuel efficiency by avoiding repeated stops by adjusting the control device more evenly, and that it can be used by people who cannot drive, such as the elderly, children, and the disabled. will be. In addition, it has the advantage of solving fatigue caused by long-time driving, greatly reducing the risk of traffic accidents, speeding up traffic flow on the road and reducing traffic congestion.

Furthermore, in the case of using an autonomous vehicle, there is no need to drive, so leisure time is secured and various user experiences can be derived. More specifically, since they do not drive, they are likely to be exposed to various multimedia environments. However, only basic scenarios have been explored for various user experiences in the case of using an autonomous vehicle.

Korea Patent No. 10-2184598 (2020. 12. 3.), Driving Prediction and Safety Driving System Based on Judgment of Driver Emergency Situation of Autonomous Driving Vehicle}

The present invention is to solve the problems of the prior art, and an object of the present invention is an autonomous driving multimedia system that provides a new user experience by loading a virtual environment when multimedia contents are loaded while an autonomous vehicle is driving. is in providing

An autonomous driving multimedia system according to an aspect of the present invention includes a sensing unit that detects surrounding objects while driving and an output unit that outputs multimedia content, and the output unit includes the detected object in the multimedia content and outputs the same.

At this time, the output unit may delete or change the sensed object and include it in the multimedia content and output the object.

In addition, the multimedia content may be displayed on a glass of a vehicle made of a transparent display.

In addition, the output unit may output the multimedia content including the actual road currently being driven.

In addition, when manual driving is being performed when the output of the multimedia contents is started, autonomous driving may be started together.

According to the present invention, when multimedia contents are output while the self-driving vehicle is running, detected surrounding objects are output together, thus providing a new user experience, and safe self-driving since surrounding vehicles, pedestrians, and roads are output in mixed reality. Build the environment as well.

1 is a configuration diagram of an autonomous driving multimedia system according to an embodiment of the present invention.
FIG. 2 is a configuration diagram showing the self-driving vehicle in FIG. 1 in more detail.
FIG. 3 is a configuration diagram showing the sensing unit in FIG. 2 in more detail.
4 is a configuration diagram showing the output unit in FIG. 2 in more detail.
FIG. 5 is a configuration diagram showing the control unit in FIG. 2 in more detail.
6 to 10 are diagrams illustrating the operation of an autonomous driving multimedia system according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves.

In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted.

In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, an autonomous driving multimedia system according to an embodiment of the present invention will be described. 1 is a configuration diagram of an autonomous driving multimedia system according to an embodiment of the present invention, FIG. 2 is a configuration diagram showing the autonomous vehicle in FIG. 1 in more detail, and FIG. 3 is a more detailed configuration diagram of the sensing unit in FIG. 4 is a configuration diagram showing the output unit in FIG. 2 in more detail, FIG. 5 is a configuration diagram showing the control unit in FIG. 2 in more detail, and FIGS. 6 to 10 show the present invention. It is a diagram illustrating the operation of an autonomous driving multimedia system according to an embodiment of.

Referring to the drawings, first of all, an autonomous driving multimedia system 1000 according to an embodiment of the present invention largely includes an autonomous vehicle 100 and a server 200 that stores multimedia contents and supports output of a virtual environment. .

In this case, the autonomous vehicle 100 includes an input unit 110, a sensing unit 120, an output unit 130, a control unit 140, a communication unit 150, and a driving unit 160.

The input unit 110 is a device that receives a user input for driving. In case of manual mode, the input unit 110 of the autonomous vehicle 100 may include a steering input device, an acceleration input device, and a brake input device. In addition, the input unit 110 serves to input the destination of the self-driving vehicle, and the control unit 140 controls driving or receives information from the server 200 according to the input destination and continuously sets a route.

In addition, the input unit 110 also serves to select multimedia content. These multimedia contents are stored in the server 200 or a storage unit (not shown), and the input unit 110 plays a role in selecting which contents to load.

The detector 120 may include a radar 121 and a lidar 122, and may further include a camera 123. First, the radar 121 may generate information about an object outside the self-driving vehicle 100 by using radio waves. The radar 121 may include an electromagnetic wave transmitter, an electromagnetic wave receiver, and at least one processor electrically connected to the electromagnetic wave transmitter and electromagnetic wave receiver, processing a received signal, and generating data about an object based on the processed signal. there is.

The radar 121 may be implemented in a pulse radar method or a continuous wave radar method in terms of radio wave emission principles. The radar 121 may be implemented in a frequency modulated continuous wave (FMCW) method or a frequency shift keyong (FSK) method according to a signal waveform among continuous wave radar methods. The radar 121 detects an object based on a Time of Flight (TOF) method or a phase-shift method through electromagnetic waves, and measures the position of the detected object, the distance to the detected object, and the relative speed. can be detected. In this case, the radar 121 may be disposed at an appropriate location outside the vehicle to detect an object located in front, rear or side of the vehicle.

The lidar 122 may generate information about an object outside the self-driving vehicle 100 by using laser light. The LIDAR 122 is electrically connected to the light transmitter (not shown), the light receiver (not shown), and the light transmitter and the light receiver, processes the received signal, and generates data for an object based on the processed signal. It may include at least one processor that

The lidar 122 may be implemented in a Time of Flight (TOF) method or a phase-shift method. The lidar 122 may be implemented as a driven or non-driven type. When implemented as a driven type, the lidar 122 is rotated by a motor and may detect objects around the autonomous vehicle 100. When implemented as a non-driving type, the lidar 122 may detect an object located within a predetermined range with respect to the vehicle by light steering. The autonomous vehicle 100 may include a plurality of non-driven lidars. The lidar 122 detects an object based on a time of flight (TOF) method or a phase-shift method using a laser light medium, and calculates the position of the detected object, the distance to the detected object, and the relative speed. can be detected. At this time, the lidar 122 may be disposed at an appropriate location outside the vehicle to detect an object located in the front, rear, or side of the vehicle.

Meanwhile, the camera 123 may generate information about an object outside the self-driving vehicle 100 by using an image. At this time, the camera 123 may capture an image of a target outside the vehicle.

The camera 123 may include at least one lens, at least one image sensor, and at least one processor electrically connected to the image sensor to process a received signal and to generate object data based on the processed signal. can

The camera 123 may be at least one of a mono camera, a stereo camera, and an AVM (Around View Monitoring) camera. The camera 123 may obtain position information of an object, distance information to the object, relative speed information to the object, and motion information of the object by using various image processing algorithms.

For example, the camera 123 may obtain distance information and relative speed information with respect to the object based on a change in the size of the object over time in the obtained image. In addition, the camera 123 may obtain distance information and relative speed information with an object through a pinhole model, road profiling, and the like.

In addition, the camera 123 may obtain basic information about motion, such as distance information to an object, relative speed information, and other people's hand gestures, based on disparity information from a stereo image obtained from a stereo camera. there is.

The camera 123 may be mounted in a position where a field of view (FOV) can be secured in the vehicle in order to photograph the outside of the vehicle. The camera 123 may be disposed close to the front windshield inside the vehicle to obtain an image of the front of the vehicle. Furthermore, the camera 123 may be disposed around a front bumper or a radiator grill. The camera 123 may be disposed close to the rear glass inside the vehicle to obtain an image behind the vehicle. In this case, the camera 123 may be disposed around a rear bumper, a trunk, or a tailgate. The camera 123 may be disposed close to at least one of the side windows in the interior of the vehicle in order to acquire an image of the side of the vehicle. Alternatively, the camera 123 may be disposed around side mirrors, fenders, or doors.

The radar 121, the lidar 122, and the camera 123 recognize roads, nearby structures, buildings, terrain, other vehicles, pedestrians, and the like around the vehicle. The detected object may be included in the virtual environment appearing in the multimedia content or deleted by the output unit 130 . In addition, the detected object may be transformed and output in a virtual environment by the output unit 130 .

In addition, since the sensing unit 120 needs to utilize the location information of the self-driving vehicle, the GPS 124 is essentially further included. In addition, the GPS 124 may include at least one of a general GPS (Global Positioning System) and DGPS (Differential Global Positioning System) to generate location data of the autonomous vehicle 100 . Location data of the self-driving vehicle 100 may be generated based on a signal generated by at least one of the GPS and DGPS.

In this case, the GPS 124 may correct the location data based on at least one of an Inertial Measurement Unit (IMU) and the camera 123 of the sensing unit 120 . Also, the GPS 124 may be referred to as a Global Navigation Satellite System (GNSS).

Meanwhile, the sensing unit 120 may further include a microphone 125 so that the occupant, in addition to the guardian, may intervene in autonomous driving by voice or the like even if the occupant does not perform a steering operation.

The biometric information sensor 126 senses a passenger's heartbeat, blood pressure, brain wave, etc. to promote safety of the passenger. Meanwhile, the biometric information sensor 126 may additionally perform a function of preventing erroneous boarding by sensing fingerprint and iris information when entering and exiting a vehicle. Utilizing the biometric information sensor 126, the self-driving vehicle 100 can detect that a passenger has boarded or alighted.

The output unit 130 includes a general output module 131, a virtual environment output module 132, a road output module 133, and a state output module 134. Here, the general output module 131, the virtual environment output module 132, and the road output module 133 are preferably made of a transparent display on the front, side, or rear glass of the vehicle.

First, the general output module 131 outputs the multimedia content received from the server, and the virtual environment output module 132 includes the objects detected by the sensor 120 in the multimedia content and outputs the output. At this time, the virtual environment output module 132 may delete or change the detected object and include it in the multimedia content to be output. For example, the virtual environment output module 132 may delete the detected pedestrian or vehicle or change and output the face of the pedestrian according to a deep fake technique.

In addition, the road output module 133 serves to output roads included in the virtual environment. At this time, the background of the virtual environment and the road may change together as the vehicle moves. That is, when driving in a virtual environment, objects disposed remotely in the virtual environment become closer as the vehicle travels, and objects disposed close to the virtual environment move away as the vehicle travels. To this end, the server 200 or a storage unit (not shown) may store continuous images of roads captured while driving in a virtual environment.

However, if the virtual environment is not related to driving, there is a possibility that the degree of immersion in viewing may drop sharply. Therefore, the output of the road in the virtual environment is very important, so the role of the road output module 133 is important. Accordingly, the road output module 133 may replace the currently driving road with a road included in the virtual environment and output the same.

By the way, since the road is a very important element when feeling driving, it is preferable that the above-described sensing unit detects a real road and the road output module 133 outputs a real road, not a virtual road, in a virtual environment. In this case, it is possible to output using a mixed reality technique in which real roads are mixed in a virtual environment.

The state output module 134 performs an output notifying the outside of the vehicle that a virtual environment is currently being output to the vehicle. Even in this case, any one of shape, shape, and color may be displayed on the outside of the vehicle or text may be output.

Meanwhile, the controller 140 includes a drive control module 141 and an input/output control module 142. First, the driving control module 141 may be configured as a main ECU and controls the driving unit 160 of the autonomous vehicle 100 . In this case, the drive control module 141 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train driving control device may include a power source driving control device and a transmission driving control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device drive control device may include a seat belt drive control device for controlling seat belts.

In addition, the drive control module 141 includes at least one electronic control device (eg, a control ECU (Electronic Control Unit)). In particular, the vehicle driving device may be controlled based on the received signal. For example, the driving control module 141 may control a power train, a steering device, and a brake device based on a signal received from the sensing unit 120 . In addition, the driving control module 141 also plays a role of completely switching to autonomous driving when manual driving is being performed when the output of the above-described multimedia contents is started. This is because the risk of driving manually increases depending on the viewing of the content.

The input/output control module 142 controls the above-described output unit 130 and calculates a deviation between the virtual environment and the real environment detected by the sensing unit. For example, if the width of the road in the virtual environment is different from the width of the road currently being driven, the road in the virtual environment is matched to the width of the road currently being driven. In addition, when the directions of the current driving environment and the virtual environment are different from each other, the input/output control module 142 preferably matches the directions so that the output unit outputs the output, thereby increasing the degree of immersion in virtual reality.

Furthermore, the input/output control module 142 controls the output unit to output an object or a background or to change and output the object or background.

The communication unit 150 may exchange signals with at least one of the server 200 located outside the autonomous vehicle 100, another vehicle, and a terminal. The communication unit 150 may include at least one of a transmission antenna, a reception antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

For example, the communication device may exchange signals with an external device based on C-V2X (Cellular V2X) technology. In addition, the communication unit 150 transmits external devices and signals based on Dedicated Short Range Communications (DSRC) technology based on IEEE 802.11p PHY/MAC layer technology and IEEE 1609 Network/Transport layer technology or Wireless Access in Vehicular Environment (WAVE) standard. can be exchanged.

The operation of the present invention is exemplified below. 6 to 10 are diagrams illustrating the operation of an autonomous driving multimedia system according to an embodiment of the present invention.

First, FIG. 6 illustrates an external environment that is actually visible, and when content such as thriller water is output, it is exemplified that a pedestrian is output with a change as shown in FIG. 7 . Furthermore, in FIG. 8, it is exemplified that content is displayed together against a bright forest background as shown in FIG. 9 even though driving on a road late at night.

In addition, as shown in FIG. 10, when the detection unit detects a pedestrian, which is an object greater than the set value, the output of the multimedia content that was outputting the virtual environment is changed to transparent so that the actual street situation can be seen to promote the safety of drivers and pedestrians. there is.

As such, those skilled in the art to which the present invention pertains will be able to understand that the present invention may be embodied in other specific forms without changing its technical spirit or essential features.

In addition, the present specification and drawings disclose preferred embodiments of the present invention, and although specific terms are used, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention. It is not intended to limit the scope of the invention. It is obvious to those skilled in the art that other modified examples based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein.

1000: autonomous driving multimedia system
100: autonomous vehicle
110: input unit
120: sensing unit
130: output unit
140: control unit
150: communication department
160: driving unit
200: server

Claims (5)

a sensing unit that senses surrounding objects while driving; and
an output unit that outputs multimedia content;
including,
The self-driving multimedia system according to claim 1 , wherein the output unit includes the detected object in the multimedia content and outputs it.
According to claim 1,
The self-driving multimedia system of claim 1 , wherein the output unit deletes or changes the detected object and outputs the detected object by including it in the multimedia content.
According to claim 1,
The multimedia content is an autonomous driving multimedia system, characterized in that displayed on the glass of the vehicle made of a transparent display.
According to claim 1,
The self-driving multimedia system according to claim 1 , wherein the output unit outputs multimedia content including an actual road currently being driven.
According to claim 1,
The self-driving multimedia system, characterized in that, when manual driving is being performed when the output of the multimedia contents is started, autonomous driving is started together.