KR20210081484A - Method for providing intelligent contents and autonomous driving system - Google Patents

Abstract

Disclosed are a method for a server in an autonomous driving system to intelligently provide content in an autonomous driving vehicle, and the autonomous driving system thereof. In accordance with one embodiment of the present specifications, the server includes a communication module and a processor. The processor acquires input from at least one input device of the autonomous driving vehicle through the communication module, and outputs content related with the input to at least one output device in the autonomous driving vehicle, while outputting the content based on output device characteristic information of the at least one output device and input device characteristic information of the at least one input device, thereby providing various dialogue patterns considering input/output device characteristics of the autonomous driving vehicle. The autonomous driving vehicle, the user terminal and the server of the present specification can be connected with an artificial intelligence module, a drone (unmanned aerial vehicle: UAV), a robot, an augmented reality (AR) device, a virtual reality (VR) device, a 5G service-related device and the like.

Description

Intelligent content provision method and autonomous driving system {METHOD FOR PROVIDING INTELLIGENT CONTENTS AND AUTONOMOUS DRIVING SYSTEM}

The present specification relates to an intelligent content providing method and an autonomous driving system, and more particularly, to a method in which a server provides intelligent content in an autonomous driving vehicle in consideration of characteristics of an input/output device in the autonomous driving vehicle.

A vehicle may be classified into an internal combustion engine vehicle, an external combustion engine vehicle, a gas turbine vehicle, an electric vehicle, or the like, according to a type of a prime mover used.

An autonomous vehicle refers to a vehicle that can operate by itself without the manipulation of a driver or a passenger, and an autonomous driving system refers to a system that monitors and controls such an autonomous vehicle so that it can operate by itself.

The present specification aims to solve the above-mentioned needs and/or problems.

Another object of the present specification is to implement a method and an autonomous driving system for intelligently providing content according to a vehicle state based on characteristics of an input device and an output device in an autonomous driving vehicle.

In an intelligent content providing method of an autonomous driving system according to an embodiment of the present specification, the method comprising: acquiring, by a server, an input from at least one input device of an autonomous driving vehicle; and outputting, by a server, the content related to the input to at least one output device in the autonomous vehicle, wherein the outputting includes: output device characteristic information of the at least one output device and the at least one and outputting the content based on at least one of input device characteristic information of the input device.

The outputting may include setting a format of the content output to each of the at least one output device based on a location in the vehicle of each of the at least one output device.

The outputting may include setting a format of the content based on information related to the state of the vehicle.

The information related to the state of the vehicle may include a traffic amount, a signal, a driver's location or a driver's state.

The information related to the state of the vehicle may include a driving mode of the vehicle, and the outputting may include determining the amount of the content based on the driving mode.

When the vehicle is in the autonomous driving mode, the number of UI objects included in the content may be greater than the number of UI objects included in the content when the vehicle is in the manual driving mode.

The outputting may include setting the format of the content based on information related to characteristics of an input obtained from each of the at least one input device.

In an autonomous driving system according to another embodiment of the present specification, there is provided a server for intelligently providing content in an autonomous vehicle, comprising: a communication module; and a processor, wherein the processor obtains an input from at least one input device of the autonomous vehicle through the communication module, and outputs content related to the input to at least one output device within the autonomous vehicle. However, the processor may output the content based on at least one of the output device characteristic information of the at least one output device and the input device characteristic information of the at least one input device.

The processor may set the format of the content output to each of the at least one output device based on the location in the vehicle of each of the at least one output device.

The processor may set the format of the content based on information related to the state of the vehicle.

The information related to the state of the vehicle may include a traffic amount, a signal, a driver's location or a driver's state.

The information related to the state of the vehicle may include a driving mode of the vehicle, and the processor may determine the amount of the content based on the driving mode.

The processor may be characterized in that the number of UI objects included in the content when the vehicle is in the autonomous driving mode is greater than the number of UI objects included in the content when the vehicle is in the manual driving mode.

The processor may set the format of the content based on information related to characteristics of an input obtained from each of the at least one input device.

The effect of the intelligent content providing method of the autonomous driving system according to an embodiment of the present specification will be described as follows.

According to the present specification, various conversation patterns in consideration of input/output device characteristics of an autonomous vehicle may be provided.

In addition, according to the present specification, a conversation pattern suitable for a situation may be provided to a user riding in an autonomous vehicle, thereby providing a rich user experience and interface.

The effects obtainable in the present specification are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which this specification belongs from the description below. .

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.
1 illustrates a block diagram of a wireless communication system to which the methods proposed in the present specification can be applied.
2 shows an example of a signal transmission/reception method in a wireless communication system.
3 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.
4 shows an example of a vehicle-to-vehicle basic operation using 5G communication.
5 is a diagram illustrating a vehicle according to an embodiment of the present specification.
6 is a control block diagram of a vehicle according to an embodiment of the present specification.
7 is a control block diagram of an autonomous driving apparatus according to an embodiment of the present specification.
8 is a signal flow diagram of an autonomous vehicle according to an embodiment of the present specification.
9 is a view illustrating the interior of a vehicle according to an embodiment of the present specification.
10 is a block diagram referenced to describe a vehicle cabin system according to an embodiment of the present specification.
11 is a diagram referenced to describe a user's usage scenario according to an embodiment of the present specification.
12 illustrates a block diagram of a schematic system in which a speech recognition method according to an embodiment of the present invention is implemented.
13 is a schematic block diagram of a voice recognition apparatus in a voice recognition system environment according to an embodiment of the present invention.
14 is a schematic block diagram of a voice recognition apparatus in a voice recognition system environment according to another embodiment of the present invention.
15 is a schematic block diagram of an intelligent processor capable of implementing speech recognition according to an embodiment of the present invention.
16 shows an autonomous driving system 100 according to the present disclosure.
17 is a flowchart illustrating a method for providing intelligent content in an autonomous vehicle according to an embodiment of the present specification.
18 is a flowchart illustrating an intelligent content providing method of a vehicle (autonomous driving vehicle) and a server (cloud server) according to an embodiment of the present specification.
19 illustrates an example of providing content based on a driver's voice input in an autonomous driving mode of a vehicle according to an embodiment of the present specification.
20 illustrates an example of providing content based on a driver's voice input and driver's gaze information in an autonomous driving mode of a vehicle according to an embodiment of the present specification.
21 illustrates an example of providing content based on a driver's voice input in a manual driving mode of a vehicle according to an embodiment of the present specification.
22 illustrates another example of providing content based on a driver's voice input in a manual driving mode of a vehicle according to an embodiment of the present specification.
23 illustrates an example of providing content based on a sensing input of a vehicle in an autonomous driving mode of a vehicle according to an embodiment of the present specification.
24 illustrates an example of providing content based on a sensing input of a vehicle in a manual driving mode of a vehicle according to an embodiment of the present specification.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present specification , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as "comprises" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

A. Example UE and 5G network block diagram

1 illustrates a block diagram of a wireless communication system to which the methods proposed in the present specification can be applied.

Referring to FIG. 1 , a device (autonomous driving device) including an autonomous driving module may be defined as a first communication device ( 910 in FIG. 1 ), and a processor 911 may perform a detailed autonomous driving operation.

A 5G network including another vehicle communicating with the autonomous driving device may be defined as a second communication device ( 920 in FIG. 1 ), and the processor 921 may perform a detailed autonomous driving operation.

The 5G network may be represented as the first communication device and the autonomous driving device may be represented as the second communication device.

For example, the first communication device or the second communication device may be a base station, a network node, a transmitting terminal, a receiving terminal, a wireless device, a wireless communication device, an autonomous driving device, or the like.

For example, a terminal or user equipment (UE) includes a vehicle, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, personal digital assistants (PDA), and a portable multimedia player (PMP). , navigation, slate PC, tablet PC, ultrabook, wearable device, for example, watch-type terminal (smartwatch), glass-type terminal (smart glass), HMD ( head mounted display) and the like. For example, the HMD may be a display device worn on the head. For example, an HMD may be used to implement VR, AR or MR. Referring to FIG. 1 , a first communication device 910 and a second communication device 920 include a processor 911,921, a memory 914,924, and one or more Tx/Rx RF modules (radio frequency module, 915,925). , including Tx processors 912 and 922 , Rx processors 913 and 923 , and antennas 916 and 926 . Tx/Rx modules are also called transceivers. Each Tx/Rx module 915 transmits a signal via a respective antenna 926 . The processor implements the functions, processes and/or methods salpinned above. The processor 921 may be associated with a memory 924 that stores program code and data. Memory may be referred to as a computer-readable medium. More specifically, in DL (communication from a first communication device to a second communication device), the transmit (TX) processor 912 implements various signal processing functions for the L1 layer (ie, the physical layer). The receive (RX) processor implements the various signal processing functions of L1 (ie, the physical layer).

The UL (second communication device to first communication device communication) is handled in the first communication device 910 in a manner similar to that described with respect to the receiver function in the second communication device 920 . Each Tx/Rx module 925 receives a signal via a respective antenna 926 . Each Tx/Rx module provides an RF carrier and information to the RX processor 923 . The processor 921 may be associated with a memory 924 that stores program code and data. Memory may be referred to as a computer-readable medium.

B. Signal transmission/reception method in wireless communication system

2 illustrates physical channels and general signal transmission used in a 3GPP system.

In a wireless communication system, a terminal receives information through a downlink (DL) from a base station, and the terminal transmits information through an uplink (UL) to the base station. Information transmitted and received between the base station and the terminal includes data and various control information, and various physical channels exist according to the type/use of the information they transmit and receive.

When the terminal is powered on or newly enters a cell, the terminal performs an initial cell search operation, such as synchronizing with the base station (S201). To this end, the terminal receives a primary synchronization signal (PSS) and a secondary synchronization signal (SSS) from the base station, synchronizes with the base station, and obtains information such as a cell ID. Thereafter, the terminal may receive a physical broadcast channel (PBCH) from the base station to obtain intra-cell broadcast information. On the other hand, the terminal may receive a downlink reference signal (DL RS) in the initial cell search step to check the downlink channel state.

After completing the initial cell search, the UE acquires more specific system information by receiving a Physical Downlink Control Channel (PDCCH) and a Physical Downlink Control Channel (PDSCH) according to information carried on the PDCCH. It can be done (S202).

On the other hand, when the base station is initially accessed or there is no radio resource for signal transmission, the terminal may perform a random access procedure (RACH) for the base station (S203 to S206). To this end, the UE transmits a specific sequence as a preamble through a Physical Random Access Channel (PRACH) (S203 and S205), and a response message to the preamble through the PDCCH and the corresponding PDSCH ((Random Access (RAR)) Response) message) In the case of contention-based RACH, a contention resolution procedure may be additionally performed ( S206 ).

After performing the procedure as described above, the UE performs PDCCH/PDSCH reception (S207) and Physical Uplink Shared Channel (PUSCH)/Physical Uplink Control Channel (Physical Uplink) as a general uplink/downlink signal transmission procedure. Control Channel (PUCCH) transmission (S208) may be performed. In particular, the UE may receive downlink control information (DCI) through the PDCCH. Here, the DCI includes control information such as resource allocation information for the terminal, and different formats may be applied according to the purpose of use.

On the other hand, the control information that the terminal transmits to the base station through the uplink or the terminal receives from the base station includes a downlink/uplink ACK/NACK signal, a channel quality indicator (CQI), a precoding matrix index (PMI), and a rank indicator (RI). ) and the like. The UE may transmit the above-described control information such as CQI/PMI/RI through PUSCH and/or PUCCH.

Referring to FIG. 2 , an initial access (IA) procedure in a 5G communication system will be additionally described.

The UE may perform cell search, system information acquisition, beam alignment for initial access, DL measurement, and the like based on the SSB. The SSB is mixed with an SS/PBCH (Synchronization Signal/Physical Broadcast channel) block.

SSB consists of PSS, SSS and PBCH. The SSB is configured in four consecutive OFDM symbols, and PSS, PBCH, SSS/PBCH or PBCH are transmitted for each OFDM symbol. PSS and SSS consist of 1 OFDM symbol and 127 subcarriers, respectively, and PBCH consists of 3 OFDM symbols and 576 subcarriers.

Cell discovery refers to a process in which the UE acquires time/frequency synchronization of a cell, and detects a cell ID (Identifier) (eg, Physical layer Cell ID, PCI) of the cell. PSS is used to detect a cell ID within a cell ID group, and SSS is used to detect a cell ID group. PBCH is used for SSB (time) index detection and half-frame detection.

There are 336 cell ID groups, and there are 3 cell IDs for each cell ID group. There are a total of 1008 cell IDs. Information on the cell ID group to which the cell ID of the cell belongs is provided/obtained through the SSS of the cell, and information on the cell ID among 336 cells in the cell ID is provided/obtained through the PSS

The SSB is transmitted periodically according to the SSB period (periodicity). The SSB basic period assumed by the UE during initial cell discovery is defined as 20 ms. After cell access, the SSB period may be set to one of {5ms, 10ms, 20ms, 40ms, 80ms, 160ms} by the network (eg, BS).

Next, the acquisition of system information (SI) will be described.

The SI is divided into a master information block (MIB) and a plurality of system information blocks (SIB). SI other than MIB may be referred to as Remaining Minimum System Information (RMSI). The MIB includes information/parameters for monitoring the PDCCH scheduling the PDSCH carrying the System Information Block1 (SIB1) and is transmitted by the BS through the PBCH of the SSB. SIB1 includes information related to availability and scheduling (eg, transmission period, SI-window size) of the remaining SIBs (hereinafter, SIBx, where x is an integer of 2 or more). SIBx is included in the SI message and transmitted through the PDSCH. Each SI message is transmitted within a periodically occurring time window (ie, an SI-window).

Referring to FIG. 2 , a random access (RA) process in a 5G communication system will be additionally described.

The random access process is used for a variety of purposes. For example, the random access procedure may be used for network initial access, handover, and UE-triggered UL data transmission. The UE may acquire UL synchronization and UL transmission resources through a random access procedure. The random access process is divided into a contention-based random access process and a contention free random access process. The detailed procedure for the contention-based random access process is as follows.

The UE may transmit the random access preamble through the PRACH as Msg1 of the random access procedure in the UL. Random access preamble sequences having two different lengths are supported. The long sequence length 839 applies for subcarrier spacings of 1.25 and 5 kHz, and the short sequence length 139 applies for subcarrier spacings of 15, 30, 60 and 120 kHz.

When the BS receives the random access preamble from the UE, the BS sends a random access response (RAR) message (Msg2) to the UE. The PDCCH scheduling the PDSCH carrying the RAR is CRC-masked and transmitted with a random access (RA) radio network temporary identifier (RNTI) (RA-RNTI). The UE detecting the PDCCH masked by the RA-RNTI may receive the RAR from the PDSCH scheduled by the DCI carried by the PDCCH. The UE checks whether the random access response information for the preamble it has transmitted, that is, Msg1, is in the RAR. Whether or not random access information for Msg1 transmitted by itself exists may be determined by whether a random access preamble ID for the preamble transmitted by the UE exists. If there is no response to Msg1, the UE may retransmit the RACH preamble within a predetermined number of times while performing power ramping. The UE calculates the PRACH transmit power for the retransmission of the preamble based on the most recent path loss and power ramping counter.

The UE may transmit UL transmission on the uplink shared channel as Msg3 of the random access procedure based on the random access response information. Msg3 may include an RRC connection request and a UE identifier. As a response to Msg3, the network may send Msg4, which may be treated as a contention resolution message on DL. By receiving Msg4, the UE can enter the RRC connected state.

C. Beam Management (BM) Procedure of 5G Communication System

The BM process may be divided into (1) a DL BM process using SSB or CSI-RS, and (2) a UL BM process using a sounding reference signal (SRS). In addition, each BM process may include Tx beam sweeping to determine a Tx beam and Rx beam sweeping to determine an Rx beam.

Let's look at the DL BM process using SSB.

A configuration for a beam report using the SSB is performed during channel state information (CSI)/beam configuration in RRC_CONNECTED.

- The UE receives from the BS a CSI-ResourceConfig IE including a CSI-SSB-ResourceSetList for SSB resources used for BM. The RRC parameter csi-SSB-ResourceSetList indicates a list of SSB resources used for beam management and reporting in one resource set. Here, the SSB resource set may be set to {SSBx1, SSBx2, SSBx3, SSBx4, ...}. The SSB index may be defined from 0 to 63.

- UE receives signals on SSB resources from the BS based on the CSI-SSB-ResourceSetList.

- When the CSI-RS reportConfig related to reporting on SSBRI and reference signal received power (RSRP) is configured, the UE reports the best SSBRI and RSRP corresponding thereto to the BS. For example, when the reportQuantity of the CSI-RS reportConfig IE is set to 'ssb-Index-RSRP', the UE reports the best SSBRI and the corresponding RSRP to the BS.

If the CSI-RS resource is configured in the same OFDM symbol(s) as the SSB, and 'QCL-TypeD' is applicable, the UE has the CSI-RS and the SSB similarly located in the 'QCL-TypeD' point of view ( quasi co-located, QCL). Here, QCL-TypeD may mean QCL between antenna ports in terms of spatial Rx parameters. When the UE receives signals of a plurality of DL antenna ports in a QCL-TypeD relationship, the same reception beam may be applied.

Next, a DL BM process using CSI-RS will be described.

The Rx beam determination (or refinement) process of the UE using the CSI-RS and the Tx beam sweeping process of the BS will be described in turn. In the UE Rx beam determination process, the repetition parameter is set to 'ON', and in the BS Tx beam sweeping process, the repetition parameter is set to 'OFF'.

First, a process of determining the Rx beam of the UE will be described.

- The UE receives the NZP CSI-RS resource set IE including the RRC parameter for 'repetition' from the BS through RRC signaling. Here, the RRC parameter 'repetition' is set to 'ON'.

- The UE repeats signals on the resource(s) in the CSI-RS resource set in which the RRC parameter 'repetition' is set to 'ON' in different OFDM symbols through the same Tx beam (or DL spatial domain transmission filter) of the BS receive

- The UE determines its own Rx beam.

- The UE omits CSI reporting. That is, the UE may omit the CSI report when the multi-RRC parameter 'repetition' is set to 'ON'.

Next, the Tx beam determination process of the BS will be described.

- The UE receives the NZP CSI-RS resource set IE including the RRC parameter for 'repetition' from the BS through RRC signaling. Here, the RRC parameter 'repetition' is set to 'OFF' and is related to the Tx beam sweeping process of the BS.

- The UE receives signals on resources in the CSI-RS resource set in which the RRC parameter 'repetition' is set to 'OFF' through different Tx beams (DL spatial domain transmission filter) of the BS.

- The UE selects (or determines) the best beam.

- The UE reports the ID (eg, CRI) and related quality information (eg, RSRP) for the selected beam to the BS. That is, when the CSI-RS is transmitted for the BM, the UE reports the CRI and the RSRP to the BS.

Next, a UL BM process using SRS will be described.

- The UE receives the RRC signaling (eg, SRS-Config IE) including the (RRC parameter) usage parameter set to 'beam management' from the BS. SRS-Config IE is used for SRS transmission configuration. The SRS-Config IE includes a list of SRS-Resources and a list of SRS-ResourceSets. Each SRS resource set means a set of SRS-resources.

- The UE determines Tx beamforming for the SRS resource to be transmitted based on the SRS-SpatialRelation Info included in the SRS-Config IE. Here, the SRS-SpatialRelation Info is set for each SRS resource and indicates whether to apply the same beamforming as that used in SSB, CSI-RS, or SRS for each SRS resource.

- If SRS-SpatialRelationInfo is configured in the SRS resource, the same beamforming as that used in SSB, CSI-RS, or SRS is applied and transmitted. However, if SRS-SpatialRelationInfo is not configured in the SRS resource, the UE arbitrarily determines Tx beamforming and transmits the SRS through the determined Tx beamforming.

Next, a beam failure recovery (BFR) process will be described.

In a beamformed system, Radio Link Failure (RLF) may frequently occur due to rotation, movement, or beamforming blockage of the UE. Therefore, BFR is supported in NR to prevent frequent RLF from occurring. BFR is similar to the radio link failure recovery process, and can be supported when the UE knows new candidate beam(s). For beam failure detection, the BS sets beam failure detection reference signals to the UE, and the UE determines that the number of beam failure indications from the physical layer of the UE is within a period set by the RRC signaling of the BS. When a threshold set by RRC signaling is reached (reach), a beam failure is declared (declare). after beam failure is detected, the UE triggers beam failure recovery by initiating a random access procedure on the PCell; Beam failure recovery is performed by selecting a suitable beam (if the BS provides dedicated random access resources for certain beams, these are prioritized by the UE). Upon completion of the random access procedure, it is considered that beam failure recovery has been completed.

D. URLLC (Ultra-Reliable and Low Latency Communication)

URLLC transmission defined in NR is (1) a relatively low traffic size, (2) a relatively low arrival rate (low arrival rate), (3) extremely low latency requirements (eg, 0.5, 1ms), (4) a relatively short transmission duration (eg, 2 OFDM symbols), and (5) transmission for an urgent service/message. In the case of UL, transmission for a specific type of traffic (eg, URLLC) is multiplexed with other previously scheduled transmission (eg, eMBB) in order to satisfy a more stringent latency requirement. Needs to be. In this regard, as one method, information to be preempted for a specific resource is given to the previously scheduled UE, and the resource is used by the URLLC UE for UL transmission.

For NR, dynamic resource sharing between eMBB and URLLC is supported. eMBB and URLLC services may be scheduled on non-overlapping time/frequency resources, and URLLC transmission may occur on resources scheduled for ongoing eMBB traffic. The eMBB UE may not know whether the PDSCH transmission of the corresponding UE is partially punctured, and the UE may not be able to decode the PDSCH due to corrupted coded bits. In consideration of this, NR provides a preemption indication. The preemption indication may be referred to as an interrupted transmission indication.

With respect to the preemption indication, the UE receives the DownlinkPreemption IE through RRC signaling from the BS. When the UE is provided with the DownlinkPreemption IE, for monitoring the PDCCH carrying DCI format 2_1, the UE is configured with the INT-RNTI provided by the parameter int-RNTI in the DownlinkPreemption IE. The UE is additionally configured with a set of serving cells by INT-ConfigurationPerServing Cell including a set of serving cell indices provided by servingCellID and a corresponding set of positions for fields in DCI format 2_1 by positionInDCI, dci-PayloadSize It is established with the information payload size for DCI format 2_1 by , and is set with the indicated granularity of time-frequency resources by timeFrequencySect.

The UE receives DCI format 2_1 from the BS based on the DownlinkPreemption IE.

When the UE detects the DCI format 2_1 for the serving cell in the configured set of serving cells, the UE determines that the DCI format of the set of PRBs and the set of symbols of the monitoring period immediately preceding the monitoring period to which the DCI format 2_1 belongs. It can be assumed that there is no transmission to the UE in the PRBs and symbols indicated by 2_1. For example, the UE sees that the signal in the time-frequency resource indicated by the preemption is not the scheduled DL transmission for itself and decodes data based on the signals received in the remaining resource region.

E. mMTC (massive MTC)

mMTC (massive machine type communication) is one of the scenarios of 5G to support hyper-connectivity service that communicates simultaneously with a large number of UEs. In this environment, the UE communicates intermittently with a very low transmission rate and mobility. Therefore, mMTC is primarily aimed at how long the UE can run at a low cost. In relation to mMTC technology, 3GPP deals with MTC and NB (NarrowBand)-IoT.

The mMTC technology has features such as repeated transmission of PDCCH, PUCCH, physical downlink shared channel (PDSCH), PUSCH, and the like, frequency hopping, retuning, and guard period.

That is, a PUSCH (or PUCCH (particularly, long PUCCH) or PRACH) including specific information and a PDSCH (or PDCCH) including a response to specific information are repeatedly transmitted. Repeated transmission is performed through frequency hopping, and for repeated transmission, (RF) retuning is performed in a guard period from a first frequency resource to a second frequency resource, and specific information And a response to specific information may be transmitted/received through a narrowband (ex. 6 RB (resource block) or 1 RB).

F. Basic operation between autonomous vehicles using 5G communication

3 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.

The autonomous vehicle transmits specific information transmission to the 5G network (S1). The specific information may include autonomous driving-related information. Then, the 5G network may determine whether to remotely control the vehicle (S2). Here, the 5G network may include a server or module for performing remote control related to autonomous driving. In addition, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle (S3).

G. Application operation between autonomous vehicle and 5G network in 5G communication system

Hereinafter, the operation of the autonomous vehicle using 5G communication will be described in more detail with reference to FIGS. 1 and 2 and the above salpin wireless communication technology (BM procedure, URLLC, Mmtc, etc.).

First, the method proposed in this specification, which will be described later, and the basic procedure of the application operation to which the eMBB technology of 5G communication is applied will be described.

As in steps S1 and S3 of FIG. 3 , in order for the autonomous vehicle to transmit/receive signals and information to and from the 5G network, the autonomous vehicle performs an initial access procedure with the 5G network before step S1 of FIG. 3 . and a random access procedure.

More specifically, the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB to obtain DL synchronization and system information. A beam management (BM) process and a beam failure recovery process may be added to the initial access procedure, and in the process of the autonomous vehicle receiving a signal from the 5G network, QCL (quasi-co location) ) relationship can be added.

In addition, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission. The 5G network may transmit a UL grant for scheduling transmission of specific information to the autonomous vehicle. have. Accordingly, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. In addition, the 5G network transmits a DL grant for scheduling transmission of a 5G processing result for the specific information to the autonomous vehicle. Accordingly, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle based on the DL grant.

Next, the method proposed in the present specification to be described later and the basic procedure of the application operation to which the URLLC technology of 5G communication is applied will be described.

As described above, after the autonomous vehicle performs an initial access procedure and/or a random access procedure with the 5G network, the autonomous vehicle may receive a DownlinkPreemption IE from the 5G network. Then, the autonomous vehicle receives DCI format 2_1 including a pre-emption indication from the 5G network based on the DownlinkPreemption IE. In addition, the autonomous vehicle does not perform (or expect or assume) the reception of eMBB data in the resource (PRB and/or OFDM symbol) indicated by the pre-emption indication. Thereafter, the autonomous vehicle may receive a UL grant from the 5G network when it is necessary to transmit specific information.

Next, the method proposed in the present specification, which will be described later, and the basic procedure of the application operation to which the mMTC technology of 5G communication is applied will be described.

Among the steps of FIG. 3, the parts that are changed by the application of mMTC technology will be mainly described.

In step S1 of FIG. 3 , the autonomous vehicle receives a UL grant from the 5G network to transmit specific information to the 5G network. Here, the UL grant includes information on the number of repetitions for the transmission of the specific information, and the specific information may be repeatedly transmitted based on the information on the number of repetitions. That is, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. In addition, repeated transmission of specific information may be performed through frequency hopping, transmission of the first specific information may be transmitted in a first frequency resource, and transmission of the second specific information may be transmitted in a second frequency resource. The specific information may be transmitted through a narrowband of 6RB (Resource Block) or 1RB (Resource Block).

F. Vehicle-to-vehicle autonomous driving operation using 5G communication

4 illustrates an example of a vehicle-to-vehicle basic operation using 5G communication.

The first vehicle transmits specific information to the second vehicle (S61). The second vehicle transmits a response to the specific information to the first vehicle (S62).

On the other hand, depending on whether the 5G network is directly (sidelink communication transmission mode 3) or indirectly (sidelink communication transmission mode 4) involved in the resource allocation of the specific information and the response to the specific information, the vehicle-to-vehicle application operation Configuration may vary.

Next, a vehicle-to-vehicle application operation using 5G communication will be examined.

First, how the 5G network is directly involved in resource allocation of vehicle-to-vehicle signal transmission/reception will be described.

The 5G network may transmit DCI format 5A to the first vehicle for scheduling of mode 3 transmission (PSCCH and/or PSSCH transmission). Here, a physical sidelink control channel (PSCCH) is a 5G physical channel for scheduling specific information transmission, and a physical sidelink shared channel (PSSCH) is a 5G physical channel for transmitting specific information. Then, the first vehicle transmits SCI format 1 for scheduling specific information transmission to the second vehicle on the PSCCH. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

Next, how the 5G network is indirectly involved in resource allocation of signal transmission/reception will be examined.

The first vehicle senses a resource for mode 4 transmission in the first window. Then, the first vehicle selects a resource for mode 4 transmission in the second window based on the sensing result. Here, the first window means a sensing window, and the second window means a selection window. The first vehicle transmits SCI format 1 for scheduling of transmission of specific information to the second vehicle on the PSCCH based on the selected resource. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

The above salpin 5G communication technology may be applied in combination with the methods proposed in the present specification to be described later, or may be supplemented to specify or clarify the technical characteristics of the methods proposed in the present specification.

Driving

(1) Vehicle exterior

5 is a diagram illustrating a vehicle according to an embodiment of the present specification.

Referring to FIG. 5 , the vehicle 10 according to the embodiment of the present specification is defined as a transportation means traveling on a road or track. The vehicle 10 is a concept including a car, a train, and a motorcycle. The vehicle 10 may be a concept including all of an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, and an electric vehicle having an electric motor as a power source. The vehicle 10 may be a vehicle owned by an individual. The vehicle 10 may be a shared vehicle. The vehicle 10 may be an autonomous vehicle.

(2) Components of the vehicle

6 is a control block diagram of a vehicle according to an embodiment of the present specification.

Referring to FIG. 6 , the vehicle 10 includes a user interface device 200 , an object detection device 210 , a communication device 220 , a driving manipulation device 230 , a main ECU 240 , and a driving control device 250 . ), an autonomous driving device 260 , a sensing unit 270 , and a location data generating device 280 . The object detecting device 210 , the communication device 220 , the driving manipulation device 230 , the main ECU 240 , the driving control device 250 , the autonomous driving device 260 , the sensing unit 270 , and the location data generating device 280 may be implemented as electronic devices that each generate electrical signals and exchange electrical signals with each other.

1) User interface device

The user interface device 200 is a device for communication between the vehicle 10 and a user. The user interface device 200 may receive a user input and provide information generated in the vehicle 10 to the user. The vehicle 10 may implement a user interface (UI) or a user experience (UX) through the user interface device 200 . The user interface device 200 may include an input device, an output device, and a user monitoring device.

2) Object detection device

The object detection apparatus 210 may generate information about an object outside the vehicle 10 . The information about the object may include at least one of information on the existence of the object, location information of the object, distance information between the vehicle 10 and the object, and relative speed information between the vehicle 10 and the object. . The object detecting apparatus 210 may detect an object outside the vehicle 10 . The object detecting apparatus 210 may include at least one sensor capable of detecting an object outside the vehicle 10 . The object detecting apparatus 210 may include at least one of a camera, a radar, a lidar, an ultrasonic sensor, and an infrared sensor. The object detecting apparatus 210 may provide data on an object generated based on a sensing signal generated by a sensor to at least one electronic device included in the vehicle.

2.1) Camera

The camera may generate information about an object outside the vehicle 10 by using the image. The camera may include at least one lens, at least one image sensor, and at least one processor that is electrically connected to the image sensor to process a received signal, and generate data about the object based on the processed signal.

The camera may be at least one of a mono camera, a stereo camera, and an AVM (Around View Monitoring) camera. The camera may obtain position information of the object, distance information from the object, or relative speed information with the object by using various image processing algorithms. For example, the camera may acquire distance information and relative velocity information from an object based on a change in the size of the object over time from the acquired image. For example, the camera may acquire distance information and relative speed information with respect to an object through a pinhole model, road surface profiling, or the like. For example, the camera may acquire distance information and relative velocity information from an object based on disparity information in a stereo image obtained from the stereo camera.

The camera may be mounted at a position in which a field of view (FOV) can be secured in the vehicle to photograph the outside of the vehicle. The camera may be disposed adjacent to the front windshield in the interior of the vehicle to acquire an image of the front of the vehicle. The camera may be placed around the front bumper or radiator grill. The camera may be disposed adjacent to the rear glass in the interior of the vehicle to acquire an image of the rear of the vehicle. The camera may be placed around the rear bumper, trunk or tailgate. The camera may be disposed adjacent to at least one of the side windows in the interior of the vehicle in order to acquire an image of the side of the vehicle. Alternatively, the camera may be disposed around a side mirror, a fender or a door.

2.2) Radar

The radar may generate information about an object outside the vehicle 10 using radio waves. The radar may include an electromagnetic wave transmitter, an electromagnetic wave receiver, and at least one processor that is electrically connected to the electromagnetic wave transmitter and the electromagnetic wave receiver, processes a received signal, and generates data for an object based on the processed signal. The radar may be implemented in a pulse radar method or a continuous wave radar method in terms of a radio wave emission principle. The radar may be implemented in a frequency modulated continuous wave (FMCW) method or a frequency shift keyong (FSK) method according to a signal waveform among continuous wave radar methods. The radar detects an object based on an electromagnetic wave, a time of flight (TOF) method or a phase-shift method, and detects the position of the detected object, the distance to the detected object, and the relative speed. can The radar may be placed at a suitable location outside of the vehicle to detect objects located in front, rear or side of the vehicle.

2.3) Lidar

The lidar may generate information about an object outside the vehicle 10 by using laser light. The lidar may include at least one processor that is electrically connected to the light transmitter, the light receiver, and the light transmitter and the light receiver, processes the received signal, and generates data about the object based on the processed signal. . The lidar may be implemented in a time of flight (TOF) method or a phase-shift method. Lidar can be implemented as driven or non-driven. When implemented as a driving type, the lidar is rotated by a motor and may detect an object around the vehicle 10 . When implemented as a non-driven type, the lidar may detect an object located within a predetermined range with respect to the vehicle by light steering. Vehicle 100 may include a plurality of non-driven lidar. LiDAR detects an object based on a time of flight (TOF) method or a phase-shift method with a laser light medium, and calculates the position of the detected object, the distance to the detected object, and the relative speed. can be detected. The lidar may be placed at a suitable location outside of the vehicle to detect an object located in front, rear or side of the vehicle.

3) communication device

The communication apparatus 220 may exchange signals with a device located outside the vehicle 10 . The communication device 220 may exchange signals with at least one of an infrastructure (eg, a server, a broadcasting station), another vehicle, and a terminal. The communication device 220 may include at least one of a transmit antenna, a receive antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

For example, the communication apparatus may exchange a signal with an external device based on C-V2X (Cellular V2X) technology. For example, the C-V2X technology may include LTE-based sidelink communication and/or NR-based sidelink communication. Contents related to C-V2X will be described later.

For example, communication devices communicate with external devices based on IEEE 802.11p PHY/MAC layer technology and IEEE 1609 Network/Transport layer technology-based Dedicated Short Range Communications (DSRC) technology or WAVE (Wireless Access in Vehicular Environment) standard. can be exchanged for DSRC (or WAVE standard) technology is a communication standard prepared to provide ITS (Intelligent Transport System) service through short-distance dedicated communication between in-vehicle devices or between roadside devices and in-vehicle devices. The DSRC technology may use a frequency of 5.9 GHz and may be a communication method having a data transmission rate of 3 Mbps to 27 Mbps. IEEE 802.11p technology can be combined with IEEE 1609 technology to support DSRC technology (or WAVE standard).

The communication apparatus of the present specification may exchange a signal with an external device using either one of the C-V2X technology or the DSRC technology. Alternatively, the communication apparatus of the present specification may exchange signals with an external device by hybridizing C-V2X technology and DSRC technology.

4) Driving control device

The driving operation device 230 is a device that receives a user input for driving. In the manual mode, the vehicle 10 may be driven based on a signal provided by the driving manipulation device 230 . The driving manipulation device 230 may include a steering input device (eg, a steering wheel), an acceleration input device (eg, an accelerator pedal), and a brake input device (eg, a brake pedal).

5) Main ECU

The main ECU 240 may control the overall operation of at least one electronic device included in the vehicle 10 .

6) drive control device

The drive control device 250 is a device that electrically controls various vehicle drive devices in the vehicle 10 . The drive control device 250 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train drive control device may include a power source drive control device and a transmission drive control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device drive control device may include a safety belt drive control device for seat belt control.

The drive control device 250 includes at least one electronic control device (eg, a control ECU (Electronic Control Unit)).

The pitch control device 250 may control the vehicle driving device based on a signal received from the autonomous driving device 260 . For example, the control device 250 may control a power train, a steering device, and a brake device based on a signal received from the autonomous driving device 260 .

7) autonomous driving device

The autonomous driving device 260 may generate a path for autonomous driving based on the obtained data. The autonomous driving device 260 may generate a driving plan for driving along the generated path. The autonomous driving device 260 may generate a signal for controlling the movement of the vehicle according to the driving plan. The autonomous driving device 260 may provide the generated signal to the driving control device 250 .

The autonomous driving apparatus 260 may implement at least one Advanced Driver Assistance System (ADAS) function. ADAS includes Adaptive Cruise Control (ACC), Autonomous Emergency Braking (AEB), Forward Collision Warning (FCW), Lane Keeping Assist (LKA), ), Lane Change Assist (LCA), Target Following Assist (TFA), Blind Spot Detection (BSD), Adaptive High Beam Control (HBA) , Auto Parking System (APS), Pedestrian Collision Warning System (PD Collision Warning System), Traffic Sign Recognition (TSR), Trafffic Sign Assist (TSA), Night Vision System At least one of a Night Vision (NV), a Driver Status Monitoring (DSM), and a Traffic Jam Assist (TJA) may be implemented.

The autonomous driving device 260 may perform a switching operation from the autonomous driving mode to the manual driving mode or a switching operation from the manual driving mode to the autonomous driving mode. For example, the autonomous driving device 260 may switch the mode of the vehicle 10 from the autonomous driving mode to the manual driving mode or from the manual driving mode to the autonomous driving mode based on a signal received from the user interface device 200 . can be converted to

8) Sensing unit

The sensing unit 270 may sense the state of the vehicle. The sensing unit 270 may include an inertial measurement unit (IMU) sensor, a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensor, a heading sensor, a position module, and a vehicle. It may include at least one of a forward/reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor, a temperature sensor, a humidity sensor, an ultrasonic sensor, an illuminance sensor, and a pedal position sensor. Meanwhile, an inertial measurement unit (IMU) sensor may include at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor.

The sensing unit 270 may generate state data of the vehicle based on a signal generated by at least one sensor. The vehicle state data may be information generated based on data sensed by various sensors provided inside the vehicle. The sensing unit 270 may include vehicle attitude data, vehicle motion data, vehicle yaw data, vehicle roll data, vehicle pitch data, vehicle collision data, vehicle direction data, vehicle angle data, and vehicle speed. data, vehicle acceleration data, vehicle inclination data, vehicle forward/reverse data, vehicle weight data, battery data, fuel data, tire pressure data, vehicle interior temperature data, vehicle interior humidity data, steering wheel rotation angle data, vehicle exterior illumination Data, pressure data applied to the accelerator pedal, pressure data applied to the brake pedal, and the like may be generated.

9) Location data generating device

The location data generating device 280 may generate location data of the vehicle 10 . The location data generating apparatus 280 may include at least one of a Global Positioning System (GPS) and a Differential Global Positioning System (DGPS). The location data generating apparatus 280 may generate location data of the vehicle 10 based on a signal generated from at least one of GPS and DGPS. According to an embodiment, the location data generating apparatus 280 may correct the location data based on at least one of an Inertial Measurement Unit (IMU) of the sensing unit 270 and a camera of the object detecting apparatus 210 . The location data generating device 280 may be referred to as a Global Navigation Satellite System (GNSS).

The vehicle 10 may include an internal communication system 50 . A plurality of electronic devices included in the vehicle 10 may exchange signals via the internal communication system 50 . Signals may contain data. The internal communication system 50 may use at least one communication protocol (eg, CAN, LIN, FlexRay, MOST, Ethernet).

(3) Components of an autonomous driving device

7 is a control block diagram of an autonomous driving apparatus according to an embodiment of the present specification.

Referring to FIG. 7 , the autonomous driving device 260 may include a memory 140 , a processor 170 , an interface unit 180 , and a power supply unit 190 .

The memory 140 is electrically connected to the processor 170 . The memory 140 may store basic data for the unit, control data for operation control of the unit, and input/output data. The memory 140 may store data processed by the processor 170 . The memory 140 may be configured as at least one of ROM, RAM, EPROM, flash drive, and hard drive in terms of hardware. The memory 140 may store various data for the overall operation of the autonomous driving device 260 , such as a program for processing or controlling the processor 170 . The memory 140 may be implemented integrally with the processor 170 . According to an embodiment, the memory 140 may be classified into sub-configurations of the processor 170 .

The interface unit 180 may exchange signals with at least one electronic device provided in the vehicle 10 in a wired or wireless manner. The interface unit 280 includes an object detecting device 210 , a communication device 220 , a driving manipulation device 230 , a main ECU 240 , a driving control device 250 , a sensing unit 270 , and a location data generating device. A signal may be exchanged with at least one of 280 by wire or wirelessly. The interface unit 280 may be composed of at least one of a communication module, a terminal, a pin, a cable, a port, a circuit, an element, and a device.

The power supply unit 190 may supply power to the autonomous driving device 260 . The power supply unit 190 may receive power from a power source (eg, a battery) included in the vehicle 10 and supply power to each unit of the autonomous driving apparatus 260 . The power supply unit 190 may be operated according to a control signal provided from the main ECU 240 . The power supply unit 190 may include a switched-mode power supply (SMPS).

The processor 170 may be electrically connected to the memory 140 , the interface unit 280 , and the power supply unit 190 to exchange signals. Processor 170, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors (processors), controller It may be implemented using at least one of controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

The processor 170 may be driven by power provided from the power supply 190 . The processor 170 may receive data, process data, generate a signal, and provide a signal while power is supplied by the power supply unit 190 .

The processor 170 may receive information from another electronic device in the vehicle 10 through the interface unit 180 . The processor 170 may provide a control signal to another electronic device in the vehicle 10 through the interface unit 180 .

The autonomous driving device 260 may include at least one printed circuit board (PCB). The memory 140 , the interface unit 180 , the power supply unit 190 , and the processor 170 may be electrically connected to the printed circuit board.

(4) Operation of autonomous driving device

8 is a signal flow diagram of an autonomous vehicle according to an embodiment of the present specification.

1) Receive operation

Referring to FIG. 8 , the processor 170 may perform a reception operation. The processor 170 may receive data from at least one of the object detecting device 210 , the communication device 220 , the sensing unit 270 , and the location data generating device 280 through the interface unit 180 . can The processor 170 may receive object data from the object detection apparatus 210 . The processor 170 may receive HD map data from the communication device 220 . The processor 170 may receive vehicle state data from the sensing unit 270 . The processor 170 may receive location data from the location data generating device 280 .

2) processing/judgment action

The processor 170 may perform a processing/determination operation. The processor 170 may perform a processing/determination operation based on the driving situation information. The processor 170 may perform a processing/determination operation based on at least one of object data, HD map data, vehicle state data, and location data.

2.1) Driving plan data generation operation

The processor 170 may generate driving plan data. For example, the processor 1700 may generate Electronic Horizon Data, which is understood as driving plan data within a range from a point where the vehicle 10 is located to a horizon. Horizon may be understood as a point in front of a preset distance from a point where the vehicle 10 is located based on a preset driving route The horizon is a point where the vehicle 10 is located along a preset driving route. It may mean a point to which the vehicle 10 can reach after a predetermined time from the point.

The electronic horizon data may include horizon map data and horizon pass data.

2.1.1) Horizon Map Data

The horizon map data may include at least one of topology data, road data, HD map data, and dynamic data. According to an embodiment, the horizon map data may include a plurality of layers. For example, the horizon map data may include a first layer matching topology data, a second layer matching road data, a third layer matching HD map data, and a fourth layer matching dynamic data. The horizon map data may further include static object data.

Topology data can be described as a map created by connecting road centers. The topology data is suitable for roughly indicating the location of the vehicle, and may be in the form of data mainly used in navigation for drivers. The topology data may be understood as data on road information excluding information on lanes. The topology data may be generated based on data received from an external server through the communication device 220 . The topology data may be based on data stored in at least one memory provided in the vehicle 10 .

The road data may include at least one of slope data of the road, curvature data of the road, and speed limit data of the road. The road data may further include data on an overtaking prohibited section. The road data may be based on data received from an external server through the communication device 220 . The road data may be based on data generated by the object detecting apparatus 210 .

HD map data includes detailed lane-by-lane topology information of the road, connection information of each lane, and characteristic information for vehicle localization (eg, traffic signs, Lane Marking/attributes, Road furniture, etc.). can The HD map data may be based on data received from an external server through the communication device 220 .

The dynamic data may include various dynamic information that may be generated on the road. For example, the dynamic data may include construction information, variable speed lane information, road surface condition information, traffic information, moving object information, and the like. The dynamic data may be based on data received from an external server through the communication device 220 . The dynamic data may be based on data generated by the object detection apparatus 210 .

The processor 170 may provide map data within a range from the point where the vehicle 10 is located to the horizon.

2.1.2) Horizon Pass Data

The horizon pass data may be described as a trajectory that the vehicle 10 can take within a range from a point where the vehicle 10 is located to the horizon. The horizon pass data may include data representing a relative probability of selecting any one road at a decision point (eg, a fork, a junction, an intersection, etc.). The relative probability may be calculated based on the time it takes to arrive at the final destination. For example, at the decision point, if the time taken to arrive at the final destination is shorter when selecting the first road than when selecting the second road, the probability of selecting the first road is higher than the probability of selecting the second road. can be calculated higher.

The horizon pass data may include a main path and a sub path. The main path may be understood as a track connecting roads with a high relative probability of being selected. The sub-path may diverge at at least one decision point on the main path. The sub-path may be understood as a trajectory connecting at least one road having a low relative probability of being selected from at least one decision point on the main path.

3) Control signal generation operation

The processor 170 may perform a control signal generating operation. The processor 170 may generate a control signal based on the Electronic Horizon data. For example, the processor 170 may generate at least one of a powertrain control signal, a brake device control signal, and a steering device control signal based on the electronic horizon data.

The processor 170 may transmit the generated control signal to the driving control device 250 through the interface unit 180 . The drive control device 250 may transmit a control signal to at least one of the power train 251 , the brake device 252 , and the steering device 253 .

cabin

9 is a view illustrating the interior of a vehicle according to an embodiment of the present specification. 10 is a block diagram referenced to describe a vehicle cabin system according to an embodiment of the present specification.

(1) Components of the cabin

9 to 10 , the vehicle cabin system 300 (hereinafter, the cabin system) may be defined as a convenience system for a user using the vehicle 10 . The cabin system 300 may be described as a top-level system including a display system 350 , a cargo system 355 , a seat system 360 , and a payment system 365 . The cabin system 300 includes a main controller 370 , a memory 340 , an interface unit 380 , a power supply unit 390 , an input device 310 , an imaging device 320 , a communication device 330 , and a display system. 350 , a cargo system 355 , a seat system 360 , and a payment system 365 . Depending on the embodiment, the cabin system 300 may further include other components in addition to the components described herein, or may not include some of the components described herein.

1) Main controller

The main controller 370 is electrically connected to the input device 310 , the communication device 330 , the display system 350 , the cargo system 355 , the seat system 360 , and the payment system 365 to exchange signals. can do. The main controller 370 may control the input device 310 , the communication device 330 , the display system 350 , the cargo system 355 , the seat system 360 , and the payment system 365 . The main controller 370, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors (processors), It may be implemented using at least one of controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

The main controller 370 may include at least one sub-controller. According to an embodiment, the main controller 370 may include a plurality of sub-controllers. Each of the plurality of sub-controllers may individually control devices and systems included in the grouped cabin system 300 . Devices and systems included in the cabin system 300 may be grouped by function or grouped based on a seatable seat.

The main controller 370 may include at least one processor 371 . Although the main controller 370 is illustrated as including one processor 371 in FIG. 6 , the main controller 371 may include a plurality of processors. The processor 371 may be classified as any one of the sub-controllers described above.

The processor 371 may receive a signal, information, or data from the user terminal through the communication device 330 . The user terminal may transmit a signal, information or data to the cabin system 300 .

The processor 371 may specify a user based on image data received from at least one of an internal camera and an external camera included in the imaging device. The processor 371 may specify a user by applying an image processing algorithm to image data. For example, the processor 371 may specify the user by comparing the information received from the user terminal with the image data. For example, the information may include at least one of route information, body information, passenger information, luggage information, location information, preferred content information, preferred food information, disability information, and use history information of the user. .

The main controller 370 may include an artificial intelligence agent 372 . The artificial intelligence agent 372 may perform machine learning based on data obtained through the input device 310 . The artificial intelligence agent 372 may control at least one of the display system 350 , the cargo system 355 , the seat system 360 , and the payment system 365 based on the machine learning result.

2) Essential components

The memory 340 is electrically connected to the main controller 370 . The memory 340 may store basic data for the unit, control data for operation control of the unit, and input/output data. The memory 340 may store data processed by the main controller 370 . The memory 340 may be configured as at least one of ROM, RAM, EPROM, flash drive, and hard drive in terms of hardware. The memory 340 may store various data for the overall operation of the cabin system 300 , such as a program for processing or controlling the main controller 370 . The memory 340 may be implemented integrally with the main controller 370 .

The interface unit 380 may exchange signals with at least one electronic device provided in the vehicle 10 in a wired or wireless manner. The interface unit 380 may be configured with at least one of a communication module, a terminal, a pin, a cable, a port, a circuit, an element, and a device.

The power supply unit 390 may supply power to the cabin system 300 . The power supply unit 390 may receive power from a power source (eg, a battery) included in the vehicle 10 to supply power to each unit of the cabin system 300 . The power supply 390 may be operated according to a control signal provided from the main controller 370 . For example, the power supply unit 390 may be implemented as a switched-mode power supply (SMPS).

The cabin system 300 may include at least one printed circuit board (PCB). The main controller 370 , the memory 340 , the interface unit 380 , and the power supply unit 390 may be mounted on at least one printed circuit board.

3) input device

The input device 310 may receive a user input. The input device 310 may convert a user input into an electrical signal. The electrical signal converted by the input device 310 may be converted into a control signal and provided to at least one of the display system 350 , the cargo system 355 , the seat system 360 , and the payment system 365 . At least one processor included in the main controller 370 or the cabin system 300 may generate a control signal based on an electrical signal received from the input device 310 .

The input device 310 may include at least one of a touch input unit, a gesture input unit, a mechanical input unit, and a voice input unit. The touch input unit may convert the user's touch input into an electrical signal. The touch input unit may include at least one touch sensor to detect a user's touch input. According to an embodiment, the touch input unit may be formed integrally with at least one display included in the display system 350 to implement a touch screen. Such a touch screen may provide both an input interface and an output interface between the cabin system 300 and a user. The gesture input unit may convert the user's gesture input into an electrical signal. The gesture input unit may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input. According to an embodiment, the gesture input unit may detect a user's 3D gesture input. To this end, the gesture input unit may include a light output unit that outputs a plurality of infrared rays or a plurality of image sensors. The gesture input unit may sense the user's 3D gesture input through a time of flight (TOF) method, a structured light method, or a disparity method. The mechanical input unit may convert a user's physical input (eg, pressing or rotating) through the mechanical device into an electrical signal. The mechanical input unit may include at least one of a button, a dome switch, a jog wheel, and a jog switch. Meanwhile, the gesture input unit and the mechanical input unit may be integrally formed. For example, the input device 310 includes a gesture sensor and may include a jog dial device that is formed so as to be able to deposit and withdraw from a part of a surrounding structure (eg, at least one of a seat, an armrest, and a door). . When the jog dial device is in a flat state with the surrounding structures, the jog dial device may function as a gesture input unit. When the jog dial device protrudes relative to the surrounding structure, the jog dial device may function as a mechanical input unit. The voice input unit may convert the user's voice input into an electrical signal. The voice input unit may include at least one microphone. The voice input unit may include a beam foaming MIC.

4) video device

The imaging device 320 may include at least one camera. The imaging device 320 may include at least one of an internal camera and an external camera. The internal camera may capture an image in the cabin. The external camera may capture an image outside the vehicle. The internal camera may acquire an image in the cabin. The imaging device 320 may include at least one internal camera. It is preferable that the imaging device 320 includes the number of cameras corresponding to the number of people who can board. The imaging device 320 may provide an image acquired by an internal camera. At least one processor included in the main controller 370 or the cabin system 300 detects the user's motion based on the image acquired by the internal camera, and generates a signal based on the detected motion, the display system 350 , it may be provided to at least one of the cargo system 355 , the seat system 360 , and the payment system 365 . The external camera may acquire an image outside the vehicle. The imaging device 320 may include at least one external camera. The imaging device 320 preferably includes the number of cameras corresponding to the boarding door. The imaging device 320 may provide an image acquired by an external camera. At least one processor included in the main controller 370 or the cabin system 300 may acquire user information based on an image acquired by an external camera. At least one processor included in the main controller 370 or the cabin system 300, based on user information, authenticates the user, the user's body information (eg, height information, weight information, etc.), the user's Passenger information, luggage information of the user, and the like may be acquired.

5) communication device

The communication device 330 may wirelessly exchange signals with an external device. The communication apparatus 330 may exchange a signal with an external device through a network or may directly exchange a signal with an external device. The external device may include at least one of a server, a mobile terminal, and another vehicle. The communication device 330 may exchange signals with at least one user terminal. The communication device 330 may include at least one of an antenna, a radio frequency (RF) circuit capable of implementing at least one communication protocol, and an RF device to perform communication. According to an embodiment, the communication device 330 may use a plurality of communication protocols. The communication device 330 may switch a communication protocol according to a distance from the mobile terminal.

For example, the communication apparatus may exchange a signal with an external device based on C-V2X (Cellular V2X) technology. For example, the C-V2X technology may include LTE-based sidelink communication and/or NR-based sidelink communication. Contents related to C-V2X will be described later.

For example, communication devices communicate with external devices based on IEEE 802.11p PHY/MAC layer technology and IEEE 1609 Network/Transport layer technology-based Dedicated Short Range Communications (DSRC) technology or WAVE (Wireless Access in Vehicular Environment) standard. can be exchanged for DSRC (or WAVE standard) technology is a communication standard prepared to provide ITS (Intelligent Transport System) service through short-distance dedicated communication between in-vehicle devices or between roadside devices and in-vehicle devices. The DSRC technology may use a frequency of 5.9 GHz and may be a communication method having a data transmission rate of 3 Mbps to 27 Mbps. IEEE 802.11p technology can be combined with IEEE 1609 technology to support DSRC technology (or WAVE standard).

The communication apparatus of the present specification may exchange a signal with an external device using either one of the C-V2X technology or the DSRC technology. Alternatively, the communication apparatus of the present specification may exchange signals with an external device by hybridizing C-V2X technology and DSRC technology.

6) Display system

The display system 350 may display a graphic object. The display system 350 may include at least one display device. For example, the display system 350 may include a publicly available first display device 410 and an individually available second display device 420 .

6.1) Common Display Devices

The first display device 410 may include at least one display 411 for outputting visual content. The display 411 included in the first display device 410 is a flat panel display. It may be implemented as at least one of a curved display, a rollable display, and a flexible display. For example, the first display device 410 may include a first display 411 positioned at the rear of the seat and formed to be able to be put in and out of the cabin and a first mechanism for moving the first display 411 . The first display 411 may be disposed in a slot formed in the seat main frame so as to be retractable. According to an embodiment, the first display device 410 may further include a flexible area control mechanism. The first display may be formed to be flexible, and the flexible area of the first display may be adjusted according to the location of the user. For example, the first display device 410 may include a second display positioned on the ceiling of the cabin and formed to be rollable and a second mechanism for winding or unwinding the second display. The second display may be formed to enable screen output on both sides. For example, the first display device 410 may include a third display positioned on the ceiling in the cabin and formed to be flexible and a third mechanism for bending or unfolding the third display. According to an embodiment, the display system 350 may further include at least one processor that provides a control signal to at least one of the first display device 410 and the second display device 420 . The processor included in the display system 350 may generate a control signal based on a signal received from at least one of the main controller 370 , the input device 310 , the imaging device 320 , and the communication device 330 . can

The display area of the display included in the first display device 410 may be divided into a first area 411a and a second area 411b. The first area 411a may be defined as a content display area. For example, the first area 411 may display at least one of entertainment contents (eg, movies, sports, shopping, music, etc.), video conferences, food menus, and graphic objects corresponding to the augmented reality screen. can The first area 411a may display a graphic object corresponding to driving situation information of the vehicle 10 . The driving situation information may include at least one of object information outside the vehicle, navigation information, and vehicle state information. The object information outside the vehicle may include information on the existence of an object, location information of the object, distance information between the vehicle 300 and the object, and relative speed information between the vehicle 300 and the object. The navigation information may include at least one of map information, set destination information, route information according to the destination setting, information on various objects on a route, lane information, and current location information of the vehicle. The vehicle state information may include vehicle attitude information, vehicle speed information, vehicle inclination information, vehicle weight information, vehicle direction information, vehicle battery information, vehicle fuel information, vehicle tire pressure information, and vehicle steering information. , vehicle interior temperature information, vehicle interior humidity information, pedal position information, vehicle engine temperature information, and the like. The second area 411b may be defined as a user interface area. For example, the second area 411b may output an artificial intelligence agent screen. According to an exemplary embodiment, the second area 411b may be located in an area divided by the seat frame. In this case, the user may look at the content displayed in the second area 411b between the plurality of sheets. According to an embodiment, the first display device 410 may provide holographic content. For example, the first display device 410 may provide holographic content for a plurality of users so that only the user who requested the content can view the corresponding content.

6.2) Personal Display Devices

The second display device 420 may include at least one display 421 . The second display device 420 may provide the display 421 at a position where only individual passengers can check the display contents. For example, the display 421 may be disposed on an arm rest of the seat. The second display device 420 may display a graphic object corresponding to the user's personal information. The second display device 420 may include the number of displays 421 corresponding to the number of passengers available to board. The second display device 420 may implement a touch screen by forming a layer structure with each other or integrally formed with the touch sensor. The second display device 420 may display a graphic object for receiving a user input of seat adjustment or room temperature adjustment.

7) Cargo system

The cargo system 355 may provide a product to the user according to the user's request. The cargo system 355 may be operated based on an electrical signal generated by the input device 310 or the communication device 330 . The cargo system 355 may include a cargo box. The cargo box may be hidden in a portion of the bottom of the seat in a state in which the goods are loaded. When an electrical signal based on a user input is received, the cargo box may be exposed as a cabin. A user may select a necessary product from among the items loaded in the exposed cargo box. The cargo system 355 may include a sliding moving mechanism and a product pop-up mechanism for exposure of the cargo box according to a user input. The cargo system 355 may include a plurality of cargo boxes to provide various types of goods. In the cargo box, a weight sensor for determining whether to provide each product may be built-in.

8) Seat system

The seat system 360 may provide the user with a seat tailored to the user. The seat system 360 may be operated based on an electrical signal generated by the input device 310 or the communication device 330 . The seat system 360 may adjust at least one element of the seat based on the obtained user body data. The seat system 360 may include a user detection sensor (eg, a pressure sensor) for determining whether the user is seated. The seat system 360 may include a plurality of seats on which a plurality of users can each be seated. Any one of the plurality of sheets may be disposed to face at least another one. At least two users inside the cabin can sit facing each other.

9) Payment system

The payment system 365 may provide a payment service to the user. The payment system 365 may operate based on an electrical signal generated by the input device 310 or the communication device 330 . The payment system 365 may calculate a price for at least one service used by the user and request that the calculated price be paid.

(2) Scenarios for using autonomous vehicles

11 is a diagram referenced to describe a user's usage scenario according to an embodiment of the present specification.

1) Destination prediction scenario

The first scenario S111 is a user's destination prediction scenario. The user terminal may install an application capable of interworking with the cabin system 300 . The user terminal may predict the destination of the user based on the user's contextual information through the application. The user terminal may provide vacancy information in the cabin through the application.

2) Cabin interior layout preparation scenario

The second scenario S112 is a cabin interior layout preparation scenario. The cabin system 300 may further include a scanning device for acquiring data about a user located outside the vehicle 300 . The scanning device may scan the user to obtain body data and baggage data of the user. The user's body data and baggage data may be used to set the layout. The user's body data may be used for user authentication. The scanning device may include at least one image sensor. The image sensor may acquire a user image by using light in a visible light band or an infrared band.

The seat system 360 may set a layout in the cabin based on at least one of the user's body data and baggage data. For example, the seat system 360 may provide a space for loading luggage or a space for installing a car seat.

3) User welcome scenario

The third scenario S113 is a user welcome scenario. The cabin system 300 may further include at least one guide light. The guide light may be disposed on the floor in the cabin. When the boarding of the user is detected, the cabin system 300 may output a guide light to allow the user to sit on a preset seat among a plurality of seats. For example, the main controller 370 may implement a moving light by sequentially lighting a plurality of light sources according to time from an opened door to a preset user seat.

4) Seat adjustment service scenario

The fourth scenario S114 is a seat adjustment service scenario. The seat system 360 may adjust at least one element of the seat matching the user based on the obtained body information.

5) Scenarios for providing personal content

The fifth scenario S115 is a personal content provision scenario. The display system 350 may receive user personal data through the input device 310 or the communication device 330 . The display system 350 may provide content corresponding to the user's personal data.

6) Product offering scenario

A sixth scenario S116 is a product provision scenario. The cargo system 355 may receive user data through the input device 310 or the communication device 330 . The user data may include user's preference data and user's destination data. Cargo system 355, based on the user data, may provide a product.

7) Payment Scenario

The seventh scenario S117 is a payment scenario. The payment system 365 may receive data for price calculation from at least one of the input device 310 , the communication device 330 , and the cargo system 355 . The payment system 365 may calculate the user's vehicle usage price based on the received data. The payment system 365 may request payment of a fee from the user (eg, the user's mobile terminal) at the calculated price.

8) User's Display System Control Scenario

The eighth scenario S118 is a user's display system control scenario. The input device 310 may receive a user input in at least one form and convert it into an electrical signal. The display system 350 may control displayed content based on the electrical signal.

9) AI Agent Scenario

The ninth scenario S119 is a multi-channel artificial intelligence (AI) agent scenario for a plurality of users. The artificial intelligence agent 372 may classify a user input for each of a plurality of users. The artificial intelligence agent 372 is, based on the electrical signal converted by the plurality of user individual user inputs, at least one of the display system 350 , the cargo system 355 , the seat system 360 , and the payment system 365 . can control

10) Multimedia content provision scenario for multiple users

The tenth scenario S120 is a multimedia content provision scenario targeting a plurality of users. The display system 350 may provide content that all users can view together. In this case, the display system 350 may individually provide the same sound to a plurality of users through speakers provided for each sheet. The display system 350 may provide content that can be individually viewed by a plurality of users. In this case, the display system 350 may provide individual sound through a speaker provided for each sheet.

11) Scenarios for ensuring user safety

The eleventh scenario S121 is a user safety securing scenario. When acquiring information about objects around the vehicle that threatens the user, the main controller 370 may control an alarm for objects around the vehicle to be output through the display system 350 .

12) Loss Prevention Scenario

A twelfth scenario ( S122 ) is a scenario for preventing loss of a user's belongings. The main controller 370 may obtain data about the user's belongings through the input device 310 . The main controller 370 may acquire the user's movement data through the input device 310 . The main controller 370 may determine whether the user leaves the belongings and alights based on the movement data and the data on the belongings. The main controller 370 may control an alarm related to belongings to be output through the display system 350 .

13) Drop-off report scenario

The thirteenth scenario S123 is a drop off report scenario. The main controller 370 may receive the user's getting off data through the input device 310 . After the user gets off, the main controller 370 may provide report data according to getting off to the user's mobile terminal through the communication device 330 . The report data may include vehicle 10 total usage fee data.

H. Speech Recognition System and AI Processing

12 illustrates a block diagram of a schematic system in which a speech recognition method according to an embodiment of the present invention is implemented.

Referring to FIG. 12 , the system in which the speech recognition method is implemented includes a speech recognition device 10 , a network system 16 , and a text-to-speech (TTS) system 18 as a speech synthesis engine. may include

The at least one voice recognition device 10 may include a mobile phone 11 , a PC 12 , a notebook computer 13 and other server devices 14 , and the vehicle of FIG. 5 . The PC 12 and the notebook computer 13 may be connected to at least one network system 16 through a wireless access point 15 . According to an embodiment of the present invention, the voice recognition apparatus 10 may include an audio book and a smart speaker.

Meanwhile, the TTS system 18 may be implemented in a server included in a network, or may be implemented as on-device processing and embedded in the voice recognition device 10 . In an embodiment of the present invention, it is assumed that the TTS system 18 is built-in and implemented in the voice recognition device 10 .

Also, the voice recognition system of FIG. 12 may be included in the vehicle 10 of FIG. 5 . That is, the voice recognition system of FIG. 12 includes the user interface device 200 of FIG. 6 , the processor 170 and/or interface 180 of FIG. 7 , the input device 310 of FIG. 10 , the main controller 370 and/or Alternatively, it may be included in at least one component of the interface unit 380 .

Hereinafter, a voice processing process (speech recognition and voice output (TTS) process) performed in a device environment and/or a cloud environment or server environment will be described with reference to FIGS. 13 and 14 . 13 and 14 , the device environments 50 and 70 may be referred to as client devices, and the cloud environments 60 and 80 may be referred to as servers. 13 illustrates an example in which the voice input may be performed in the device 50 , but the process of processing the input voice to synthesize the voice, that is, the overall operation of the voice processing is performed in the cloud environment 60 . In contrast, FIG. 14 illustrates an example of on-device processing in which the device 70 performs the overall operation of voice processing for synthesizing voice by processing the above-described input voice.

13 is a schematic block diagram of a voice recognition apparatus in a voice recognition system environment according to an embodiment of the present invention.

Various components are required to process voice events in an end-to-end voice UI environment. Sequences that process speech events are performed by collecting speech signals (Signal acquisition and playback), speech pre-processing, speech activation (Voice Activation), speech recognition (Speech Recognition), natural language understanding (Natural Language Processing) and Finally, the device performs a speech synthesis process that responds to the user.

The client device 50 may include an input module. The input module may receive a user input from a user. For example, the input module may receive a user input from a connected external device (eg, a keyboard or a headset). Also for example, the input module may include a touch screen. Also, for example, the input module may include a hardware key located in the user terminal.

According to an embodiment, the input module may include at least one microphone capable of receiving the user's utterance as a voice signal. The input module may include a speech input system, and may receive the user's speech as a voice signal through the speech input system. The at least one microphone may determine a digital input signal for the user's utterance by generating an input signal for an audio input. According to an embodiment, a plurality of microphones may be implemented as an array. The array may be arranged in a geometric pattern, eg, a linear geometry, a circular geometry, or any other configuration. For example, for a given point, an array of four sensors may be arranged in a circular pattern separated by 90 degrees to receive sound from four directions. In some implementations, the microphone may include a spatially different array of sensors in data communication, including a networked array of sensors. The microphone may include an omnidirectional type, a directional type (eg, a shotgun microphone), and the like.

The client device 50 may include a pre-processing module 51 capable of pre-processing a user input (voice signal) received through the input module (eg, a microphone).

The preprocessing module 51 may include an adaptive echo canceller (AEC) function, thereby canceling an echo included in a user input (voice signal) input through the microphone. The pre-processing module 51 may include a noise suppression (NS) function, thereby removing background noise included in the user input. The pre-processing module 51 may include an end-point detect (EPD) function, so as to detect the end point of the user's voice and find a portion in which the user's voice exists. In addition, the pre-processing module 51 may include an automatic gain control (AGC) function, so that the volume of the user input may be adjusted to be suitable for recognizing and processing the user input.

The client device 50 may include a voice activation module 52 . The voice recognition activation module 52 may recognize a wake up command for recognizing a user's call (eg, wake-up word). The voice recognition activation module 52 may detect a predetermined keyword (eg, Hi LG) from a user input that has undergone a pre-processing process. The voice recognition activation module 52 may exist in a standby state to perform an Always-on keyword detection function.

The client device 50 may transmit the user's voice input to the cloud server. Automatic speech recognition (ASR), natural language understanding (NLU) operations, which are key components for processing user speech, are typically, but not necessarily limited to, traditionally run in the cloud due to computing, storage, power constraints, etc. , may be done within the client device 50 .

The cloud may include a cloud device 60 that processes a user input transmitted from a client. The cloud device 60 may exist in the form of a server.

The cloud device 60 includes an Auto Speech Recognition (ASR) module 61, an Intelligent Processor (Artificial Intelligent Agent) 62, a Natural Language Understanding (NLU) module 63, a text-to-speech ( A Text-to-Speech, TTS) module 64 and a service manager 65 may be included.

The ASR module 61 may convert the user voice input received from the client device 50 into text data.

The ASR module 61 includes a front-end speech pre-processor. A front-end speech preprocessor extracts representative features from the speech input. For example, a front-end speech preprocessor performs a Fourier transform on the speech input to extract spectral features characterizing the speech input as a sequence of representative multidimensional vectors. Further, the ASR module 61 may include one or more speech recognition models (eg, acoustic models and/or language models) and implement one or more speech recognition engines. Examples of speech recognition models include hidden Markov models, Gaussian-Mixture Models, Deep Neural Network Models, n-gram language models, and other statistical models. Examples of speech recognition engines include dynamic time warp based engines and weighted finite state transformers (WFST) based engines. The one or more speech recognition models and the one or more speech recognition engines provide intermediate recognition results (eg, phonemes, phonemic strings, and subwords), and ultimately text recognition results (eg, words, word strings, or tokens). sequence) can be used to process the extracted representative features of the front-end speech preprocessor.

When the ASR module 61 generates a recognition result comprising a text string (eg, words, or a sequence of words, or a sequence of tokens), the recognition result is sent to a natural language processing module (NLU) (NLU) for intent inference. 63). In some examples, the ASR module 61 generates multiple candidate textual representations of the speech input. Each candidate text representation is a sequence of words or tokens corresponding to speech input.

The NLU module 63 may determine the user's intention by performing syntactic analysis or semantic analysis. The grammatical analysis may divide grammatical units (eg, words, phrases, morphemes, etc.) and determine which grammatical elements the divided units have. The semantic analysis may be performed using semantic matching, rule matching, formula matching, and the like. Accordingly, the NUL module 63 may obtain a certain domain, an intent, or a parameter necessary for expressing the intent of the user input.

The NLU module 63 may determine the user's intention and parameters by using a mapping rule divided into a domain, an intention, and a parameter necessary to identify the intention. For example, one domain (eg, alarm) may include multiple intents (eg, set alarm, clear alarm), and one intent may include multiple parameters (eg, time, repetition). number of times, alarm sound, etc.). The plurality of rules may include, for example, one or more essential element parameters. The matching rule may be stored in a Natural Language Understanding Database.

The NLU module 63 identifies the meaning of a word extracted from a user input using linguistic features (eg, grammatical elements) such as morphemes and phrases, and matches the meaning of the identified word to a domain and intent to determine the user's intentions.

For example, the NLU module 63 may determine the user intent by calculating how many words extracted from the user input are included in each domain and intent. According to an embodiment, the NLU module 63 may determine the parameter of the user input by using the word based on the understanding of the intention.

According to an embodiment, the NLU module 63 may determine the user's intention by using a natural language recognition database in which linguistic features for recognizing the intention of the user input are stored.

Also, according to an embodiment, the NLU module 63 may determine the user's intention by using a personal language model (PLM). For example, the NLU module 63 may determine the user's intention by using personalized information (eg, contact list, music list, schedule information, social network information, etc.).

The personalized language model may be stored in, for example, a natural language recognition database. According to an embodiment, not only the NLU module 63 but also the ASR module 61 may recognize the user's voice by referring to the personalized language model stored in the natural language recognition database.

The NLU module 63 may further include a natural language generation module (not shown). The natural language generating module may change specified information into text form. The information changed to the text form may be in the form of natural language utterance. The specified information may include, for example, information on additional input, information guiding completion of an operation corresponding to the user input, or information guiding the additional input of the user. The information changed in the text form may be transmitted to the client device and displayed on the display, or transmitted to the TTS module and changed to the voice form.

The speech synthesis module (TTS module) 64 may change information in a text format into information in a speech format. The TTS module 64 may receive text-type information from the natural language generating module of the NLU module 63 , convert the text-type information into voice-type information, and transmit it to the client device 50 . The client device 50 may output the information in the form of voice through a speaker.

The speech synthesis module 64 synthesizes the speech output based on the provided text. For example, the result generated by the speech recognition module (ASR) 61 is in the form of a text string. The speech synthesis module 64 converts the text string into an audible speech output. Speech synthesis module 64 uses any suitable speech synthesis technique to generate speech output from text, including concatenative synthesis, unit selection synthesis, diphone synthesis, domain- including, but not limited to, specific synthesis, formant synthesis, articulatory synthesis, hidden Markov model (HMM) based synthesis, and sinewave synthesis.

In some examples, speech synthesis module 64 is configured to synthesize individual words based on a phoneme string corresponding to the words. For example, a phonemic string is associated with a word in the generated text string. Phoneme strings are stored in metadata associated with the word. The speech synthesis module 64 is configured to directly process the phoneme string in the metadata to synthesize words in speech form.

Because cloud environments generally have more processing power or resources than client devices, it is possible to obtain higher-quality speech output in client-side synthesis. However, the present invention is not limited thereto, and it goes without saying that the voice synthesis process may be actually performed in the client device (see FIG. 14 ).

Meanwhile, according to an embodiment of the present invention, the cloud environment may further include an artificial intelligence processor (AI processor) 62 . The intelligent processor 62 may be designed to perform at least some of the functions performed by the above-described ASR module 61 , the NLU module 62 and/or the TTS module 64 . In addition, the intelligent processor module 62 may contribute to performing an independent function of each of the ASR module 61 , the NLU module 62 and/or the TTS module 64 .

The intelligent processor module 62 may perform the aforementioned functions through deep learning (deep learning). In the deep learning, when there is some data, it is represented in a form that a computer can understand (for example, in the case of an image, pixel information is expressed as a column vector), and many studies ( How to make better representation techniques and how to create models to learn them) are underway, and as a result of these efforts, deep neural networks (DNNs), convolutional neural networks (CNNs) are underway. ), Recurrent Boltzmann Machine (RNN), Restricted Boltzmann Machine (RBM), deep belief networks (DBN), and various deep learning techniques such as Deep Q-Network can be applied to fields such as computer vision, voice recognition, natural language processing, and voice/signal processing.

Currently, all major commercial speech recognition systems (MS Cortana, Skype Translator, Google Now, Apple Siri, etc.) are based on deep learning techniques.

In particular, the intelligent processor module 62 performs various natural language processing processes including automatic translation, emotion analysis, and information retrieval using a deep artificial neural network structure in the field of natural language processing. can

Meanwhile, the cloud environment may include a service manager 65 capable of supporting the function of the intelligent processor 62 by collecting various personalized information. The personalized information obtained through the service manager includes at least one data (calendar application, messaging service, music application usage, etc.) used by the client device 50 through the cloud environment, the client device 50 and/or the cloud. At least one sensing data collected by 60 (camera, microphone, temperature, humidity, gyro sensor, C-V2X, pulse, ambient light, iris scan, etc.), the client It may include off-device data that is not directly related to the device 50 . For example, the personalized information may include maps, SMS, News, Music, Stock, Weather, and Wikipedia information.

The intelligent processor 62 is expressed as a separate block to be distinguished from the ASR module 61, the NLU module 63, and the TTS module 64 for convenience of explanation, but the intelligent processor 62 includes each module ( 61, 62, 64) may perform at least some or all of the functions.

14 is a schematic block diagram of a voice recognition apparatus in a voice recognition system environment according to another embodiment of the present invention.

The client device 70 and the cloud environment 80 shown in FIG. 14 may correspond to the client device 50 and the cloud environment 60 mentioned in FIG. 13 with only differences in some configurations and functions. Accordingly, for specific functions of the corresponding blocks, reference may be made to FIG. 13 .

Referring to FIG. 14 , the client device 70 includes a preprocessing module 71 , a voice activation module 72 , an ASR module 73 , an intelligent processor 74 , an NLU module 75 , and a TTS module. (76) may be included. Also, the client device 70 may include an input module (at least one microphone) and at least one output module.

In addition, the cloud environment 80 may include cloud knowledge for storing personalized information in the form of knowledge.

The function of each module shown in FIG. 14 may refer to FIG. 13 . However, since the ASR module 73, the NLU module 75, and the TTS module 76 are included in the client device 70, communication with the cloud may not be required for voice processing such as voice recognition and voice synthesis. , thereby enabling an immediate and real-time voice processing operation.

Each of the modules shown in FIGS. 13 and 14 is only an example for explaining a voice processing process, and may have more or fewer modules than the modules shown in FIGS. 13 and 14 . It should also be noted that two or more modules may be combined or may have different modules or different arrangements of modules. The various modules illustrated in FIGS. 13 and 14 may be implemented in one or more signal processing and/or application specific integrated circuits, hardware, software instructions for execution by one or more processors, firmware, or combinations thereof.

15 is a schematic block diagram of an intelligent processor capable of implementing speech recognition according to an embodiment of the present invention.

Referring to FIG. 15 , the intelligent processor 74 may support an interactive operation with a user in addition to performing the ASR operation, the NLU operation, and the TTS operation in the voice processing process described with reference to FIGS. 13 and 14 . have. Alternatively, the intelligent processor 74 uses the context information to make the information included in the text representations received from the ASR module 61 by the NLU module 63 of FIG. 13 clearer, supplement or additionally define the information can contribute to carrying out

Here, the context information may include preferences of a client device user, hardware and/or software states of the client device, various sensor information collected before, during, or immediately after user input, previous interactions between the intelligent processor and the user. (eg, conversation), and the like. Of course, the context information in this document is a characteristic that is dynamic and varies according to time, location, content of conversation, and other factors.

The intelligent processor 74 may further include a context fusion and learning module 741 , local knowledge 742 , and dialog management 743 .

The context fusion and learning module 741 may learn the user's intention based on at least one piece of data. The at least one data may include at least one sensing data obtained from a client device or a cloud environment. In addition, the at least one data may include speaker identification, acoustic event detection, speaker's personal information (gender and age detection), and voice activity detection (VAD). , may include emotion information (Emotion Classification).

The speaker identification may mean specifying a person speaking in a conversation group registered by voice. The speaker identification may include identifying a previously registered speaker or registering as a new speaker. Acoustic event detection can recognize the type of sound and the location of the sound by recognizing the sound itself beyond the speech recognition technology. Voice activity detection (VAD) is a speech processing technique in which the presence or absence of human speech (voice) is detected in an audio signal, which may include music, noise, or other sounds. According to an example, the intelligent processor 74 may check whether speech is present from the input audio signal. According to an example, the intelligent processor 74 may distinguish speech data from non-speech data using a deep neural network (DNN) model. In addition, the intelligent processor 74 may perform an emotion classification operation on the speech data using a deep neural network (DNN) model. According to the emotion classification operation, speech data may be classified into anger, boredom, fear, happiness, and sadness.

The context fusion and learning module 741 may include a DNN model to perform the above-described operation, and may confirm the intention of a user input based on the DNN model and sensing information collected from a client device or cloud environment. .

It goes without saying that the at least one data is merely exemplary, and any data that can be referenced to confirm the user's intention in the voice processing process may be included. Of course, the at least one data may be obtained through the above-described DNN model.

The intelligent processor 74 may include Local Knowledge 742 . The local knowledge 742 may include user data. The user data may include a user's preference, a user address, a user's initial setting language, a user's contact list, and the like. According to an example, the intelligent processor 74 may additionally define user intention by supplementing information included in the user's voice input by using the user's specific information. For example, in response to a user's request "Please invite my friends to my birthday party," the intelligent processor 74 may configure the user to determine who the "Friends" are and when and where the "Birthday Party" will be held. The local knowledge 742 can be used without requiring the user to provide more specific information.

The intelligent processor 74 may further include a dialog management 743 . The intelligent processor 74 may provide a dialog interface to enable voice conversation with the user. The dialog interface may refer to a process of outputting a response to a user's voice input through a display or a speaker. Here, the final result output through the dialog interface may be based on the above-described ASR operation, NLU operation, and TTS operation.

16 shows an autonomous driving system 100 according to the present disclosure.

As shown in FIG. 16 , the autonomous driving system may include an autonomous driving vehicle 10 and a cloud server 30 performing wireless communication with the autonomous driving vehicle. Here, the autonomous driving vehicle 10 of FIG. 16 may be the vehicle 10 of FIG. 5 .

The autonomous vehicle 10 may include an input device 110 , an output device 120 , a processor 130 , and a communication unit 150 .

The input device 110 includes the user interface device 200 of FIG. 6 , the object detection device 210 , the driving manipulation device 230 , the sensing unit 270 , the location data generating device 280 , and the interface 180 of FIG. 7 . ), the object detection device 210 of FIG. 8 , the sensing unit 270 , the location data generating device 280 , and the input device 310 of FIG. 10 may include at least one component, and the user of FIG. 6 . The interface device 200 , the object detecting device 210 , the driving manipulation device 230 , the sensing unit 270 , the location data generating device 280 , the interface 180 of FIG. 7 , and the object detecting device 210 of FIG. 8 . ), the sensing unit 270 , the location data generating device 280 , and the input device 310 of FIG. 10 may perform the functions described above.

The output device 120 includes at least one of the user interface device 200 of FIG. 6 , the interface 180 of FIG. 7 , the imaging device 320 of FIG. 10 , the display system 350 , and the interface unit 380 of FIG. and performs the functions described above of the user interface device 200 of FIG. 6 , the interface 180 of FIG. 7 , the imaging device 320 of FIG. 10 , the display system 350 , and the interface unit 380 of FIG. can do.

The processor 130 of FIG. 16 is at least one of the autonomous driving device 260 of FIG. 6 , the main ECU 240 , the processor 170 of FIG. 7 , the processor 170 of FIG. 8 , and the main controller 370 of FIG. 10 . It may include one component, and may include the autonomous driving device 260 of FIG. 6 , the main ECU 240 , the processor 170 of FIG. 7 , the processor 170 of FIG. 8 , and the main controller 370 of FIG. 10 . of the above-described functions.

The communication unit 150 of FIG. 16 includes at least one of the communication device 220 of FIG. 6 , the interface 180 of FIG. 7 , the communication device 220 of FIG. 8 , and the communication device 330 of FIG. 10 . The above-described functions of the communication device 220 of FIG. 6 , the interface 180 of FIG. 7 , the communication device 220 of FIG. 8 , and the communication device 330 of FIG. 10 may be performed.

The cloud server 30 of FIG. 16 may include a communication unit 310 and a processor 320 .

The communication unit 310 may perform wired/wireless communication with the autonomous vehicle 10 . In more detail, the communication unit 310 may receive various types of data described below from the communication unit of the autonomous vehicle, and transmit the received data to the processor 320 of the cloud server. For example, the communication unit 310 may provide a user input (eg, voice input) from the autonomous driving vehicle, characteristic information related to the input device, characteristic information related to the output device, and state information related to the autonomous driving vehicle (eg, safety level, driver's health status, etc.).

The processor 320 may include a device environment setting unit 321 , an input/output analysis unit 322 , and a content generation unit 323 .

The device environment setting unit may generate device characteristic information including characteristic information related to an input device and characteristic information related to an output device transmitted from the communication unit 310 , and transmit the device characteristic information to the content generator.

The input/output analysis unit may generate input characteristic information based on the user's input transmitted from the communication unit and the state information of the autonomous vehicle, and may transmit the generated input characteristic information to the content generation unit. For example, the vehicle state information may include whether the driving mode of the vehicle is an autonomous driving mode or a manual driving mode.

The content generation unit may set the format (image, sound, etc.) of the content based on the input characteristic information transmitted from the input/output analysis unit and the device characteristic information transmitted from the device environment setting unit. In addition, the content generation unit outputs an output pattern for outputting content based on the input characteristic information transmitted from the input/output analysis unit and the device characteristic information transmitted from the device environment setting unit (eg, whether to continue the conversation long or short) , reconstructs a voice input based on the format of the generated output pattern and content, and transmits the reconstructed voice input to the autonomous vehicle through the communication unit.

17 is a flowchart illustrating a method for providing intelligent content in an autonomous vehicle according to an embodiment of the present specification.

As shown in FIG. 17 , first, the autonomous vehicle may obtain an input from an input device ( S1710 ). Here, the autonomous vehicle may acquire object information detected from the front of the vehicle, a voice input of a driver of the vehicle, sensing information acquired by the vehicle, and the like from the input device.

Subsequently, the autonomous vehicle may output the content related to the input based on the output environment information of the output device ( S1730 ). For example, the autonomous driving vehicle may reconfigure an input according to the current output environment of the vehicle based on the output environment information of the output device, generate content using the reconstructed input, and use the generated content to the output device. can be output through

Also, in reconstructing the input, the autonomous vehicle may reconstruct the input based on information related to characteristics of the input device as well as output environment information.

18 is a flowchart illustrating an intelligent content providing method of a vehicle (autonomous driving vehicle) and a server (cloud server) according to an embodiment of the present specification.

18 , first, the vehicle and the server may be connected through wireless communication (S1801). For example, the vehicle and the server may perform wireless communication based on the mMTC service described in E. mMTC above.

Subsequently, the server may share information related to an input device and/or an output device included in the vehicle with the vehicle (S1803). For example, the server may receive the characteristic information related to the input device 110 and the characteristic information related to the output device 120 of FIG. 16 from the vehicle through the communication unit.

Next, the server may set the format of the content to be output based on the vehicle input device information and the output device information (S1805).

Subsequently, the server may acquire state data of the vehicle (S1807).

For example, the state data of the vehicle may include data (information) related to the driving mode of the vehicle, data (information) related to the driver, and/or data (information) related to at least one component of a cabin inside the vehicle. .

Also, the server may obtain an input from an input device of the vehicle ( S1809 ).

Then, the server may determine the characteristics related to the input obtained from the vehicle (S1811).

Then, the server may generate content based on the input characteristics (S1813). For example, the server may generate content using input characteristics and a preset content format (S1813).

Finally, the server may output the content from the vehicle based on the output device information (S1815). For example, the server may output content in different formats for each type of output device in the vehicle.

19 illustrates an example of providing content based on a driver's voice input in an autonomous driving mode of a vehicle according to an embodiment of the present specification, and FIG. 20 is a diagram illustrating a driver's actions in an autonomous driving mode of a vehicle according to an embodiment of the present specification. An example of providing content based on a voice input and driver's gaze information is illustrated.

In FIG. 19 , a server (not shown) may receive characteristic information of an input device (microphone) of the vehicle and characteristic information of an output device (first display, second display, and third display) from the vehicle.

As shown in FIG. 19 , at least one input device (eg, a microphone) of the vehicle in the autonomous driving mode may acquire a user's voice input of "find a nearby restaurant". Upon obtaining the user's voice input, the vehicle may transmit the voice input to the server through wireless communication.

Upon receiving the user's voice input, the server executes each output device (first display, second display) based on characteristic information of each output device (first display, second display, and third display) and characteristic information of the input device (microphone). The format of content to be output to the second display and the third display) can be set.

For example, upon receiving the user's voice input, the server may set the format of the content to be displayed on the first display of the first size located in the driver's seat of the vehicle among the plurality of output devices in a graphic format (map). Also, the server may set the format of content to be displayed on the second display of the second size larger than the first size in the text format (restaurant list) while being located in the center of the front of the vehicle among the plurality of output devices. Also, among the plurality of output devices, the server may be set in a detailed UI format (restaurant detail screen) on a third display having a third size smaller than the second size while being positioned in the passenger seat of the vehicle.

Also, upon receiving the user's voice input, the server may set the format of content to be output through a speaker positioned to partially overlap the second display of the vehicle among the plurality of output devices as a voice language format.

Here, the server may obtain vehicle state information indicating that the driving mode of the vehicle is an autonomous driving mode from the vehicle, and may obtain a voice input of “find a nearby restaurant” from an input device (microphone) of the vehicle. Then, the server acquires the input characteristic that the input of "find a nearby restaurant" is a voice input, and based on the vehicle state information called 'autonomous driving mode', the server receives more UI objects than when the voice input is 'manual driving mode' The vehicle may be controlled to output the included content to a plurality of output devices. For example, the server may display a map screen on the first display, display a list of restaurants on the second display, based on the voice input of “find a restaurant nearby”, and display the restaurant details of the highest recommendation on the third display. While displaying the screen, it is possible to transmit a control command to the vehicle to output a voice through the speaker saying, "This is a list of nearby restaurants. According to 000's schedule, to an Italian restaurant with a rating of 4 or higher that can be reserved for two people..."

Meanwhile, unlike the case of FIG. 19 , in FIG. 20 , an example of providing content in consideration of not only the driver's voice input but also the driver's intention determined by the driver's eye tracking information will be described.

In detail, as shown in FIG. 20 , the server may receive, from the vehicle, gaze information obtained by tracking the movement of the driver's pupil together with the voice input "How's that restaurant" obtained from the vehicle's input device. .

Then, the server may determine the driver's intention based on the voice input and gaze information received from the vehicle, and display a map screen in which the predicted route is displayed on the first display based on the driver's intention and the voice input. In addition, the server may display the restaurant list in the driver's gaze direction on the second display based on the voice input and gaze information. Also, the server may display the detailed restaurant screen on the third display in the driver's gaze direction based on the voice input and gaze information. In addition, the server may output a voice saying "Do you want to make a reservation and guide you to the selected restaurant?..."

21 illustrates an example of providing content based on a driver's voice input in a manual driving mode of a vehicle according to an embodiment of the present specification, and FIG. 22 is a view illustrating an example of providing content in a manual driving mode of a vehicle according to an embodiment of the present specification. Another example of providing content based on the driver's voice input is shown.

In FIG. 21 , the server may receive characteristic information of the input device of the vehicle and the characteristic information of the output devices (the first display, the second display, and the third display) from the vehicle.

As shown in FIG. 19 , at least one input device of the vehicle in the manual driving mode may obtain a user's voice input of “find a nearby restaurant”. Upon obtaining the user's voice input, the vehicle may transmit the voice input to the server through wireless communication.

Upon receiving the user's voice input, the server may set the format of content to be output to each output device based on the characteristic information of each output device and the characteristic information of the input device.

For example, upon receiving the user's voice input, the server displays on the first display of the first size, the second display of the second size, and the third display of the third size located in the driver's seat of the vehicle among the plurality of output devices You can set the format of the content to be used. Also, the server may set a format of content to be output through a speaker of the vehicle among the plurality of output devices.

Here, the server may obtain vehicle state information indicating that the driving mode of the vehicle is a manual driving mode from the vehicle, and may obtain a voice input of “find a nearby restaurant” from an input device (microphone) of the vehicle. Then, the server acquires the input characteristic that the input of "find a nearby restaurant" is a voice input, and based on the vehicle state information of 'manual driving mode', the server receives fewer UI objects than when the voice input is 'autonomous driving mode' The vehicle may be controlled to output the included content to a plurality of output devices. For example, the server displays the speedometer screen on the first display based on the voice input "Find a nearby restaurant" obtained from the vehicle in the manual driving mode, and displays the map and restaurant list on the second display, While displaying a blank screen (Not Used) on the third display, it is possible to transmit a control command to the vehicle to output a voice saying "This is a list of nearby restaurants according to OO's preference. Shall I guide you to OOO?" through the speaker.

Meanwhile, unlike the case of FIG. 20 , in FIG. 20 , an example in which a voice input of “How about a restaurant OOO?” is obtained as a driver's voice input and content is provided in consideration of this will be described.

In detail, as shown in FIG. 22 , the server may receive from the vehicle a voice input “How about a restaurant OOO?” obtained from an input device of the vehicle.

Then, the server displays the speedometer screen on the first display based on the voice input received from the vehicle, displays the map and restaurant list on the second display, and displays a blank screen (Not Used) on the third display. , it is possible to transmit a control command to the vehicle to output a voice saying "Do you want to make a reservation and guide to the OOO restaurant?" through the speaker.

23 illustrates an example of providing content based on a sensing input of a vehicle in an autonomous driving mode of a vehicle according to an embodiment of the present specification, and FIG. An example of providing content based on a sensing input of a vehicle is illustrated.

23 , in the autonomous driving mode, the vehicle may detect a pedestrian in front of the vehicle through a forward pedestrian detection sensor. Subsequently, the vehicle may transmit an input sensed through the front pedestrian detection sensor to the server.

The server may set the format of content to be output to a plurality of output devices in the vehicle and generate the content by using the input detected through the front pedestrian detection sensor and the state information of the vehicle such as the autonomous driving mode. For example, the server displays a map showing an expected route on the first display, displays additional content 1 (warn popup) on the second display, and displays additional content 2 (stop warning, pause) on the third display ) can be controlled to display the vehicle. In addition, the server can control the vehicle to output a voice input saying "There is a pedestrian ahead. I will maneuver to avoid it. Be careful." through the vehicle's speaker.

Alternatively, as shown in FIG. 24 , in the manual driving mode, the vehicle may detect a pedestrian in front of the vehicle through the forward pedestrian detection sensor. Subsequently, the vehicle may transmit an input sensed through the front pedestrian detection sensor to the server.

The server may set the format of the content to be output to a plurality of output devices in the vehicle and generate the content by using the input detected through the front pedestrian detection sensor and the vehicle state information of the manual driving mode.

Here, the server may control the vehicle to output a smaller number of UIs to the vehicle in the 'manual driving mode' than when the vehicle is in the 'autonomous driving mode'. For example, the server displays only the speedometer on the first display, displays a map and a warning popup, which are relatively simple contents, on the second display, and displays a blank screen (Not Used) on the third display. can be controlled In addition, the server may control the vehicle to output a relatively short voice input of "Pedestrian Appears! Be careful!" through the vehicle's speaker.

The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (14)

A method for providing intelligent content for an autonomous driving system, the method comprising:
obtaining, by the server, an input from at least one input device of the autonomous vehicle; and
outputting, by the server, the content related to the input to at least one output device in the autonomous vehicle;
The output step is
Outputting the content based on at least one of output device characteristic information of the at least one output device and input device characteristic information of the at least one input device,
Way.
According to claim 1,
The output step is
It characterized in that the format of the content output to each of the at least one output device is set based on the location in the vehicle of each of the at least one output device,
Way.
According to claim 1,
The output step is
characterized in that the format of the content is set based on information related to the state of the vehicle,
Way.
4. The method of claim 3,
The information related to the state of the vehicle, characterized in that it includes traffic volume, signal, driver location or driver state,
Way.
5. The method of claim 4,
The information related to the state of the vehicle includes a driving mode of the vehicle,
The output step is
Characterized in determining the amount of the content based on the driving mode,
Way.
6. The method of claim 5,
When the vehicle is in the autonomous driving mode, the number of UI objects included in the content is greater than the number of UI objects included in the content when the vehicle is in the manual driving mode,
Way.
According to claim 1,
The output step is
characterized in that the format of the content is set based on information related to the characteristics of the input obtained from each of the at least one input device,
Way.
A server for intelligently providing content from an autonomous vehicle in an autonomous driving system, the server comprising:
communication module; and
processor; including;
The processor is
obtaining an input from at least one input device of the autonomous vehicle through the communication module;
outputting the content related to the input to at least one output device in the autonomous vehicle,
The processor is
Outputting the content based on at least one of output device characteristic information of the at least one output device and input device characteristic information of the at least one input device,
server.
9. The method of claim 8,
The processor is
It characterized in that the format of the content output to each of the at least one output device is set based on the location in the vehicle of each of the at least one output device,
server.
9. The method of claim 8,
The processor is
characterized in that the format of the content is set based on information related to the state of the vehicle,
server.
11. The method of claim 10,
The information related to the state of the vehicle, characterized in that it includes traffic volume, signal, driver location or driver state,
server.
12. The method of claim 11,
The information related to the state of the vehicle includes a driving mode of the vehicle,
The processor is
Characterized in determining the amount of the content based on the driving mode,
server.
13. The method of claim 12,
The processor is
When the vehicle is in the autonomous driving mode, the number of UI objects included in the content is greater than the number of UI objects included in the content when the vehicle is in the manual driving mode,
server.
9. The method of claim 8,
The processor is
characterized in that the format of the content is set based on information related to the characteristics of the input obtained from each of the at least one input device,
server.

Images