WO2021090969A1

WO2021090969A1 - Autonomous driving vehicle and method for providing augmented reality in autonomous driving vehicle

Info

Publication number: WO2021090969A1
Application number: PCT/KR2019/014930
Authority: WO
Inventors: 지애띠; 이기형; 정두경; 정준영; 최성환
Original assignee: 엘지전자 주식회사
Priority date: 2019-11-05
Filing date: 2019-11-05
Publication date: 2021-05-14
Also published as: KR20220087588A

Abstract

Disclosed are an autonomous driving vehicle and a method for providing augmented reality in an autonomous driving vehicle. An autonomous driving vehicle according to one embodiment of the present specification comprises: a plurality of displays; a plurality of crystal sound speakers that overlap the plurality of displays; and a processor that acquires device status information of the autonomous driving vehicle, displays a first image through a first display area of the plurality of displays on the basis of the device status information, and outputs, through a first speaker area of the plurality of crystal sound speakers, a first sound related to the first image on the basis of the first image, such that a heating phenomenon caused by an existing speaker in the autonomous driving vehicle can be prevented. The autonomous driving vehicle, a user terminal, and a server of the present specification can be associated with an artificial intelligence module, a drone (unmanned aerial vehicle, UAV), an augmented reality (AR) device, a virtual reality (VR) device, and devices related to 5G services, etc.

Description

Method of providing augmented reality in autonomous vehicles and autonomous vehicles

The present specification relates to an autonomous driving vehicle and a method of providing augmented reality in an autonomous driving vehicle, and more specifically, to an autonomous driving vehicle for providing an intuitive augmented reality to a user.

Vehicles can be classified into internal combustion engine vehicles, external combustion engine vehicles, gas turbine vehicles, or electric vehicles, depending on the type of prime mover used.

An autonomous driving vehicle refers to a vehicle that can operate by itself without driver or passenger manipulation, and an autonomous driving system refers to a system that monitors and controls such autonomous vehicles so that they can operate on their own. .

Meanwhile, a technology that provides an augmented reality that can inform a driver of a danger situation in the front/rear/side through an autonomous vehicle is being provided.

The present specification aims to solve the above-described necessity and/or problem.

In addition, the present specification aims to implement an autonomous vehicle for intuitively providing augmented reality to a driver in the form of sound and image.

In the autonomous driving vehicle providing augmented reality according to an embodiment of the present specification, a plurality of displays; A plurality of crystal sound speakers overlapping the plurality of displays; And obtaining device status information of the autonomous vehicle, displaying a first image through a first display area of the plurality of displays based on the device status information, and displaying the plurality of crystal sounds based on the first image. And a processor that outputs a first sound related to the first image through a first speaker area of the speaker.

The processor may determine at least one display among the plurality of displays based on the device status information, and display the first image through the first display area of the determined at least one display, the first speaker The area may be an area that at least partially overlaps the first display area.

The processor may determine the first display area and the first speaker area based on a location of the attention target outside the vehicle.

The processor may determine the first display area and the first speaker area based on a degree of danger of a target of attention outside the vehicle.

The processor determines at least one display from among the plurality of displays based on the device context information, displays the first image through the determined at least one display, and displays the first image based on a location where the first image is displayed. Thus, at least one speaker among the plurality of crystal sound speakers may be determined, and the first sound may be output through the determined at least one speaker.

A method for providing augmented reality by an autonomous vehicle according to an exemplary embodiment of the present specification, the method comprising: acquiring device status information of the autonomous vehicle; Displaying a first image through a first display area of a plurality of displays based on the device status information; And outputting a first sound related to the first image through a first speaker area of a plurality of crystal sound speakers overlapping the plurality of displays based on the first image.

Determining at least one display of the plurality of displays based on the device status information, and displaying the first image through the first display area of the determined at least one display, wherein the first One speaker area may be an area that at least partially overlaps the first display area.

And determining the first display area and the first speaker area based on the position of the attention object outside the vehicle.

And determining the first display area and the first speaker area based on the degree of danger of the attention object outside the vehicle.

Determining at least one display among the plurality of displays based on the device context information, displaying the first image through the determined at least one display, and the first image based on a position at which the first image is displayed. It may include determining at least one speaker among a plurality of crystal sound speakers, and outputting the first sound through the determined at least one speaker.

An effect of the method of providing an augmented reality for an autonomous vehicle according to an exemplary embodiment of the present specification will be described as follows.

According to the present specification, it is possible to prevent heat generation caused by an existing speaker in an autonomous vehicle.

In addition, according to the present specification, it is possible to prevent confusion of a driver caused by the output of redundant information between a speaker of an autonomous vehicle and other devices.

In addition, according to the present specification, a sound related to an augmented reality object in an autonomous vehicle can be clearly transmitted to the driver.

In addition, according to the present specification, by accurately classifying a plurality of augmented reality objects, it is possible to intuitively and quickly provide more important information to the driver in the current situation.

The effects obtainable in the present specification are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. .

The accompanying drawings, which are included as part of the detailed description to aid in understanding of the present specification, provide embodiments of the present specification, and describe technical features of the present specification together with the detailed description.

1 illustrates a block diagram of a wireless communication system to which the methods proposed in the present specification can be applied.

2 shows an example of a signal transmission/reception method in a wireless communication system.

3 shows an example of a basic operation of an autonomous vehicle and a 5G network in a 5G communication system.

4 shows an example of a vehicle-to-vehicle basic operation using 5G communication.

5 is a view showing a vehicle according to an embodiment of the present specification.

6 is a control block diagram of a vehicle according to an exemplary embodiment of the present specification.

7 is a control block diagram of an autonomous driving apparatus according to an embodiment of the present specification.

8 is a signal flow diagram of an autonomous vehicle according to an embodiment of the present specification.

9 is a view showing the interior of a vehicle according to an embodiment of the present specification.

10 is a block diagram referenced to describe a vehicle cabin system according to an exemplary embodiment of the present specification.

11 is a diagram referenced to explain a usage scenario of a user according to an embodiment of the present specification.

12 illustrates an AI device 100 according to an embodiment of the present specification.

13 shows an AI server 200 according to an embodiment of the present specification.

14 shows an AI system 1 according to an embodiment of the present specification.

15 is a flowchart illustrating a method of providing an augmented reality for an autonomous vehicle according to an exemplary embodiment of the present specification.

16 illustrates a plurality of displays and a plurality of crystal sound speakers of an autonomous vehicle according to an exemplary embodiment of the present specification.

17 shows a crystal sound speaker output system according to an embodiment of the present specification.

18 is a flowchart illustrating a more detailed process of outputting AR object information according to an embodiment of the present specification.

19 illustrates an example of outputting AR object information to a central portion of a crystal sound display according to an embodiment of the present specification.

20 illustrates an example of outputting AR object information in a diagonal form to a crystal sound display according to an embodiment of the present specification.

21 illustrates an example of sequentially outputting AR object information in one direction of a crystal sound display according to an embodiment of the present specification.

22 illustrates an example of outputting AR object information through a plurality of crystal sound displays according to an embodiment of the present specification.

23 illustrates an example of outputting AR object information based on an external attention target location according to an embodiment of the present specification.

24 illustrates an example of sequentially outputting AR object information along a path guidance direction of a navigation according to an embodiment of the present specification.

25 illustrates an example of outputting information on a plurality of AR objects and one AR sound according to an embodiment of the present specification.

26 illustrates an example of outputting a plurality of AR object information and a plurality of AR sounds according to an embodiment of the present specification.

27 illustrates an example of outputting a plurality of AR object information through a plurality of crystal sound displays according to an embodiment of the present specification.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for constituent elements used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present specification It should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first and second may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, terms such as "comprises" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

[Description of Figure 1]

A. UE 및 5G 네트워크 블록도 예시A. UE and 5G network block diagram example

Referring to FIG. 1, a device including an autonomous driving module (autonomous driving device) is defined as a first communication device (910 in FIG. 1 ), and a processor 911 may perform a detailed autonomous driving operation.

A 5G network including other vehicles communicating with the autonomous driving device may be defined as a second communication device (920 in FIG. 1), and the processor 921 may perform a detailed autonomous driving operation.

The 5G network may be referred to as a first communication device and an autonomous driving device may be referred to as a second communication device.

For example, the first communication device or the second communication device may be a base station, a network node, a transmission terminal, a reception terminal, a wireless device, a wireless communication device, an autonomous driving device, and the like.

For example, a terminal or user equipment (UE) is a vehicle, mobile phone, smart phone, laptop computer, digital broadcasting terminal, personal digital assistants (PDA), portable multimedia player (PMP). , Navigation, slate PC, tablet PC, ultrabook, wearable device, for example, smartwatch, smart glass, HMD ( head mounted display)). For example, the HMD may be a display device worn on the head. For example, HMD can be used to implement VR, AR or MR. Referring to FIG. 1, a first communication device 910 and a second communication device 920 include a processor (processor, 911,921), memory (914,924), one or more Tx/Rx RF modules (radio frequency module, 915,925). ,

Tx processors

912 and 922,

Rx processors

913 and 923, and

antennas

916 and 926. The Tx/Rx module is also called a transceiver. Each Tx/Rx module 915 transmits a signal through a respective antenna 926. The processor implements the previously salpin functions, processes and/or methods. The processor 921 may be associated with a memory 924 that stores program code and data. The memory may be referred to as a computer-readable medium. More specifically, in the DL (communication from the first communication device to the second communication device), the transmission (TX) processor 912 implements various signal processing functions for the L1 layer (ie, the physical layer). The receive (RX) processor implements the various signal processing functions of L1 (ie, the physical layer).

The UL (communication from the second communication device to the first communication device) is handled in the first communication device 910 in a manner similar to that described with respect to the receiver function in the second communication device 920. Each Tx/Rx module 925 receives a signal through a respective antenna 926. Each Tx/Rx module provides an RF carrier and information to the RX processor 923. The processor 921 may be associated with a memory 924 that stores program code and data. The memory may be referred to as a computer-readable medium.

[Description of Figure 2]

B. 무선 통신 시스템에서 신호 송/수신 방법B. Signal transmission/reception method in wireless communication system

2 illustrates physical channels and general signal transmission used in a 3GPP system.

In a wireless communication system, a terminal receives information from a base station through a downlink (DL), and the terminal transmits information to the base station through an uplink (UL). The information transmitted and received by the base station and the terminal includes data and various control information, and various physical channels exist according to the type/use of the information transmitted and received by them.

When the terminal is powered on or newly enters a cell, the terminal performs an initial cell search operation such as synchronizing with the base station (S201). To this end, the UE receives a Primary Synchronization Signal (PSS) and a Secondary Synchronization Signal (SSS) from the base station to synchronize with the base station and obtain information such as cell ID. Thereafter, the terminal may receive a physical broadcast channel (PBCH) from the base station to obtain intra-cell broadcast information. Meanwhile, the UE may receive a downlink reference signal (DL RS) in the initial cell search step to check the downlink channel state.

After completing the initial cell search, the UE acquires more detailed system information by receiving a physical downlink control channel (PDCCH) and a physical downlink shared channel (PDSCH) according to the information carried on the PDCCH. It can be done (S202).

Meanwhile, when accessing the base station for the first time or when there is no radio resource for signal transmission, the terminal may perform a random access procedure (RACH) with respect to the base station (S203 to S206). To this end, the UE transmits a specific sequence as a preamble through a physical random access channel (PRACH) (S203 and S205), and a response message to the preamble through a PDCCH and a corresponding PDSCH (RAR (Random Access Response) message) In the case of contention-based RACH, a contention resolution procedure may be additionally performed (S206).

After performing the above-described procedure, the UE receives PDCCH/PDSCH (S207) and physical uplink shared channel (PUSCH)/physical uplink control channel as a general uplink/downlink signal transmission procedure. Control Channel; PUCCH) transmission (S208) may be performed. In particular, the terminal may receive downlink control information (DCI) through the PDCCH. Here, the DCI includes control information such as resource allocation information for the terminal, and different formats may be applied according to the purpose of use.

On the other hand, the control information transmitted by the terminal to the base station through uplink or received by the base station by the terminal is a downlink/uplink ACK/NACK signal, a channel quality indicator (CQI), a precoding matrix index (PMI), a rank indicator (RI) ) And the like. The UE may transmit control information such as the above-described CQI/PMI/RI through PUSCH and/or PUCCH.

With reference to FIG. 2, an initial access (IA) procedure in a 5G communication system will be additionally described.

The UE may perform cell search, system information acquisition, beam alignment for initial access, and DL measurement based on the SSB. SSB is used interchangeably with a Synchronization Signal/Physical Broadcast Channel (SS/PBCH) block.

SSB consists of PSS, SSS and PBCH. The SSB is composed of four consecutive OFDM symbols, and PSS, PBCH, SSS/PBCH or PBCH are transmitted for each OFDM symbol. The PSS and SSS are each composed of 1 OFDM symbol and 127 subcarriers, and the PBCH is composed of 3 OFDM symbols and 576 subcarriers.

Cell discovery refers to a process in which the UE acquires time/frequency synchronization of a cell and detects a cell identifier (eg, Physical layer Cell ID, PCI) of the cell. PSS is used to detect a cell ID within a cell ID group, and SSS is used to detect a cell ID group. PBCH is used for SSB (time) index detection and half-frame detection.

There are 336 cell ID groups, and 3 cell IDs exist for each cell ID group. There are a total of 1008 cell IDs. Information on the cell ID group to which the cell ID of the cell belongs is provided/obtained through the SSS of the cell, and information on the cell ID among 336 cells in the cell ID is provided/obtained through the PSS.

SSB is transmitted periodically according to the SSB period. The SSB basic period assumed by the UE during initial cell search is defined as 20 ms. After cell access, the SSB period may be set to one of {5ms, 10ms, 20ms, 40ms, 80ms, 160ms} by the network (eg, BS).

Next, it looks at obtaining system information (SI).

SI is divided into a master information block (MIB) and a plurality of system information blocks (SIB). SI other than MIB may be referred to as RMSI (Remaining Minimum System Information). The MIB includes information/parameters for monitoring the PDCCH that schedules the PDSCH carrying System Information Block1 (SIB1), and is transmitted by the BS through the PBCH of the SSB. SIB1 includes information related to availability and scheduling (eg, transmission period, SI-window size) of the remaining SIBs (hereinafter, SIBx, x is an integer greater than or equal to 2). SIBx is included in the SI message and is transmitted through the PDSCH. Each SI message is transmitted within a periodic time window (ie, SI-window).

Referring to FIG. 2, a random access (RA) process in a 5G communication system will be additionally described.

The random access process is used for various purposes. For example, the random access procedure may be used for initial network access, handover, and UE-triggered UL data transmission. The UE may acquire UL synchronization and UL transmission resources through a random access process. The random access process is divided into a contention-based random access process and a contention free random access process. The detailed procedure for the contention-based random access process is as follows.

The UE may transmit the random access preamble as Msg1 of the random access procedure in the UL through the PRACH. Random access preamble sequences having two different lengths are supported. The long sequence length 839 is applied for subcarrier spacing of 1.25 and 5 kHz, and the short sequence length 139 is applied for subcarrier spacing of 15, 30, 60 and 120 kHz.

When the BS receives the random access preamble from the UE, the BS transmits a random access response (RAR) message (Msg2) to the UE. The PDCCH for scheduling the PDSCH carrying RAR is transmitted after being CRC masked with a random access (RA) radio network temporary identifier (RNTI) (RA-RNTI). A UE that detects a PDCCH masked with RA-RNTI may receive an RAR from a PDSCH scheduled by a DCI carried by the PDCCH. The UE checks whether the preamble transmitted by the UE, that is, random access response information for Msg1, is in the RAR. Whether there is random access information for Msg1 transmitted by the UE may be determined based on whether there is a random access preamble ID for the preamble transmitted by the UE. If there is no response to Msg1, the UE may retransmit the RACH preamble within a predetermined number of times while performing power ramping. The UE calculates the PRACH transmission power for retransmission of the preamble based on the most recent path loss and power ramping counter.

The UE may transmit UL transmission as Msg3 in a random access procedure on an uplink shared channel based on random access response information. Msg3 may include an RRC connection request and a UE identifier. In response to Msg3, the network may send Msg4, which may be treated as a contention resolution message on the DL. By receiving Msg4, the UE can enter the RRC connected state.

C. 5G 통신 시스템의 빔 관리(Beam Management, BM) 절차C. Beam Management (BM) procedure of 5G communication system

The BM process may be divided into (1) a DL BM process using SSB or CSI-RS and (2) a UL BM process using a sounding reference signal (SRS). In addition, each BM process may include Tx beam sweeping to determine the Tx beam and Rx beam sweeping to determine the Rx beam.

Let's look at the DL BM process using SSB.

Configuration for beam report using SSB is performed when channel state information (CSI)/beam is configured in RRC_CONNECTED.

-The UE receives a CSI-ResourceConfig IE including CSI-SSB-ResourceSetList for SSB resources used for BM from BS. The RRC parameter csi-SSB-ResourceSetList represents a list of SSB resources used for beam management and reporting in one resource set. Here, the SSB resource set may be set to {SSBx1, SSBx2, SSBx3, SSBx4, 쪋}. The SSB index may be defined from 0 to 63.

-The UE receives signals on SSB resources from the BS based on the CSI-SSB-ResourceSetList.

-When the CSI-RS reportConfig related to reporting on SSBRI and reference signal received power (RSRP) is configured, the UE reports the best SSBRI and RSRP corresponding thereto to the BS. For example, when the reportQuantity of the CSI-RS reportConfig IE is set to'ssb-Index-RSRP', the UE reports the best SSBRI and corresponding RSRP to the BS.

When the UE is configured with CSI-RS resources in the same OFDM symbol(s) as the SSB, and'QCL-TypeD' is applicable, the UE is similarly co-located in terms of'QCL-TypeD' where the CSI-RS and SSB are ( quasi co-located, QCL). Here, QCL-TypeD may mean that QCL is performed between antenna ports in terms of a spatial Rx parameter. When the UE receives signals from a plurality of DL antenna ports in a QCL-TypeD relationship, the same reception beam may be applied.

Next, a DL BM process using CSI-RS will be described.

The Rx beam determination (or refinement) process of the UE using CSI-RS and the Tx beam sweeping process of the BS are sequentially described. In the UE's Rx beam determination process, the repetition parameter is set to'ON', and in the BS's Tx beam sweeping process, the repetition parameter is set to'OFF'.

First, a process of determining the Rx beam of the UE is described.

-The UE receives the NZP CSI-RS resource set IE including the RRC parameter for'repetition' from the BS through RRC signaling. Here, the RRC parameter'repetition' is set to'ON'.

-The UE repeats signals on the resource(s) in the CSI-RS resource set in which the RRC parameter'repetition' is set to'ON' in different OFDM symbols through the same Tx beam (or DL spatial domain transmission filter) of the BS Receive.

-The UE determines its own Rx beam.

-The UE omits CSI reporting. That is, the UE may omit CSI reporting when the shopping price RRC parameter'repetition' is set to'ON'.

Next, a process of determining the Tx beam of the BS will be described.

-The UE receives the NZP CSI-RS resource set IE including the RRC parameter for'repetition' from the BS through RRC signaling. Here, the RRC parameter'repetition' is set to'OFF', and is related to the Tx beam sweeping process of the BS.

-The UE receives signals on resources in the CSI-RS resource set in which the RRC parameter'repetition' is set to'OFF' through different Tx beams (DL spatial domain transmission filters) of the BS.

-The UE selects (or determines) the best beam.

-The UE reports the ID (eg, CRI) and related quality information (eg, RSRP) for the selected beam to the BS. That is, when the CSI-RS is transmitted for the BM, the UE reports the CRI and the RSRP for it to the BS.

Next, a UL BM process using SRS will be described.

-The UE receives RRC signaling (eg, SRS-Config IE) including a usage parameter set to'beam management' (RRC parameter) from the BS. SRS-Config IE is used for SRS transmission configuration. The SRS-Config IE includes a list of SRS-Resources and a list of SRS-ResourceSets. Each SRS resource set means a set of SRS-resources.

-The UE determines Tx beamforming for the SRS resource to be transmitted based on the SRS-SpatialRelation Info included in the SRS-Config IE. Here, the SRS-SpatialRelation Info is set for each SRS resource, and indicates whether to apply the same beamforming as the beamforming used in SSB, CSI-RS or SRS for each SRS resource.

-If SRS-SpatialRelationInfo is set in the SRS resource, the same beamforming as the beamforming used in SSB, CSI-RS or SRS is applied and transmitted. However, if SRS-SpatialRelationInfo is not set in the SRS resource, the UE randomly determines Tx beamforming and transmits the SRS through the determined Tx beamforming.

Next, a beam failure recovery (BFR) process will be described.

In a beamformed system, Radio Link Failure (RLF) may frequently occur due to rotation, movement, or beamforming blockage of the UE. Therefore, BFR is supported in NR to prevent frequent RLF from occurring. BFR is similar to the radio link failure recovery process, and may be supported when the UE knows the new candidate beam(s). For beam failure detection, the BS sets beam failure detection reference signals to the UE, and the UE sets the number of beam failure indications from the physical layer of the UE within a period set by RRC signaling of the BS. When a threshold set by RRC signaling is reached, a beam failure is declared. After the beam failure is detected, the UE triggers beam failure recovery by initiating a random access procedure on the PCell; Beam failure recovery is performed by selecting a suitable beam (if the BS has provided dedicated random access resources for certain beams, these are prioritized by the UE). Upon completion of the random access procedure, it is considered that the beam failure recovery is complete.

D. URLLC (Ultra-Reliable and Low Latency Communication)D. URLLC (Ultra-Reliable and Low Latency Communication)

URLLC transmission as defined by NR is (1) relatively low traffic size, (2) relatively low arrival rate, (3) extremely low latency requirement (e.g. 0.5, 1ms), (4) It may mean a relatively short transmission duration (eg, 2 OFDM symbols), and (5) transmission of an urgent service/message. In the case of UL, transmission for a specific type of traffic (e.g., URLLC) must be multiplexed with another transmission (e.g., eMBB) scheduled in advance in order to satisfy a more stringent latency requirement. Needs to be. In this regard, as one method, information that a specific resource will be preempted is given to the previously scheduled UE, and the URLLC UE uses the corresponding resource for UL transmission.

In the case of NR, dynamic resource sharing between eMBB and URLLC is supported. eMBB and URLLC services can be scheduled on non-overlapping time/frequency resources, and URLLC transmission can occur on resources scheduled for ongoing eMBB traffic. The eMBB UE may not be able to know whether the PDSCH transmission of the corresponding UE is partially punctured, and the UE may not be able to decode the PDSCH due to corrupted coded bits. In consideration of this point, the NR provides a preemption indication. The preemption indication may be referred to as an interrupted transmission indication.

Regarding the preemption indication, the UE receives the DownlinkPreemption IE through RRC signaling from the BS. When the UE is provided with the DownlinkPreemption IE, the UE is configured with the INT-RNTI provided by the parameter int-RNTI in the DownlinkPreemption IE for monitoring of the PDCCH carrying DCI format 2_1. The UE is additionally configured with a set of serving cells by INT-ConfigurationPerServing Cell including a set of serving cell indexes provided by servingCellID and a corresponding set of positions for fields in DCI format 2_1 by positionInDCI, and dci-PayloadSize It is set with the information payload size for DCI format 2_1 by and is set with the indication granularity of time-frequency resources by timeFrequencySect.

The UE receives DCI format 2_1 from the BS based on the DownlinkPreemption IE.

When the UE detects the DCI format 2_1 for the serving cell in the set set of serving cells, the UE is the DCI format among the set of PRBs and symbols of the monitoring period immediately preceding the monitoring period to which the DCI format 2_1 belongs. It may be assumed that there is no transmission to the UE in the PRBs and symbols indicated by 2_1. For example, the UE considers that the signal in the time-frequency resource indicated by the preemption is not a DL transmission scheduled to it, and decodes data based on the signals received in the remaining resource regions.

E. mMTC (massive MTC)E. mMTC (massive MTC)

Massive Machine Type Communication (mMTC) is one of 5G scenarios to support hyper-connection services that communicate with a large number of UEs at the same time. In this environment, the UE communicates intermittently with a very low transmission rate and mobility. Therefore, mMTC aims at how long the UE can be driven at a low cost for a long time. Regarding mMTC technology, 3GPP deals with MTC and NB (NarrowBand)-IoT.

The mMTC technology has features such as repetitive transmission of PDCCH, PUCCH, physical downlink shared channel (PDSCH), and PUSCH, frequency hopping, retuning, and guard period.

That is, a PUSCH (or PUCCH (especially, long PUCCH) or PRACH) including specific information and a PDSCH (or PDCCH) including a response to specific information are repeatedly transmitted. Repetitive transmission is performed through frequency hopping, and for repetitive transmission, (RF) retuning is performed in a guard period from a first frequency resource to a second frequency resource, and specific information And a response to specific information may be transmitted/received through a narrowband (ex. 6 resource block (RB) or 1 RB).

[Description of Figure 3]

F. 5G 통신을 이용한 자율 주행 차량 간 기본 동작F. Basic operation between autonomous vehicles using 5G communication

The autonomous vehicle transmits specific information transmission to the 5G network (S1). The specific information may include autonomous driving related information. In addition, the 5G network may determine whether to remotely control the vehicle (S2). Here, the 5G network may include a server or module that performs remote control related to autonomous driving. In addition, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle (S3).

G. 5G 통신 시스템에서 자율 주행 차량과 5G 네트워크 간의 응용 동작G. Application operation between autonomous vehicle and 5G network in 5G communication system

Hereinafter, the operation of an autonomous vehicle using 5G communication will be described in more detail with reference to Salpin wireless communication technology (BM procedure, URLLC, Mmtc, etc.) prior to FIGS. 1 and 2.

First, a basic procedure of an application operation to which the eMBB technology of 5G communication is applied and the method proposed in the present specification to be described later will be described.

As in steps S1 and S3 of FIG. 3, in order for the autonomous vehicle to transmit/receive the 5G network, signals, information, etc., the autonomous vehicle is an initial access procedure with the 5G network prior to step S1 of FIG. 3. And a random access procedure.

More specifically, the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB in order to obtain DL synchronization and system information. In the initial access procedure, a beam management (BM) process and a beam failure recovery process may be added. In the process of receiving a signal from the 5G network by an autonomous vehicle, a quasi-co location (QCL) ) Relationships can be added.

In addition, the autonomous vehicle performs a random access procedure with a 5G network to obtain UL synchronization and/or transmit UL. And, the 5G network may transmit a UL grant for scheduling transmission of specific information to the autonomous vehicle. have. Accordingly, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. In addition, the 5G network transmits a DL grant for scheduling transmission of a 5G processing result for the specific information to the autonomous vehicle. Accordingly, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle based on the DL grant.

Next, a basic procedure of an application operation to which the URLLC technology of 5G communication is applied and the method proposed in the present specification to be described later will be described.

As described above, after the autonomous vehicle performs an initial access procedure and/or a random access procedure with the 5G network, the autonomous vehicle may receive a DownlinkPreemption IE from the 5G network. In addition, the autonomous vehicle receives DCI format 2_1 including a pre-emption indication from the 5G network based on the DownlinkPreemption IE. And, the autonomous vehicle does not perform (or expect or assume) the reception of eMBB data in the resource (PRB and/or OFDM symbol) indicated by the pre-emption indication. Thereafter, the autonomous vehicle may receive a UL grant from the 5G network when it is necessary to transmit specific information.

Next, the method proposed in the present specification to be described later and the basic procedure of the application operation to which the mMTC technology of 5G communication is applied will be described.

Among the steps of FIG. 3, a description will be made focusing on the parts that are changed by the application of the mMTC technology.

In step S1 of FIG. 3, the autonomous vehicle receives a UL grant from the 5G network to transmit specific information to the 5G network. Here, the UL grant includes information on the number of repetitions for transmission of the specific information, and the specific information may be repeatedly transmitted based on the information on the number of repetitions. That is, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. Further, repetitive transmission of specific information may be performed through frequency hopping, transmission of first specific information may be transmitted in a first frequency resource, and transmission of second specific information may be transmitted in a second frequency resource. The specific information may be transmitted through a narrowband of 6RB (Resource Block) or 1RB (Resource Block).

[Description of Figure 4]

F. 5G 통신을 이용한 차량 대 차량 간의 자율 주행 동작F. Vehicle-to-vehicle autonomous driving operation using 5G communication

4 illustrates an example of a vehicle-to-vehicle basic operation using 5G communication.

The first vehicle transmits specific information to the second vehicle (S61). The second vehicle transmits a response to the specific information to the first vehicle (S62).

On the other hand, depending on whether the 5G network directly (side link communication transmission mode 3) or indirectly (sidelink communication transmission mode 4) is involved in the resource allocation of the specific information and the response to the specific information The composition may vary.

Next, a vehicle-to-vehicle application operation using 5G communication will be described.

First, a method in which a 5G network is directly involved in resource allocation for vehicle-to-vehicle signal transmission/reception will be described.

The 5G network may transmit DCI format 5A to the first vehicle for scheduling of mode 3 transmission (PSCCH and/or PSSCH transmission). Here, a physical sidelink control channel (PSCCH) is a 5G physical channel for scheduling specific information transmission, and a physical sidelink shared channel (PSSCH) is a 5G physical channel for transmitting specific information. In addition, the first vehicle transmits SCI format 1 for scheduling specific information transmission to the second vehicle on the PSCCH. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

Next, we will look at how the 5G network indirectly participates in resource allocation for signal transmission/reception.

The first vehicle senses a resource for mode 4 transmission in the first window. Then, the first vehicle selects a resource for mode 4 transmission in the second window based on the sensing result. Here, the first window means a sensing window, and the second window means a selection window. The first vehicle transmits SCI format 1 for scheduling specific information transmission to the second vehicle on the PSCCH based on the selected resource. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

The above salpin 5G communication technology may be applied in combination with the methods proposed in the present specification to be described later, or may be supplemented to specify or clarify the technical characteristics of the methods proposed in the present specification.

[Description of Figure 5]

주행Driving

(1) 차량 외관(1) Vehicle appearance

Referring to FIG. 5, the vehicle 10 according to the embodiment of the present specification is defined as a transportation means traveling on a road or track. The vehicle 10 is a concept including a car, a train, and a motorcycle. The vehicle 10 may be a concept including both an internal combustion engine vehicle including an engine as a power source, a hybrid vehicle including an engine and an electric motor as a power source, an electric vehicle including an electric motor as a power source, and the like. The vehicle 10 may be a vehicle owned by an individual. The vehicle 10 may be a shared vehicle. The vehicle 10 may be an autonomous vehicle.

[Description of Figure 6]

(2) 차량의 구성 요소(2) vehicle components

Referring to FIG. 6, the vehicle 10 includes a user interface device 200, an object detection device 210, a communication device 220, a driving operation device 230, a main ECU 240, and a drive control device 250. ), an autonomous driving device 260, a sensing unit 270, and a location data generating device 280. Object detection device 210, communication device 220, driving operation device 230, main ECU 240, drive control device 250, autonomous driving device 260, sensing unit 270, and position data generating device Each of 280 may be implemented as an electronic device that generates an electrical signal and exchanges electrical signals with each other.

1) 사용자 인터페이스 장치1) User interface device

The user interface device 200 is a device for communicating with the vehicle 10 and a user. The user interface device 200 may receive a user input and provide information generated in the vehicle 10 to the user. The vehicle 10 may implement a user interface (UI) or a user experience (UX) through the user interface device 200. The user interface device 200 may include an input device, an output device, and a user monitoring device.

2) 오브젝트 검출 장치2) Object detection device

The object detection device 210 may generate information on an object outside the vehicle 10. The information on the object may include at least one of information on the presence or absence of the object, location information of the object, distance information between the vehicle 10 and the object, and relative speed information between the vehicle 10 and the object. . The object detection device 210 may detect an object outside the vehicle 10. The object detection apparatus 210 may include at least one sensor capable of detecting an object outside the vehicle 10. The object detection device 210 may include at least one of a camera, a radar, a lidar, an ultrasonic sensor, and an infrared sensor. The object detection device 210 may provide data on an object generated based on a sensing signal generated by a sensor to at least one electronic device included in the vehicle.

2.1) 카메라2.1) Camera

The camera may generate information on an object outside the vehicle 10 by using an image. The camera may include at least one lens, at least one image sensor, and at least one processor that is electrically connected to the image sensor and processes a received signal, and generates data on an object based on the processed signal.

The camera may be at least one of a mono camera, a stereo camera, and an AVM (Around View Monitoring) camera. The camera may use various image processing algorithms to obtain position information of an object, distance information to an object, or information on a relative speed to an object. For example, from the acquired image, the camera may acquire distance information and relative speed information from the object based on a change in the size of the object over time. For example, the camera may obtain distance information and relative speed information with an object through a pin hole model, road surface profiling, or the like. For example, the camera may obtain distance information and relative speed information from an object based on disparity information from a stereo image obtained from a stereo camera.

The camera may be mounted in a position where field of view (FOV) can be secured in the vehicle to photograph the outside of the vehicle. The camera may be placed in the interior of the vehicle, close to the front windshield, to acquire an image of the front of the vehicle. The camera can be placed around the front bumper or radiator grille. The camera may be placed close to the rear glass, in the interior of the vehicle, in order to acquire an image of the rear of the vehicle. The camera can be placed around the rear bumper, trunk or tailgate. The camera may be disposed in proximity to at least one of the side windows in the interior of the vehicle in order to acquire an image of the vehicle side. Alternatively, the camera may be disposed around a side mirror, a fender, or a door.

2.2) 레이다2.2) radar

The radar may use radio waves to generate information on objects outside the vehicle 10. The radar may include at least one processor that is electrically connected to the electromagnetic wave transmitter, the electromagnetic wave receiver, and the electromagnetic wave transmitter and the electromagnetic wave receiver, processes a received signal, and generates data for an object based on the processed signal. The radar may be implemented in a pulse radar method or a continuous wave radar method according to the principle of radio wave emission. The radar may be implemented in a frequency modulated continuous wave (FMCW) method or a frequency shift keyong (FSK) method according to a signal waveform among continuous wave radar methods. The radar detects an object by means of an electromagnetic wave, based on a Time of Flight (TOF) method or a phase-shift method, and detects the position of the detected object, the distance to the detected object, and the relative speed. I can. The radar may be placed at a suitable location outside the vehicle to detect objects located in front, rear or side of the vehicle.

2.3) 라이다2.3) Lida

The lidar may generate information on an object outside the vehicle 10 by using laser light. The radar may include at least one processor that is electrically connected to the optical transmitter, the optical receiver, and the optical transmitter and the optical receiver, processes a received signal, and generates data for an object based on the processed signal. . The rider may be implemented in a Time of Flight (TOF) method or a phase-shift method. The lidar can be implemented either driven or non-driven. When implemented as a drive type, the lidar is rotated by a motor, and objects around the vehicle 10 can be detected. When implemented in a non-driven manner, the lidar can detect an object located within a predetermined range with respect to the vehicle by optical steering. The vehicle 100 may include a plurality of non-driven lidars. The radar detects an object based on a time of flight (TOF) method or a phase-shift method by means of a laser light, and determines the position of the detected object, the distance to the detected object, and the relative speed. Can be detected. The lidar may be placed at an appropriate location outside the vehicle to detect objects located in front, rear or side of the vehicle.

3) 통신 장치3) Communication device

The communication device 220 may exchange signals with devices located outside the vehicle 10. The communication device 220 may exchange signals with at least one of an infrastructure (eg, a server, a broadcasting station), another vehicle, and a terminal. The communication device 220 may include at least one of a transmission antenna, a reception antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

For example, the communication device may exchange signals with external devices based on C-V2X (Cellular V2X) technology. For example, C-V2X technology may include LTE-based sidelink communication and/or NR-based sidelink communication. Contents related to C-V2X will be described later.

For example, a communication device can communicate with external devices based on the IEEE 802.11p PHY/MAC layer technology and the Dedicated Short Range Communications (DSRC) technology based on the IEEE 1609 Network/Transport layer technology, or the Wireless Access in Vehicular Environment (WAVE) standard. Can be exchanged. DSRC (or WAVE standard) technology is a communication standard designed to provide Intelligent Transport System (ITS) services through short-distance dedicated communication between vehicle-mounted devices or between roadside devices and vehicle-mounted devices. The DSRC technology may use a frequency of 5.9 GHz band, and may be a communication method having a data transmission rate of 3 Mbps to 27 Mbps. IEEE 802.11p technology can be combined with IEEE 1609 technology to support DSRC technology (or WAVE standard).

The communication apparatus of the present specification may exchange signals with an external device using only either C-V2X technology or DSRC technology. Alternatively, the communication device of the present specification may exchange signals with an external device by hybridizing C-V2X technology and DSRC technology.

4) 운전 조작 장치4) Driving operation device

The driving operation device 230 is a device that receives a user input for driving. In the case of the manual mode, the vehicle 10 may be driven based on a signal provided by the driving operation device 230. The driving operation device 230 may include a steering input device (eg, a steering wheel), an acceleration input device (eg, an accelerator pedal), and a brake input device (eg, a brake pedal).

5) 메인 ECU5) Main ECU

The main ECU 240 may control the overall operation of at least one electronic device provided in the vehicle 10.

6) 구동 제어 장치6) Drive control device

The drive control device 250 is a device that electrically controls various vehicle drive devices in the vehicle 10. The drive control device 250 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train drive control device may include a power source drive control device and a transmission drive control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device driving control device may include a safety belt driving control device for controlling the safety belt.

The drive control device 250 includes at least one electronic control device (eg, a control Electronic Control Unit (ECU)).

The vehicle type control device 250 may control the vehicle driving device based on a signal received from the autonomous driving device 260. For example, the control device 250 may control a power train, a steering device, and a brake device based on a signal received from the autonomous driving device 260.

7) 자율 주행 장치7) Autonomous driving device

The autonomous driving device 260 may generate a path for autonomous driving based on the acquired data. The autonomous driving device 260 may generate a driving plan for driving along the generated route. The autonomous driving device 260 may generate a signal for controlling the movement of the vehicle according to the driving plan. The autonomous driving device 260 may provide the generated signal to the driving control device 250.

The autonomous driving device 260 may implement at least one Advanced Driver Assistance System (ADAS) function. ADAS includes Adaptive Cruise Control (ACC), Autonomous Emergency Braking (AEB), Forward Collision Warning (FCW), and Lane Keeping Assist (LKA). ), Lane Change Assist (LCA), Target Following Assist (TFA), Blind Spot Detection (BSD), Adaptive High Beam Control System (HBA: High Beam Assist) , APS (Auto Parking System), Pedestrian Collision Warning System (PD collision warning system), Traffic Sign Recognition (TSR), Traffic Sign Assist (TSA), Night Vision System At least one of (NV: Night Vision), Driver Status Monitoring (DSM), and Traffic Jam Assist (TJA) may be implemented.

The autonomous driving apparatus 260 may perform a switching operation from an autonomous driving mode to a manual driving mode or a switching operation from a manual driving mode to an autonomous driving mode. For example, the autonomous driving device 260 may switch the mode of the vehicle 10 from the autonomous driving mode to the manual driving mode or the autonomous driving mode from the manual driving mode based on a signal received from the user interface device 200. Can be switched to.

8) 센싱부8) Sensing part

The sensing unit 270 may sense the state of the vehicle. The sensing unit 270 includes an inertial measurement unit (IMU) sensor, a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight detection sensor, a heading sensor, a position module, and a vehicle. It may include at least one of a forward/reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor, a temperature sensor, a humidity sensor, an ultrasonic sensor, an illuminance sensor, and a pedal position sensor. Meanwhile, the inertial measurement unit (IMU) sensor may include one or more of an acceleration sensor, a gyro sensor, and a magnetic sensor.

The sensing unit 270 may generate state data of the vehicle based on a signal generated by at least one sensor. The vehicle state data may be information generated based on data sensed by various sensors provided inside the vehicle. The sensing unit 270 includes vehicle attitude data, vehicle motion data, vehicle yaw data, vehicle roll data, vehicle pitch data, vehicle collision data, vehicle direction data, vehicle angle data, and vehicle speed. Data, vehicle acceleration data, vehicle inclination data, vehicle forward/reverse data, vehicle weight data, battery data, fuel data, tire pressure data, vehicle internal temperature data, vehicle internal humidity data, steering wheel rotation angle data, vehicle exterior illuminance Data, pressure data applied to the accelerator pedal, and pressure data applied to the brake pedal can be generated.

9) 위치 데이터 생성 장치9) Location data generation device

The location data generating device 280 may generate location data of the vehicle 10. The location data generating apparatus 280 may include at least one of a Global Positioning System (GPS) and a Differential Global Positioning System (DGPS). The location data generating apparatus 280 may generate location data of the vehicle 10 based on a signal generated by at least one of GPS and DGPS. According to an embodiment, the location data generation apparatus 280 may correct location data based on at least one of an IMU (Inertial Measurement Unit) of the sensing unit 270 and a camera of the object detection apparatus 210. The location data generating device 280 may be referred to as a Global Navigation Satellite System (GNSS).

Vehicle 10 may include an internal communication system 50. A plurality of electronic devices included in the vehicle 10 may exchange signals through the internal communication system 50. Signals may contain data. The internal communication system 50 may use at least one communication protocol (eg, CAN, LIN, FlexRay, MOST, Ethernet).

[Description of Figure 7]

(3) 자율 주행 장치의 구성 요소(3) Components of autonomous driving devices

Referring to FIG. 7, the autonomous driving device 260 may include a memory 140, a processor 170, an interface unit 180, and a power supply unit 190.

The memory 140 is electrically connected to the processor 170. The memory 140 may store basic data for a unit, control data for controlling the operation of the unit, and input/output data. The memory 140 may store data processed by the processor 170. In terms of hardware, the memory 140 may be configured with at least one of a ROM, a RAM, an EPROM, a flash drive, and a hard drive. The memory 140 may store various data for the overall operation of the autonomous driving device 260, such as a program for processing or controlling the processor 170. The memory 140 may be implemented integrally with the processor 170. Depending on the embodiment, the memory 140 may be classified as a sub-element of the processor 170.

The interface unit 180 may exchange signals with at least one electronic device provided in the vehicle 10 by wire or wirelessly. The interface unit 280 includes an object detection device 210, a communication device 220, a driving operation device 230, a main ECU 240, a drive control device 250, a sensing unit 270, and a position data generating device. A signal may be exchanged with at least one of 280 by wire or wirelessly. The interface unit 280 may be configured with at least one of a communication module, a terminal, a pin, a cable, a port, a circuit, an element, and a device.

The power supply unit 190 may supply power to the autonomous driving device 260. The power supply unit 190 may receive power from a power source (eg, a battery) included in the vehicle 10 and supply power to each unit of the autonomous driving device 260. The power supply unit 190 may be operated according to a control signal provided from the main ECU 240. The power supply unit 190 may include a switched-mode power supply (SMPS).

The processor 170 may be electrically connected to the memory 140, the interface unit 280, and the power supply unit 190 to exchange signals. The processor 170 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, and controllers. It may be implemented using at least one of (controllers), micro-controllers, microprocessors, and electrical units for performing other functions.

The processor 170 may be driven by power provided from the power supply unit 190. The processor 170 may receive data, process data, generate a signal, and provide a signal while power is supplied by the power supply unit 190.

The processor 170 may receive information from another electronic device in the vehicle 10 through the interface unit 180. The processor 170 may provide a control signal to another electronic device in the vehicle 10 through the interface unit 180.

The autonomous driving device 260 may include at least one printed circuit board (PCB). The memory 140, the interface unit 180, the power supply unit 190, and the processor 170 may be electrically connected to a printed circuit board.

[Description of Fig. 8] Signal flow of an autonomous vehicle

(4) operation of autonomous driving devices

1) Receiving operation

Referring to FIG. 8, the processor 170 may perform a reception operation. The processor 170 may receive data from at least one of the object detection device 210, the communication device 220, the sensing unit 270, and the location data generation device 280 through the interface unit 180. I can. The processor 170 may receive object data from the object detection apparatus 210. The processor 170 may receive HD map data from the communication device 220. The processor 170 may receive vehicle state data from the sensing unit 270. The processor 170 may receive location data from the location data generating device 280.

2) Processing/judgment operation

The processor 170 may perform a processing/determining operation. The processor 170 may perform a processing/determining operation based on the driving situation information. The processor 170 may perform a processing/determining operation based on at least one of object data, HD map data, vehicle state data, and location data.

2.1) 드라이빙 플랜 데이터 생성 동작2.1) Driving plan data generation operation

The processor 170 may generate driving plan data. For example, the processor 1700 may generate electronic horizon data. The electronic horizon data is understood as driving plan data within a range from the point where the vehicle 10 is located to the horizon. Horizon may be understood as a point in front of a preset distance from a point where the vehicle 10 is located based on a preset driving route. It may mean a point at which the vehicle 10 can reach after a predetermined time from the point.

The electronic horizon data may include horizon map data and horizon pass data.

2.1.1) 호라이즌 맵 데이터2.1.1) Horizon Map Data

The horizon map data may include at least one of topology data, road data, HD map data, and dynamic data. According to an embodiment, the horizon map data may include a plurality of layers. For example, the horizon map data may include one layer matching topology data, a second layer matching road data, a third layer matching HD map data, and a fourth layer matching dynamic data. The horizon map data may further include static object data.

Topology data can be described as a map created by connecting the center of the road. The topology data is suitable for roughly indicating the location of the vehicle, and may be in the form of data mainly used in a navigation for a driver. The topology data may be understood as data about road information excluding information about a lane. The topology data may be generated based on data received from an external server through the communication device 220. The topology data may be based on data stored in at least one memory provided in the vehicle 10.

The road data may include at least one of slope data of a road, curvature data of a road, and speed limit data of a road. The road data may further include overtaking prohibited section data. Road data may be based on data received from an external server through the communication device 220. The road data may be based on data generated by the object detection apparatus 210.

The HD map data includes detailed lane-level topology information of the road, connection information of each lane, and feature information for localization of the vehicle (e.g., traffic signs, lane marking/attributes, road furniture, etc.). I can. The HD map data may be based on data received from an external server through the communication device 220.

The dynamic data may include various dynamic information that may be generated on the road. For example, the dynamic data may include construction information, variable speed lane information, road surface condition information, traffic information, moving object information, and the like. The dynamic data may be based on data received from an external server through the communication device 220. The dynamic data may be based on data generated by the object detection apparatus 210.

The processor 170 may provide map data within a range from the point where the vehicle 10 is located to the horizon.

2.1.2) 호라이즌 패스 데이터2.1.2) Horizon Pass Data

The horizon pass data may be described as a trajectory that the vehicle 10 can take within a range from the point where the vehicle 10 is located to the horizon. The horizon pass data may include data representing a relative probability of selecting any one road at a decision point (eg, a fork, a fork, an intersection, etc.). The relative probability can be calculated based on the time it takes to reach the final destination. For example, at the decision point, if the first road is selected and the time it takes to reach the final destination is less than the second road is selected, the probability of selecting the first road is less than the probability of selecting the second road. It can be calculated higher.

Horizon pass data may include a main pass and a sub pass. The main path can be understood as a trajectory connecting roads with a high relative probability to be selected. The sub-path may be branched at at least one decision point on the main path. The sub-path may be understood as a trajectory connecting at least one road having a low relative probability to be selected from at least one decision point on the main path.

3) 제어 신호 생성 동작3) Control signal generation operation

The processor 170 may perform a control signal generation operation. The processor 170 may generate a control signal based on electronic horizon data. For example, the processor 170 may generate at least one of a powertrain control signal, a brake device control signal, and a steering device control signal based on the electronic horizon data.

The processor 170 may transmit the generated control signal to the driving control device 250 through the interface unit 180. The drive control device 250 may transmit a control signal to at least one of the power train 251, the brake device 252, and the steering device 253.

[Description of Fig. 9-10] Cabin

캐빈Cabin

9 is a view showing the interior of a vehicle according to an embodiment of the present specification. 10 is a block diagram referenced to describe a vehicle cabin system according to an exemplary embodiment of the present specification.

(1) 캐빈의 구성 요소(1) Cabin components

9 to 10, the vehicle cabin system 300 (hereinafter, the cabin system) may be defined as a convenience system for a user using the vehicle 10. The cabin system 300 may be described as a top-level system including a display system 350, a cargo system 355, a seat system 360, and a payment system 365. The cabin system 300 includes a main controller 370, a memory 340, an interface unit 380, a power supply unit 390, an input device 310, an imaging device 320, a communication device 330, and a display system. 350, a cargo system 355, a seat system 360, and a payment system 365. Depending on the embodiment, the cabin system 300 may further include other components in addition to the components described herein, or may not include some of the described components.

1) 메인 컨트롤러1) Main controller

The main controller 370 is electrically connected to the input device 310, the communication device 330, the display system 350, the cargo system 355, the seat system 360, and the payment system 365 to exchange signals. can do. The main controller 370 may control the input device 310, the communication device 330, the display system 350, the cargo system 355, the seat system 360, and the payment system 365. The main controller 370 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, It may be implemented using at least one of controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

The main controller 370 may be configured with at least one sub-controller. Depending on the embodiment, the main controller 370 may include a plurality of sub-controllers. Each of the plurality of sub-controllers may individually control devices and systems included in the grouped cabin system 300. Devices and systems included in the cabin system 300 may be grouped for each function or may be grouped based on seatable seats.

The main controller 370 may include at least one processor 371. In FIG. 10, the main controller 370 is illustrated as including one processor 371, but the main controller 371 may include a plurality of processors. The processor 371 may be classified as one of the above-described sub-controllers.

The processor 371 may receive a signal, information, or data from a user terminal through the communication device 330. The user terminal may transmit signals, information, or data to the cabin system 300.

The processor 371 may specify a user based on image data received from at least one of an internal camera and an external camera included in the imaging device. The processor 371 may specify a user by applying an image processing algorithm to image data. For example, the processor 371 may compare information received from the user terminal with image data to identify a user. For example, the information may include at least one of user's route information, body information, passenger information, luggage information, location information, preferred content information, preferred food information, disability information, and usage history information. .

The main controller 370 may include an artificial intelligence agent 372. The artificial intelligence agent 372 may perform machine learning based on data acquired through the input device 310. The artificial intelligence agent 372 may control at least one of the display system 350, the cargo system 355, the seat system 360, and the payment system 365 based on the machine-learned result.

2) 필수 구성 요소2) Essential components

The memory 340 is electrically connected to the main controller 370. The memory 340 may store basic data for a unit, control data for controlling the operation of the unit, and input/output data. The memory 340 may store data processed by the main controller 370. In terms of hardware, the memory 340 may be configured with at least one of ROM, RAM, EPROM, flash drive, and hard drive. The memory 340 may store various data for the overall operation of the cabin system 300, such as a program for processing or controlling the main controller 370. The memory 340 may be implemented integrally with the main controller 370.

The interface unit 380 may exchange signals with at least one electronic device provided in the vehicle 10 by wire or wirelessly. The interface unit 380 may be configured with at least one of a communication module, a terminal, a pin, a cable, a port, a circuit, an element, and a device.

The power supply unit 390 may supply power to the cabin system 300. The power supply unit 390 may receive power from a power source (eg, a battery) included in the vehicle 10 and supply power to each unit of the cabin system 300. The power supply unit 390 may be operated according to a control signal provided from the main controller 370. For example, the power supply unit 390 may be implemented as a switched-mode power supply (SMPS).

The cabin system 300 may include at least one printed circuit board (PCB). The main controller 370, the memory 340, the interface unit 380, and the power supply unit 390 may be mounted on at least one printed circuit board.

3) 입력 장치3) input device

The input device 310 may receive a user input. The input device 310 may convert a user input into an electrical signal. The electrical signal converted by the input device 310 may be converted into a control signal and provided to at least one of the display system 350, the cargo system 355, the seat system 360, and the payment system 365. At least one processor included in the main controller 370 or the cabin system 300 may generate a control signal based on an electrical signal received from the input device 310.

The input device 310 may include at least one of a touch input unit, a gesture input unit, a mechanical input unit, and a voice input unit. The touch input unit may convert a user's touch input into an electrical signal. The touch input unit may include at least one touch sensor to detect a user's touch input. Depending on the embodiment, the touch input unit is integrally formed with at least one display included in the display system 350, thereby implementing a touch screen. Such a touch screen may provide an input interface and an output interface between the cabin system 300 and a user. The gesture input unit may convert a user's gesture input into an electrical signal. The gesture input unit may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input. According to an embodiment, the gesture input unit may detect a user's 3D gesture input. To this end, the gesture input unit may include a light output unit that outputs a plurality of infrared light or a plurality of image sensors. The gesture input unit may detect a user's 3D gesture input through a time of flight (TOF) method, a structured light method, or a disparity method. The mechanical input unit may convert a user's physical input (eg, pressing or rotating) through a mechanical device into an electrical signal. The mechanical input unit may include at least one of a button, a dome switch, a jog wheel, and a jog switch. Meanwhile, the gesture input unit and the mechanical input unit may be integrally formed. For example, the input device 310 may include a gesture sensor, and may include a jog dial device formed to be retractable from a portion of a surrounding structure (eg, at least one of a seat, an armrest, and a door). . When the jog dial device is flat with the surrounding structure, the jog dial device may function as a gesture input unit. When the jog dial device protrudes from the surrounding structure, the jog dial device may function as a mechanical input unit. The voice input unit may convert a user's voice input into an electrical signal. The voice input unit may include at least one microphone. The voice input unit may include a beam foaming microphone.

4) 영상 장치4) Video device

The imaging device 320 may include at least one camera. The imaging device 320 may include at least one of an internal camera and an external camera. The internal camera can take an image inside the cabin. The external camera may capture an image outside the vehicle. The internal camera can acquire an image in the cabin. The imaging device 320 may include at least one internal camera. It is preferable that the imaging device 320 includes a number of cameras corresponding to the number of passengers that can be boarded. The imaging device 320 may provide an image acquired by an internal camera. At least one processor included in the main controller 370 or the cabin system 300 detects the user's motion based on the image acquired by the internal camera, generates a signal based on the detected motion, and generates a display system. It may be provided to at least one of 350, the cargo system 355, the seat system 360, and the payment system 365. The external camera may acquire an image outside the vehicle. The imaging device 320 may include at least one external camera. It is preferable that the imaging device 320 includes a number of cameras corresponding to the boarding door. The imaging device 320 may provide an image acquired by an external camera. At least one processor included in the main controller 370 or the cabin system 300 may acquire user information based on an image acquired by an external camera. At least one processor included in the main controller 370 or the cabin system 300 authenticates the user based on user information, or authenticates the user's body information (eg, height information, weight information, etc.), or Passenger information, user's luggage information, etc. can be obtained.

5) 통신 장치5) Communication device

The communication device 330 can wirelessly exchange signals with an external device. The communication device 330 may exchange signals with an external device through a network network, or may directly exchange signals with an external device. The external device may include at least one of a server, a mobile terminal, and another vehicle. The communication device 330 may exchange signals with at least one user terminal. The communication device 330 may include at least one of an antenna, a radio frequency (RF) circuit capable of implementing at least one communication protocol, and an RF element to perform communication. Depending on the embodiment, the communication device 330 may use a plurality of communication protocols. The communication device 330 may switch the communication protocol according to the distance to the mobile terminal.

6) 디스플레이 시스템6) Display system

The display system 350 may display a graphic object. The display system 350 may include at least one display device. For example, the display system 350 may include a first display device 410 that can be commonly used and a second display device 420 that can be used individually.

6.1) 공용 디스플레이 장치6.1) Common display device

The first display device 410 may include at least one display 411 that outputs visual content. The display 411 included in the first display device 410 is a flat panel display. It may be implemented as at least one of a curved display, a rollable display, and a flexible display. For example, the first display device 410 may include a first display 411 positioned at the rear of a seat and formed to be in and out of a cabin, and a first mechanism for moving the first display 411. The first display 411 may be disposed in a slot formed in the main frame of the sheet so as to be retractable. According to an embodiment, the first display device 410 may further include a flexible area control mechanism. The first display may be formed to be flexible, and the flexible area of the first display may be adjusted according to the user's position. For example, the first display device 410 may include a second display positioned on a ceiling in a cabin and formed to be rollable, and a second mechanism for winding or unwinding the second display. The second display may be formed to enable screen output on both sides. For example, the first display device 410 may include a third display positioned on a ceiling in a cabin and formed to be flexible, and a third mechanism for bending or unfolding the third display. According to an embodiment, the display system 350 may further include at least one processor that provides a control signal to at least one of the first display device 410 and the second display device 420. The processor included in the display system 350 may generate a control signal based on a signal received from at least one of the main controller 370, the input device 310, the imaging device 320, and the communication device 330. I can.

The display area of the display included in the first display device 410 may be divided into a first area 411a and a second area 411b. The first area 411a may define content as a display area. For example, the first area 411 may display at least one of entertainment content (eg, movies, sports, shopping, music, etc.), video conferences, food menus, and graphic objects corresponding to the augmented reality screen. I can. The first area 411a may display a graphic object corresponding to driving situation information of the vehicle 10. The driving situation information may include at least one of object information outside the vehicle, navigation information, and vehicle status information. The object information outside the vehicle may include information on the presence or absence of an object, location information of the object, distance information between the vehicle 300 and the object, and information on a relative speed between the vehicle 300 and the object. The navigation information may include at least one of map information, set destination information, route information according to the destination setting, information on various objects on the route, lane information, and current location information of the vehicle. The vehicle status information includes vehicle attitude information, vehicle speed information, vehicle tilt information, vehicle weight information, vehicle direction information, vehicle battery information, vehicle fuel information, vehicle tire pressure information, vehicle steering information , Vehicle interior temperature information, vehicle interior humidity information, pedal position information, vehicle engine temperature information, and the like. The second area 411b may be defined as a user interface area. For example, the second area 411b may output an artificial intelligence agent screen. According to an embodiment, the second area 411b may be located in an area divided by a sheet frame. In this case, the user can view the content displayed in the second area 411b between the plurality of sheets. According to an embodiment, the first display device 410 may provide holographic content. For example, the first display device 410 may provide holographic content for each of a plurality of users so that only a user who has requested the content can view the content.

6.2) 개인용 디스플레이 장치6.2) Personal display device

The second display device 420 may include at least one display 421. The second display device 420 may provide the display 421 at a location where only individual passengers can check the display contents. For example, the display 421 may be disposed on the arm rest of the seat. The second display device 420 may display a graphic object corresponding to the user's personal information. The second display device 420 may include a number of displays 421 corresponding to the number of persons allowed to ride. The second display device 420 may implement a touch screen by forming a layer structure or integrally with the touch sensor. The second display device 420 may display a graphic object for receiving a user input for seat adjustment or room temperature adjustment.

7) 카고 시스템7) Cargo system

The cargo system 355 may provide a product to a user according to a user's request. The cargo system 355 may be operated based on an electrical signal generated by the input device 310 or the communication device 330. The cargo system 355 may include a cargo box. The cargo box may be concealed in a portion of the lower portion of the seat while the goods are loaded. When an electrical signal based on a user input is received, the cargo box may be exposed as a cabin. The user can select a necessary product from among the items loaded in the exposed cargo box. The cargo system 355 may include a sliding moving mechanism and a product pop-up mechanism to expose a cargo box according to a user input. The cargo system 355 may include a plurality of cargo boxes to provide various types of goods. A weight sensor for determining whether to be provided for each product may be built into the cargo box.

8) 시트 시스템8) Seat system

The sheet system 360 may provide a user with a customized sheet to the user. The seat system 360 may be operated based on an electrical signal generated by the input device 310 or the communication device 330. The seat system 360 may adjust at least one element of the seat based on the acquired user body data. The seat system 360 may include a user detection sensor (eg, a pressure sensor) to determine whether the user is seated. The seat system 360 may include a plurality of seats each of which a plurality of users can seat. Any one of the plurality of sheets may be disposed to face at least the other. At least two users inside the cabin may sit facing each other.

9) 페이먼트 시스템9) Payment system

The payment system 365 may provide a payment service to a user. The payment system 365 may be operated based on an electrical signal generated by the input device 310 or the communication device 330. The payment system 365 may calculate a price for at least one service used by the user and request that the calculated price be paid.

[Description of Fig. 11] Autonomous driving scenario

(2) 자율 주행 차량 이용 시나리오(2) Scenarios for using autonomous vehicles

1) 목적지 예측 시나리오1) Destination prediction scenario

The first scenario S111 is a user's destination prediction scenario. The user terminal may install an application capable of interworking with the cabin system 300. The user terminal may predict the destination of the user based on the user's contextual information through the application. The user terminal may provide information on empty seats in the cabin through an application.

2) 캐빈 인테리어 레이아웃 준비 시나리오2) Cabin interior layout preparation scenario

The second scenario S112 is a cabin interior layout preparation scenario. The cabin system 300 may further include a scanning device for acquiring data on a user located outside the vehicle 300. The scanning device may scan the user to obtain body data and baggage data of the user. The user's body data and baggage data can be used to set the layout. The user's body data may be used for user authentication. The scanning device may include at least one image sensor. The image sensor may acquire a user image using light in the visible or infrared band.

The seat system 360 may set a layout in the cabin based on at least one of a user's body data and baggage data. For example, the seat system 360 may provide a luggage storage space or a car seat installation space.

3) 사용자 환영 시나리오3) User welcome scenario

The third scenario S113 is a user welcome scenario. The cabin system 300 may further include at least one guide light. The guide light may be disposed on the floor in the cabin. When a user's boarding is detected, the cabin system 300 may output a guide light to allow the user to sit on a preset seat among a plurality of seats. For example, the main controller 370 may implement a moving light by sequentially lighting a plurality of light sources according to time from an opened door to a preset user seat.

4) 시트 조절 서비스 시나리오4) Seat adjustment service scenario

The fourth scenario S114 is a seat adjustment service scenario. The seat system 360 may adjust at least one element of a seat matching the user based on the acquired body information.

5) 개인 컨텐츠 제공 시나리오5) Personal content provision scenario

The fifth scenario S115 is a personal content providing scenario. The display system 350 may receive user personal data through the input device 310 or the communication device 330. The display system 350 may provide content corresponding to user personal data.

6) 상품 제공 시나리오6) Product provision scenario

The sixth scenario S116 is a product provision scenario. The cargo system 355 may receive user data through the input device 310 or the communication device 330. The user data may include user preference data and user destination data. The cargo system 355 may provide a product based on user data.

7) 페이먼트 시나리오7) Payment scenario

The seventh scenario S117 is a payment scenario. The payment system 365 may receive data for price calculation from at least one of the input device 310, the communication device 330, and the cargo system 355. The payment system 365 may calculate a vehicle usage price of the user based on the received data. The payment system 365 may request payment of a fee from a user (eg, a user's mobile terminal) at the calculated price.

8) 사용자의 디스플레이 시스템 제어 시나리오8) User's display system control scenario

The eighth scenario S118 is a user's display system control scenario. The input device 310 may receive a user input in at least one form and convert it into an electrical signal. The display system 350 may control displayed content based on an electrical signal.

9) AI 에이전트 시나리오9) AI agent scenario

The ninth scenario S119 is a multi-channel artificial intelligence (AI) agent scenario for a plurality of users. The artificial intelligence agent 372 may classify a user input for each of a plurality of users. The artificial intelligence agent 372 is at least one of the display system 350, the cargo system 355, the seat system 360, and the payment system 365 based on the electrical signals converted from a plurality of user individual user inputs. Can be controlled.

10) 복수 사용자를 위한 멀티미디어 컨텐츠 제공 시나리오10) Scenario for providing multimedia contents for multiple users

The tenth scenario S120 is a scenario for providing multimedia contents targeting a plurality of users. The display system 350 may provide content that all users can watch together. In this case, the display system 350 may individually provide the same sound to a plurality of users through speakers provided for each sheet. The display system 350 may provide content that can be individually viewed by a plurality of users. In this case, the display system 350 may provide individual sounds through speakers provided for each sheet.

11) 사용자 안전 확보 시나리오11) User safety security scenario

The eleventh scenario S121 is a user safety securing scenario. When obtaining information on objects around the vehicle that threatens the user, the main controller 370 may control to output an alarm for objects around the vehicle through the display system 350.

12) 소지품 분실 예방 시나리오12) Loss of belongings prevention scenario

The twelfth scenario S122 is a scenario for preventing the loss of belongings by the user. The main controller 370 may acquire data on the user's belongings through the input device 310. The main controller 370 may acquire motion data of a user through the input device 310. The main controller 370 may determine whether the user leaves the belongings and gets off the vehicle based on the data and movement data on the belongings. The main controller 370 may control an alarm for belongings to be output through the display system 350.

13) 하차 리포트 시나리오13) Alight Report Scenario

The thirteenth scenario S123 is a getting off report scenario. The main controller 370 may receive a user's getting off data through the input device 310. After getting off the user, the main controller 370 may provide report data according to the getting off to the user's mobile terminal through the communication device 330. The report data may include data on the total usage fee of the vehicle 10.

[Description of Fig. 12-14] AI device

Artificial intelligence refers to the field of researching artificial intelligence or the methodology that can create it, and machine learning (Machine Learning) refers to the field of studying methodologies to define and solve various problems dealt with in the field of artificial intelligence. do. Machine learning is also defined as an algorithm that improves the performance of a task through continuous experience.

An artificial neural network (ANN) is a model used in machine learning, and may refer to an overall model with problem-solving capabilities, which is composed of artificial neurons (nodes) that form a network by combining synapses. The artificial neural network may be defined by a connection pattern between neurons of different layers, a learning process for updating model parameters, and an activation function for generating an output value.

The artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons, and the artificial neural network may include neurons and synapses connecting neurons. In an artificial neural network, each neuron can output a function of an activation function for input signals, weights, and biases input through synapses.

Model parameters refer to parameters determined through learning, and include weights of synaptic connections and biases of neurons. In addition, the hyperparameter refers to a parameter that must be set before learning in a machine learning algorithm, and includes a learning rate, number of iterations, mini-batch size, and initialization function.

The purpose of learning the artificial neural network can be seen as determining the model parameters that minimize the loss function. The loss function can be used as an index to determine an optimal model parameter in the learning process of the artificial neural network.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning according to the learning method.

Supervised learning refers to a method of training an artificial neural network when a label for training data is given, and a label indicates the correct answer (or result value) that the artificial neural network must infer when training data is input to the artificial neural network. It can mean. Unsupervised learning may mean a method of training an artificial neural network in a state in which a label for training data is not given. Reinforcement learning may mean a learning method in which an agent defined in a certain environment learns to select an action or sequence of actions that maximizes the cumulative reward in each state.

Among artificial neural networks, machine learning implemented as a deep neural network (DNN) including a plurality of hidden layers is sometimes referred to as deep learning (deep learning), and deep learning is a part of machine learning. Hereinafter, machine learning is used in the sense including deep learning.

<Robot>

A robot may refer to a machine that automatically processes or operates a task given by its own capabilities. In particular, a robot having a function of recognizing the environment and performing an operation by self-determining may be referred to as an intelligent robot.

Robots can be classified into industrial, medical, household, military, etc. depending on the purpose or field of use.

The robot may be provided with a driving unit including an actuator or a motor to perform various physical operations such as moving a robot joint. In addition, the movable robot includes a wheel, a brake, a propeller, and the like in a driving unit, and can travel on the ground or fly in the air through the driving unit.

<Self-Driving, Autonomous-Driving>

Autonomous driving refers to self-driving technology, and autonomous driving vehicle refers to a vehicle that is driven without a user's manipulation or with a user's minimal manipulation.

For example, in autonomous driving, a technology that maintains a driving lane, a technology that automatically adjusts the speed such as adaptive cruise control, a technology that automatically travels along a specified route, and a technology that automatically sets a route when a destination is set, etc. All of these can be included.

The vehicle includes all of a vehicle including only an internal combustion engine, a hybrid vehicle including an internal combustion engine and an electric motor, and an electric vehicle including only an electric motor, and may include not only automobiles, but also trains and motorcycles.

In this case, the autonomous vehicle can be viewed as a robot having an autonomous driving function.

Augmented reality collectively refers to virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides only CG images of real-world objects or backgrounds, AR technology provides virtually created CG images on top of real-world objects, and MR technology is a computer that mixes and combines virtual objects in the real world. It's a graphic technology.

MR technology is similar to AR technology in that it shows real and virtual objects together. However, in AR technology, a virtual object is used in a form that complements a real object, whereas in MR technology, a virtual object and a real object are used with equal characteristics.

XR technology can be applied to HMD (Head-Mount Display), HUD (Head-Up Display), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc. It can be called as.

The AI device 100 includes a TV, a projector, a mobile phone, a smartphone, a desktop computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a tablet PC, a wearable device, and a set-top box (STB). ), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, a digital signage, a robot, a vehicle, and the like.

Referring to FIG. 12, the terminal 100 includes a communication unit 110, an input unit 120, a running processor 130, a sensing unit 140, an output unit 150, a memory 170, and a processor 180. Can include.

The communication unit 110 may transmit and receive data with external devices such as other AI devices 100a to 100e or the AI server 200 using wired/wireless communication technology. For example, the communication unit 110 may transmit and receive sensor information, a user input, a learning model, and a control signal with external devices.

At this time, communication technologies used by the communication unit 110 include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), and Wireless-Fidelity (Wi-Fi). ), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, and Near Field Communication (NFC).

The input unit 120 may acquire various types of data.

In this case, the input unit 120 may include a camera for inputting an image signal, a microphone for receiving an audio signal, a user input unit for receiving information from a user, and the like. Here, by treating a camera or a microphone as a sensor, a signal obtained from the camera or a microphone may be referred to as sensing data or sensor information.

The input unit 120 may acquire training data for model training and input data to be used when acquiring an output by using the training model. The input unit 120 may obtain unprocessed input data, and in this case, the processor 180 or the running processor 130 may extract an input feature as a preprocess for the input data.

The learning processor 130 may train a model composed of an artificial neural network by using the training data. Here, the learned artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for a decision to perform a certain operation.

In this case, the learning processor 130 may perform AI processing together with the learning processor 240 of the AI server 200.

In this case, the learning processor 130 may include a memory integrated or implemented in the AI device 100. Alternatively, the learning processor 130 may be implemented using the memory 170, an external memory directly coupled to the AI device 100, or a memory maintained in an external device.

The sensing unit 140 may acquire at least one of internal information of the AI device 100, information on the surrounding environment of the AI device 100, and user information by using various sensors.

At this time, the sensors included in the sensing unit 140 include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, and a lidar. , Radar, etc.

The output unit 150 may generate output related to visual, auditory or tactile sensations.

In this case, the output unit 150 may include a display unit outputting visual information, a speaker outputting auditory information, a haptic module outputting tactile information, and the like.

The memory 170 may store data supporting various functions of the AI device 100. For example, the memory 170 may store input data, learning data, a learning model, and a learning history acquired from the input unit 120.

The processor 180 may determine at least one executable operation of the AI device 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. In addition, the processor 180 may perform the determined operation by controlling the components of the AI device 100.

To this end, the processor 180 may request, search, receive, or utilize data from the learning processor 130 or the memory 170, and perform a predicted or desirable operation among the at least one executable operation. The components of the AI device 100 can be controlled to run.

In this case, when connection of an external device is required to perform the determined operation, the processor 180 may generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

The processor 180 may obtain intention information for a user input and determine a user's requirement based on the obtained intention information.

In this case, the processor 180 uses at least one of a Speech To Text (STT) engine for converting a speech input into a character string or a Natural Language Processing (NLP) engine for obtaining intention information of a natural language. Intention information corresponding to the input can be obtained.

At this time, at least one or more of the STT engine and the NLP engine may be composed of an artificial neural network, at least partially trained according to a machine learning algorithm. And, at least one of the STT engine or the NLP engine is learned by the learning processor 130, learning by the learning processor 240 of the AI server 200, or learned by distributed processing thereof. Can be.

The processor 180 collects history information including user feedback on the operation content or operation of the AI device 100 and stores it in the memory 170 or the learning processor 130, or the AI server 200 Can be transferred to an external device. The collected history information can be used to update the learning model.

The processor 180 may control at least some of the components of the AI device 100 in order to drive the application program stored in the memory 170. Furthermore, in order to drive the application program, the processor 180 may operate by combining two or more of the constituent elements included in the AI device 100 with each other.

Referring to FIG. 13, the AI server 200 may refer to a device that trains an artificial neural network using a machine learning algorithm or uses the learned artificial neural network. Here, the AI server 200 may be configured with a plurality of servers to perform distributed processing, or may be defined as a 5G network. In this case, the AI server 200 may be included as a part of the AI device 100 to perform at least a part of AI processing together.

The AI server 200 may include a communication unit 210, a memory 230, a learning processor 240, a processor 260, and the like.

The communication unit 210 may transmit and receive data with an external device such as the AI device 100.

The memory 230 may include a model storage unit 231. The model storage unit 231 may store a model (or artificial neural network, 231a) being trained or trained through the learning processor 240.

The learning processor 240 may train the artificial neural network 231a using the training data. The learning model may be used while being mounted on the AI server 200 of an artificial neural network, or may be mounted on an external device such as the AI device 100 and used.

The learning model can be implemented in hardware, software, or a combination of hardware and software. When part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in the memory 230.

The processor 260 may infer a result value for new input data using the learning model, and generate a response or a control command based on the inferred result value.

Referring to FIG. 14, the AI system 1 includes at least one of an AI server 200, a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e. It is connected with this cloud network 10. Here, the robot 100a to which the AI technology is applied, the autonomous vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e may be referred to as the AI devices 100a to 100e.

The cloud network 10 may constitute a part of the cloud computing infrastructure or may mean a network that exists in the cloud computing infrastructure. Here, the cloud network 10 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

That is, the devices 100a to 100e and 200 constituting the AI system 1 may be connected to each other through the cloud network 10. In particular, the devices 100a to 100e and 200 may communicate with each other through a base station, but may directly communicate with each other without through a base station.

The AI server 200 may include a server that performs AI processing and a server that performs an operation on big data.

The AI server 200 includes at least one of a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e, which are AI devices constituting the AI system 1 It is connected through the cloud network 10 and may help at least part of the AI processing of the connected AI devices 100a to 100e.

In this case, the AI server 200 may train an artificial neural network according to a machine learning algorithm in place of the AI devices 100a to 100e, and may directly store the learning model or transmit it to the AI devices 100a to 100e.

At this time, the AI server 200 receives input data from the AI devices 100a to 100e, infers a result value for the received input data using a learning model, and generates a response or control command based on the inferred result value. It can be generated and transmitted to the AI devices 100a to 100e.

Alternatively, the AI devices 100a to 100e may infer a result value for input data using a direct learning model, and may generate a response or a control command based on the inferred result value.

Hereinafter, various embodiments of the AI devices 100a to 100e to which the above-described technology is applied will be described. Here, the AI devices 100a to 100e shown in FIG. 14 can be seen as a specific example of the AI device 100 shown in FIG. 12.

<AI+robot>

The robot 100a is applied with AI technology and may be implemented as a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, and the like.

The robot 100a may include a robot control module for controlling an operation, and the robot control module may refer to a software module or a chip implementing the same as hardware.

The robot 100a acquires status information of the robot 100a by using sensor information acquired from various types of sensors, detects (recognizes) the surrounding environment and objects, generates map data, or moves and travels. You can decide on a plan, decide on a response to user interaction, or decide on an action.

Here, the robot 100a may use sensor information obtained from at least one sensor among a lidar, a radar, and a camera in order to determine a moving route and a driving plan.

The robot 100a may perform the above-described operations using a learning model composed of at least one artificial neural network. For example, the robot 100a may recognize a surrounding environment and an object using a learning model, and may determine an operation using the recognized surrounding environment information or object information. Here, the learning model may be directly learned by the robot 100a or learned by an external device such as the AI server 200.

At this time, the robot 100a may perform an operation by generating a result using a direct learning model, but it transmits sensor information to an external device such as the AI server 200 and performs the operation by receiving the result generated accordingly. You may.

The robot 100a determines a movement route and a driving plan using at least one of map data, object information detected from sensor information, or object information obtained from an external device, and controls the driving unit to determine the determined movement path and travel plan. Accordingly, the robot 100a can be driven.

The map data may include object identification information on various objects arranged in a space in which the robot 100a moves. For example, the map data may include object identification information on fixed objects such as walls and doors and movable objects such as flower pots and desks. In addition, the object identification information may include a name, type, distance, location, and the like.

In addition, the robot 100a may perform an operation or run by controlling the driving unit based on the user's control/interaction. In this case, the robot 100a may acquire interaction intention information according to a user's motion or voice speech, and determine a response based on the obtained intention information to perform the operation.

The autonomous vehicle 100b may be implemented as a mobile robot, vehicle, or unmanned aerial vehicle by applying AI technology.

The autonomous driving vehicle 100b may include an autonomous driving control module for controlling an autonomous driving function, and the autonomous driving control module may refer to a software module or a chip implementing the same as hardware. The autonomous driving control module may be included inside as a configuration of the autonomous driving vehicle 100b, but may be configured as separate hardware and connected to the exterior of the autonomous driving vehicle 100b.

The autonomous driving vehicle 100b acquires status information of the autonomous driving vehicle 100b using sensor information obtained from various types of sensors, detects (recognizes) surrounding environments and objects, or generates map data, It is possible to determine a travel route and a driving plan, or to determine an action.

Here, the autonomous vehicle 100b may use sensor information obtained from at least one sensor from among a lidar, a radar, and a camera, similar to the robot 100a, in order to determine a moving route and a driving plan.

In particular, the autonomous vehicle 100b may recognize an environment or object in an area where the view is obscured or an area greater than a certain distance by receiving sensor information from external devices, or may receive information directly recognized from external devices. .

The autonomous vehicle 100b may perform the above-described operations using a learning model composed of at least one artificial neural network. For example, the autonomous vehicle 100b may recognize a surrounding environment and an object using a learning model, and may determine a driving movement using the recognized surrounding environment information or object information. Here, the learning model may be directly learned by the autonomous vehicle 100b or learned by an external device such as the AI server 200.

At this time, the autonomous vehicle 100b may perform an operation by generating a result using a direct learning model, but it operates by transmitting sensor information to an external device such as the AI server 200 and receiving the result generated accordingly. You can also do

The autonomous vehicle 100b determines a movement path and a driving plan using at least one of map data, object information detected from sensor information, or object information obtained from an external device, and controls the driving unit to determine the determined movement path and driving. The autonomous vehicle 100b can be driven according to a plan.

The map data may include object identification information on various objects arranged in a space (eg, a road) in which the autonomous vehicle 100b travels. For example, the map data may include object identification information on fixed objects such as streetlights, rocks, and buildings and movable objects such as vehicles and pedestrians. In addition, the object identification information may include a name, type, distance, location, and the like.

In addition, the autonomous vehicle 100b may perform an operation or drive by controlling a driving unit based on a user's control/interaction. In this case, the autonomous vehicle 100b may obtain information on intention of interaction according to a user's motion or voice utterance, and determine a response based on the obtained intention information to perform the operation.

<AI+XR>

The XR device 100c is applied with AI technology, such as HMD (Head-Mount Display), HUD (Head-Up Display) provided in the vehicle, TV, mobile phone, smart phone, computer, wearable device, home appliance, digital signage. , Vehicle, can be implemented as a fixed robot or a mobile robot.

The XR device 100c analyzes 3D point cloud data or image data acquired through various sensors or from an external device to generate location data and attribute data for 3D points, thereby providing information on surrounding spaces or real objects. The XR object to be acquired and output can be rendered and output. For example, the XR apparatus 100c may output an XR object including additional information on the recognized object in correspondence with the recognized object.

The XR device 100c may perform the above-described operations using a learning model composed of at least one artificial neural network. For example, the XR apparatus 100c may recognize a real object from 3D point cloud data or image data using a learning model, and may provide information corresponding to the recognized real object. Here, the learning model may be directly learned by the XR device 100c or learned by an external device such as the AI server 200.

At this time, the XR device 100c may directly generate a result using a learning model to perform an operation, but transmits sensor information to an external device such as the AI server 200 and receives the generated result to perform the operation. You can also do it.

<AI+robot+autonomous driving>

The robot 100a may be implemented as a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, etc. by applying AI technology and autonomous driving technology.

The robot 100a to which AI technology and autonomous driving technology are applied may refer to a robot having an autonomous driving function or a robot 100a interacting with the autonomous driving vehicle 100b.

The robot 100a having an autonomous driving function may collectively refer to devices that move by themselves according to a given movement line without the user's control or by determining the movement line by themselves.

The robot 100a having an autonomous driving function and the autonomous driving vehicle 100b may use a common sensing method to determine one or more of a moving route or a driving plan. For example, the robot 100a having an autonomous driving function and the autonomous driving vehicle 100b may determine one or more of a movement route or a driving plan using information sensed through a lidar, a radar, and a camera.

The robot 100a interacting with the autonomous driving vehicle 100b exists separately from the autonomous driving vehicle 100b and is linked to an autonomous driving function inside or outside the autonomous driving vehicle 100b, or ), you can perform an operation associated with the user on board.

At this time, the robot 100a interacting with the autonomous driving vehicle 100b acquires sensor information on behalf of the autonomous driving vehicle 100b and provides it to the autonomous driving vehicle 100b, or acquires sensor information and provides information on the surrounding environment or By generating object information and providing it to the autonomous driving vehicle 100b, it is possible to control or assist the autonomous driving function of the autonomous driving vehicle 100b.

Alternatively, the robot 100a interacting with the autonomous vehicle 100b may monitor a user in the autonomous vehicle 100b or control functions of the autonomous vehicle 100b through interaction with the user. . For example, when it is determined that the driver is in a drowsy state, the robot 100a may activate an autonomous driving function of the autonomous driving vehicle 100b or assist in controlling a driving unit of the autonomous driving vehicle 100b. Here, the functions of the autonomous driving vehicle 100b controlled by the robot 100a may include not only an autonomous driving function, but also functions provided by a navigation system or an audio system provided inside the autonomous driving vehicle 100b.

Alternatively, the robot 100a interacting with the autonomous driving vehicle 100b may provide information or assist a function to the autonomous driving vehicle 100b from outside of the autonomous driving vehicle 100b. For example, the robot 100a may provide traffic information including signal information to the autonomous vehicle 100b, such as a smart traffic light, or interact with the autonomous driving vehicle 100b, such as an automatic electric charger for an electric vehicle. You can also automatically connect an electric charger to the charging port.

<AI+Robot+XR>

The robot 100a may be implemented as a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, a drone, etc. by applying AI technology and XR technology.

The robot 100a to which the XR technology is applied may refer to a robot that is an object of control/interaction in an XR image. In this case, the robot 100a is distinguished from the XR device 100c and may be interlocked with each other.

When the robot 100a, which is the object of control/interaction in the XR image, acquires sensor information from sensors including a camera, the robot 100a or the XR device 100c generates an XR image based on the sensor information. And, the XR device 100c may output the generated XR image. In addition, the robot 100a may operate based on a control signal input through the XR device 100c or a user's interaction.

For example, the user can check the XR image corresponding to the viewpoint of the robot 100a linked remotely through an external device such as the XR device 100c, and adjust the autonomous driving path of the robot 100a through the interaction. , You can control motion or driving, or check information on surrounding objects.

<AI+Autonomous Driving+XR>

The autonomous vehicle 100b may be implemented as a mobile robot, a vehicle, or an unmanned aerial vehicle by applying AI technology and XR technology.

The autonomous driving vehicle 100b to which the XR technology is applied may refer to an autonomous driving vehicle including a means for providing an XR image, or an autonomous driving vehicle that is an object of control/interaction within the XR image. In particular, the autonomous vehicle 100b, which is an object of control/interaction in the XR image, is distinguished from the XR device 100c and may be interlocked with each other.

The autonomous vehicle 100b having a means for providing an XR image may obtain sensor information from sensors including a camera, and may output an XR image generated based on the acquired sensor information. For example, the autonomous vehicle 100b may provide a real object or an XR object corresponding to an object in a screen to the occupant by outputting an XR image with a HUD.

In this case, when the XR object is output to the HUD, at least a part of the XR object may be output so that it overlaps the actual object facing the occupant's gaze. On the other hand, when the XR object is output on a display provided inside the autonomous vehicle 100b, at least a part of the XR object may be output to overlap an object in the screen. For example, the autonomous vehicle 100b may output XR objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, and buildings.

When the autonomous driving vehicle 100b, which is the object of control/interaction in the XR image, acquires sensor information from sensors including a camera, the autonomous driving vehicle 100b or the XR device 100c is based on the sensor information. An XR image is generated, and the XR device 100c may output the generated XR image. In addition, the autonomous vehicle 100b may operate based on a control signal input through an external device such as the XR device 100c or a user's interaction.

[Description of Figure 15]

As shown in FIG. 15, the method of providing an augmented reality of the autonomous vehicle 10 includes steps S100 (S110, S130), and a detailed description is as follows.

First, a processor of an autonomous vehicle (e.g., the processor 170 of FIG. 7, the main controller 370 of FIG. 10, the processor 371, or the processor 180 of FIG. 12) An Augmented Reality (AR) image may be displayed on the display system 350 of FIG. 10 (S110).

For example, the processor may select a portion of a plurality of displays in the vehicle and display an AR image on the selected partial display. In selecting some of the plurality of displays, the processor may select some of the plurality of displays based on the situation of the device reflected in the AR image. For example, when the AR image indicates an urgent situation of the device, the processor may select all of the plurality of displays and output the AR image through all of the plurality of displays. For example, when the AR image simply informs a safe situation of the device, the processor may select only some displays, not all of the plurality of displays, and output the AR image through the selected partial displays.

For example, the processor may display an AR image only in a partial area even within a selected partial display among a plurality of displays in the vehicle. For example, the processor may display the AR image in a partial area of the entire area of the selected display based on the display format, display position, and display direction of the AR object included in the AR image.

Here, the plurality of displays include infotainment displays, such as a cluster, a passenger display, and a center fascia display, a wind shield, and a rear window. , May include a door window.

As will be described later, each of the plurality of displays may be provided with a crystal sound speaker overlapping with each display.

Subsequently, the processor may output the AR sound through at least a partial area of the entire area of the crystal sound speaker overlapping the display area in which the AR image is displayed (S130).

That is, the processor may output AR sound corresponding to the AR image through at least some speaker areas among the entire areas of the crystal sound speaker overlapping (overlapping) each of the plurality of curved or flat displays.

Here, the crystal sound speaker may be an output unit included in the user interface device 200 of FIG. 6. As described above, the crystal sound speaker may be provided on each display in a form overlapping with each of the plurality of displays.

Hereinafter, a method of outputting AR sound through at least a partial region of the crystal sound speaker will be described in detail.

[Description of Fig. 16]

As shown in FIG. 16, the vehicle 10 may include a plurality of

crystal sound displays

1601, 1602, 1603, 1604, 1605, 1606, 1607, 1608, 1609, 1610, 1611, 1612.

A windshield passenger display 1601 and a windshield driver display 1602 may be disposed on a windshield of a vehicle.

The cluster display 1603 may be disposed in front of the driver's seat even in the vehicle, and the passenger infotainment display 1605 may be disposed in the front of the passenger seat even in the vehicle, and the center fascia display ( The center fascia display 1604 may be disposed between the cluster display 1603 and the passenger infotainment display 1605 even in a vehicle.

The front door window driver display 1607 may be disposed on the window on the driver's seat side, and the front door window passenger display 1606 may be disposed on the window on the passenger side. .

Rear seat displays 1608 and 1609 may be disposed on the rear of the driver's seat and the auxiliary seat in a direction viewed from the rear seat even in the vehicle.

Rear

door window displays

1610 and 1611 may be disposed on a window on the side of the rear seat.

The rear window display 1612 may be disposed on the rear window even in the vehicle.

According to the present specification, each of the crystal sound displays 1601-1612 may have a curved or flat shape, and includes a curved or flat display and a curved or flat crystal sound speaker superimposed in parallel with the display.

Each crystal sound display is at least partially activated and at least partially outputs video/sound according to the control of the processor. Each crystal sound display may output a predetermined sound on an area overlapping with an area in which the AR object is displayed on the display overlapped with each crystal sound speaker, according to the control of the processor.

[Description of Fig. 17]

17, the crystal sound speaker output system provided in the vehicle includes a situation recognition unit 1710, an AR engine 1720, a sound output control unit 1730, an output unit 1740, and a sound DB 1750. Can include. Here, the context recognition unit includes the sensing unit 270 of FIG. 6, at least a part of the sensing unit 140 of FIG. 12, at least a part of the object detection device 210 of FIG. 6, and the processor 170 of FIG. 7, and It may include at least a part of the main controller 370, the processor 371, or the processor 180 of FIG. 12.

Here, the AR engine and the sound output control unit may be included in the processor 170 of FIG. 7, the main controller 370 of FIG. 10, the processor 371, or the processor 180 of FIG. 12 described above. Here, the output unit may be included in the output unit included in the user interface device 200 of FIG. 6 described above. Here, the sound DB may be included in the memory 140 of FIG. 7 and the memory 340 of FIG. 10.

The situation recognition unit may acquire device situation information of the vehicle, and may transmit the acquired device situation information to the AR engine 1720 and the sound output control unit 1730. For example, the device status information may include information related to driving of the vehicle. Specifically, the situation recognition unit may acquire vehicle status information 1711. The context recognition unit may acquire vehicle route information (Navigation) 1712. The situation awareness unit may obtain an autonomous driving state (ADAS) 1713 of the vehicle. The situation recognition unit may acquire vehicle speed, direction, gravity, and acceleration information (IMU) (inertia measurement information) 1714. The situation recognition unit may acquire reference map information (SD Map) 1715 used for autonomous driving of the vehicle. The situation recognition unit may acquire vehicle location information (GPS 1716). The situation recognition unit may acquire a user input 171 of a driver who has boarded the vehicle. In addition, the situation recognition unit may detect an object of attention or a dangerous object outside the vehicle.

The AR engine may generate AR object information and transmit the generated AR object information to the sound output control unit 1730. Here, the AR object information may include an AR image and/or an AR sound. Specifically, the AR engine may predict data used to generate an AR image and/or an AR sound (Data Prediction) 1721. In addition, the AR engine may project the generated AR image onto at least some of the plurality of displays (AR Image Projection) 1722. The AR engine may render 1923 the generated AR object information.

The sound output control unit may output AR object information on the output unit 1740. Specifically, the sound output controller may determine (Display determination) 1173 at least some crystal sound displays for outputting AR object information (AR image/AR sound) among a plurality of crystal sound displays. In addition, the sound output control unit may determine (1732) coordinates of a specific display area/sound output area for outputting AR object information and whether the AR object is moved, among the determined total areas of the crystal sound display (1732). In addition, the sound output control unit may determine (1733) a sound type to be output in the determined specific display area/sound output area. In addition, the sound output control unit may output AR object information on the output unit based on the device situation information of the vehicle transmitted from the situation recognition unit.

The sound DB may store information related to sound to be included in the AR object information, and may be included in the memory 140 of FIG. 7 and the memory 340 of FIG. 10. Information related to sound stored in the sound DB may be read out by the sound output control unit.

The output unit may output AR object information (Crystal Sound Displays) 1741 based on the control of the sound output controller. Here, the output unit may be included in the user interface device 200 of FIG. 6, the interface 180 of FIG. 7, and the display system 350 of FIG. 10.

The output unit may include a crystal sound display 1741 for outputting AR object information transmitted from the sound output control unit 1730. Here, the crystal sound display 1741 may be each of the plurality of crystal sound displays 1601-1612 described above with reference to FIG. 16.

[Description of Figure 18]

As shown in FIG. 18, the processor may first obtain device status information (S1801).

Subsequently, the processor may acquire AR object information including the AR image and AR sound (S1803).

Then, the processor may determine a display to output the AR image based on the device context information (S1805).

Subsequently, the processor may determine the crystal sound speaker to output AR sound and/or at least a partial area of the corresponding crystal sound speaker among the entire areas of the plurality of crystal sound speakers based on the device status information and the display on which the AR image is output ( S1807).

Then, the processor may output the AR image/AR sound through the determined display/crystal sound speaker (S1809).

[Description of Fig. 19]

As illustrated in FIG. 19, the processor may acquire device context information and acquire (generate) AR object information. For example, the processor may obtain information that a dangerous object (truck) exists in the front of the vehicle, and store

AR object information

1901, 1902 in the shape of a point-atom indicating the degree of danger around the dangerous object in the front. Can be generated. The processor may determine the

crystal sound displays

1910 and 1920 for outputting the AR image and AR sound included in the AR object information among the plurality of crystal sound displays based on the device context information. In the example of FIG. 19, the processor may determine the windshield driver display 1602 in front of the driver's seat as a crystal sound display for outputting an AR image and an AR sound.

Subsequently, while displaying the

AR images

1901 and 1902 on the display 1910, the processor displays the AR images on the area overlapping the display area in which the

AR images

1901 and 1902 are displayed among the entire area of the crystal sound speaker 1920. AR sound related to can be output.

For example, the processor may output an AR sound that spreads from the center of the dangerous object where the first AR image 1901 is displayed to the peripheral portion where the second AR image 1902 is displayed.

[Description of Figure 20]

As shown in FIG. 20, the processor may acquire device context information and acquire (generate) AR object information. For example, the processor may obtain information that a danger situation (lane departure) exists in the front side of the vehicle, and along the line shape of the dangerous object in front, the AR object information in the shape of a line indicating the degree of danger in the front direction ( 2001, 2002) can be created. The processor may determine the

crystal sound displays

2010 and 2020 for outputting the AR image and AR sound included in the AR object information among the plurality of crystal sound displays based on the device context information. In the example of FIG. 20, the processor may determine the windshield driver display 1602 in front of the driver's seat as a crystal sound display for outputting an AR image and an AR sound.

Subsequently, while displaying the

AR images

2001 and 2002 on the display 2010, the processor displays the AR images on the area overlapping the display area in which the

AR images

2001 and 2002 are displayed among the entire area of the crystal sound speaker 2020. AR sound related to can be output.

For example, the processor may simultaneously output AR sound over the entire area of the first AR object 2002 in a line shape.

For another example, the processor may output an AR sound reciprocating between a starting point (thick portion) and an overlapping point of the first AR object and an ending point (thin portion) and an overlapping point of the first AR object.

[Description of Figure 21]

As illustrated in FIG. 21, the processor may acquire device context information and acquire (generate) AR object information. For example, the processor may obtain navigation guide information to move to the right front in the vehicle navigation, and AR object information (2101, 2102, 2103) in the shape of a right arrow to guide the navigation guide information in the right and forward direction. Can be created. The processor may determine the

crystal sound displays

2110 and 2120 for outputting the AR image and AR sound included in the AR object information among the plurality of crystal sound displays based on the device context information. In the example of FIG. 21, the processor may determine the windshield driver display 1602 in front of the driver's seat as a crystal sound display for outputting an AR image and an AR sound.

Subsequently, the processor displays the

AR images

2101, 2102, 2103 on the display 2110 and overlaps the display area in which the

AR images

2101, 2102, 2103 are displayed among the entire area of the crystal sound speaker 2120. AR sound related to the AR image can be output on the area.

For example, the processor may simultaneously output AR sound in all areas of the arrow-shaped first AR object 2101, the second AR object 2102, and the third AR object 2103.

For another example, the processor may perform AR sound in the first area overlapped with the first AR object, the second area overlapped with the second AR object, and the third area overlapped with the third AR object among the entire area of the crystal sound speaker. Can be output in sequence. Here, the processor may output a larger AR sound output from the second region than the AR sound output from the first region, and may output a larger AR sound output from the third region than the AR sound output from the second region. have.

[Description of Figure 22]

As shown in FIG. 22, the processor acquires information related to a situation in which the vehicle invades or leaves the left lane of the vehicle, and displays an AR image for notifying a lane invasion or departure situation among a plurality of displays in the vehicle. It can be displayed on the cluster display 2214 closest to the lane. Here, the processor is a lane through one or more

crystal sound speakers

2211, 2212, and 2213 provided within a preset distance from the cluster display on which the AR image is displayed among the plurality of crystal sound speakers (or the closest to the invasion planned lane). AR sound related to an invasion or departure situation can be output. Here, the processor starts to output the AR sound from the first crystal sound speaker 2211 located at the front of the vehicle among the one or more crystal sound speakers, and then outputs the AR sound from the second crystal sound speaker 2212. Then, the third AR crystal sound speaker 2213 may sequentially output AR sounds.

In addition, when the vehicle detects a situation in which another vehicle approaches from the direction indicated by the direction lamp while the turn signal is on, the processor displays an AR image related to the situation where the other vehicle approaches from the direction indicated by the direction lamp among a plurality of displays. Can be output through one or more displays in the direction indicated by the turn signal. Here, the processor may output an AR sound related to a situation in which another vehicle approaches in a direction indicated by a direction light through a crystal sound speaker overlapped with one or more displays on which the AR image is displayed.

[Description of Figure 23]

As shown in FIG. 23, when the attention object 2301 is detected in front of the vehicle, the processor displays the vehicle's windshield display (windshield driver display and windshield passenger display) 2310 of the vehicle as an AR object. You can decide on the display on which to display the information.

Subsequently, the processor may display an AR image 2311 indicating that an object of interest has been detected through the determined windshield display. Here, the processor may output the AR sound through the speaker area 2321 including an area overlapping the area where the AR image is displayed in the windshield display on which the AR image 2311 is displayed.

[Description of Figure 24]

As shown in FIG. 24, the processor may display navigation path guide information 2411 on the windshield display 2410.

In this case, the processor may output AR sound through one or

more speaker areas

2421, 2422, 2423, and 2424 overlapping with one or more of the areas in which the navigation route guide information 2411 is displayed.

Here, among one or more speaker areas, the processor first outputs the AR sound in the first speaker area 2421 located at the bottom of the display, and then outputs the AR sound in the second speaker area 2422 located above, and then The AR sound may be output from the third speaker area 2423 located above, and the AR sound may be sequentially output from the fourth speaker area 2424 located above.

Here, among one or more speaker areas, the processor outputs the AR sound at the largest volume in the first speaker area 2421 located at the bottom of the display, and the AR sound at the second largest volume in the second speaker area 2422 located above the display. Sound is output, and then AR sound is output at the third loudest volume in the third speaker area 2423 located above, and AR sound is output at the fourth loudest volume in the fourth speaker area 2424 located above. I can.

[Description of Figure 25]

As shown in FIG. 25, when a plurality of objects of attention are detected in the left front and right front sides of the vehicle, that is, in two directions, the processor displays the first AR object 2511 on the left side of the windshield display 2510, The second AR object 2512 may be displayed on the right side of the windshield display.

Here, the processor determines a high-risk attention object (e.g., a left attention object) among a plurality of attention objects detected in two directions, left front and right front, and the first AR object in the direction in which the higher-risk attention object is located. May be displayed larger than the second AR object in the direction in which the child of the attention having a lower risk is located.

In addition, the processor outputs the AR sound related to the attention target through a specific speaker area 2521 overlapping the area in which the first AR object in the direction in which the target of interest with higher risk is located or the area around the area where the first AR object is displayed. can do.

[Description of Figure 26]

As shown in FIG. 26, when a plurality of attention targets are detected in the left front and right front sides of the vehicle, that is, in two directions, the processor displays the first AR object 2511 on the left side of the windshield display 2610, The second AR object 2512 may be displayed on the right side of the windshield display.

Here, the processor determines an attention object with a higher risk (eg, an attention object on the left) among a plurality of attention objects detected in two directions, front left and front right, and the first AR object in the direction in which the attention object with higher risk is located. May be displayed larger than the second AR object in the direction in which the child of the attention having a lower risk is located.

In addition, the processor provides the first AR sound related to the attention target through a specific speaker area 2621 overlapping the area in which the first AR object in the direction in which the target of interest is located or the area around the area where the first AR object is displayed. Can be printed. In addition, the processor provides a second AR sound related to the attention target through a specific speaker area 2622 overlapping the area in which the second AR object is displayed or the area surrounding the area in which the second AR object is displayed. Can be printed. Here, the processor may output the first AR sound related to the attention object with a higher risk than the second AR sound related to the attention object with a lower risk.

[Description of Figure 27]

As shown in FIG. 27, the processor acquires information related to a situation in which the vehicle invades or leaves the left lane of the vehicle, and displays an AR image for notifying a lane invasion or departure situation among a plurality of displays in the vehicle. It can be displayed on the cluster display 2701 closest to the lane. Here, the processor includes one or more

crystal sound speakers

2722, 2723, 2724, 2725 provided within a preset distance from the cluster display on which the AR image is displayed among the plurality of crystal sound speakers (or closest to the invasion planned lane). Through this, a first AR sound related to a lane invasion or departure situation may be output.

When a target of attention is detected in the right front of the vehicle along with information related to the situation in which the vehicle invades or leaves the vehicle's left lane, the processor includes a cluster display 2701 closest to the lane to be invaded among the plurality of displays, and While outputting the first AR sound related to a lane invasion or departure situation through one or more

crystal sound speakers

2722, 2723, 2724, 2725, an AR image 2702 indicating that a target of attention exists in the front right is displayed in a cluster display 2701. ), it is possible to output AR sound through the crystal sound speaker 2721 in the front right direction.

[Summary of Examples of this Specification]

Embodiment 1: An autonomous driving vehicle providing augmented reality, comprising: a plurality of displays; A plurality of crystal sound speakers overlapping the plurality of displays; And obtaining device status information of the autonomous vehicle, displaying a first image through a first display area of the plurality of displays based on the device status information, and displaying the plurality of crystal sounds based on the first image. And a processor that outputs a first sound related to the first image through a first speaker area of the speaker.

Embodiment 2: In Embodiment 1, the processor determines at least one display from among the plurality of displays based on the device status information, and the first display area of the at least one display is determined. One image is displayed, and the first speaker area may be an area that at least partially overlaps the first display area.

Embodiment 3: In Embodiment 2, the processor may determine the first display area and the first speaker area based on a location of the attention object outside the vehicle.

Embodiment 4: In Embodiment 2, the processor may determine the first display area and the first speaker area based on a degree of danger of a target of attention outside the vehicle.

Embodiment 5: In Embodiment 1, the processor determines at least one display among the plurality of displays based on the device status information, and displays the first image through the determined at least one display, At least one speaker among the plurality of crystal sound speakers is determined based on a position where the first image is displayed, and the first sound is output through the determined at least one speaker.

Embodiment 6: A method of providing augmented reality by an autonomous vehicle, the method comprising: acquiring device status information of the autonomous vehicle; Displaying a first image through a first display area of a plurality of displays based on the device status information; And outputting a first sound related to the first image through a first speaker area of a plurality of crystal sound speakers overlapping the plurality of displays based on the first image.

Example 7: In Example 6, determining at least one display from among the plurality of displays based on the device status information, and the first image through the first display area of the determined at least one display. And displaying, wherein the first speaker area may be an area at least partially overlapping with the first display area.

Embodiment 8: In Embodiment 7, it may include determining the first display area and the first speaker area based on the position of the attention object outside the vehicle.

Embodiment 9: In Embodiment 7, it may comprise the step of determining the first display area and the first speaker area based on the degree of danger of the attention object outside the vehicle.

Embodiment 10: In Embodiment 6, determining at least one display among the plurality of displays based on the device status information, displaying the first image through the determined at least one display, the second 1 It may be characterized in that it comprises determining at least one speaker among the plurality of crystal sound speakers based on a position where an image is displayed, and outputting the first sound through the determined at least one speaker. have.

Claims

In an autonomous vehicle that provides augmented reality,

A plurality of displays;

A plurality of crystal sound speakers overlapping the plurality of displays; And

Acquiring device status information of the autonomous vehicle,

Displaying a first image through a first display area of the plurality of displays based on the device status information,

And a processor configured to output a first sound related to the first image through a first speaker region of the plurality of crystal sound speakers based on the first image.

Autonomous vehicle.
The method of claim 1,

The processor,

Determining at least one display from among the plurality of displays based on the device context information,

Displaying the first image through the first display area of the determined at least one display,

The first speaker area is an area that at least partially overlaps the first display area,

Autonomous vehicle.
The method of claim 2,

The processor,

It characterized in that determining the first display area and the first speaker area based on the position of the attention object outside the vehicle,

Autonomous vehicle.
The method of claim 2,

The processor,

It characterized in that determining the first display area and the first speaker area based on the degree of danger of the attention object outside the vehicle,

Autonomous vehicle.
The method of claim 1,

The processor,

Determining at least one display from among the plurality of displays based on the device context information,

Displaying the first image through the determined at least one display,

Determining at least one speaker from among the plurality of crystal sound speakers based on a position where the first image is displayed,

Characterized in that outputting the first sound through the determined at least one speaker,

Autonomous vehicle.
In a method for an autonomous vehicle to provide augmented reality,

Acquiring device status information of the autonomous vehicle;

Displaying a first image through a first display area of a plurality of displays based on the device status information; And

And outputting a first sound related to the first image through a first speaker area of a plurality of crystal sound speakers overlapping the plurality of displays based on the first image.

Way.
The method of claim 6,

Determining at least one display from among the plurality of displays based on the device context information, and

And displaying the first image through the first display area of the determined at least one display,

The first speaker area is an area that at least partially overlaps the first display area,

Way.
The method of claim 7,

It characterized in that it comprises the step of determining the first display area and the first speaker area based on the position of the attention object outside the vehicle,

Way.
The method of claim 7,

It characterized in that it comprises the step of determining the first display area and the first speaker area based on the degree of danger of the attention object outside the vehicle,

Way.
The method of claim 6,

Determining at least one display from among the plurality of displays based on the device context information,

Displaying the first image through the determined at least one display,

Determining at least one speaker from among the plurality of crystal sound speakers based on a location where the first image is displayed, and

And outputting the first sound through the determined at least one speaker,

Way.