KR20210089284A - Correction of video for remote control of the vehicle in autonomous driving system - Google Patents

Abstract

Disclosed is a method for correcting an image displayed on a remote control center which remotely controls a vehicle. A remote driving assistance method of an autonomous vehicle according to one embodiment of the present specification may acquire an image through a sensor of the vehicle. The remote driving assistance method of the autonomous vehicle can be used for AI processing to determine perspective distortion of the image. An apparatus for providing an occupant service according to a communication state of the present specification may be linked to an artificial intelligence module, an Unmanned Aerial Vehicle (UAV), a robot, an Augmented Reality (AR) device, a Virtual Reality (VR) device, a device related to a 5G service, and the like.

Description

CORRECTION OF VIDEO FOR REMOTE CONTROL OF THE VEHICLE IN AUTONOMOUS DRIVING SYSTEM}

The present specification relates to an autonomous driving system and a control method thereof, and more particularly, to a method and apparatus for correcting an image displayed on a remote control center that remotely controls a vehicle.

A vehicle may be classified into an internal combustion engine vehicle, an external combustion engine vehicle, a gas turbine vehicle, an electric vehicle, or the like, according to a type of a prime mover used.

An autonomous vehicle refers to a vehicle that can operate by itself without the manipulation of a driver or a passenger, and an autonomous driving system refers to a system that monitors and controls such an autonomous vehicle so that it can operate by itself.

In addition, when autonomous driving cannot be performed or when remote control is required, systems and methods for remotely controlling a vehicle are being studied. In order to remotely control a vehicle, a remote driver must be able to check the surroundings of the remotely controlled vehicle. In addition, a method for enabling a driver located at a long distance to check a more accurate and readable image around the vehicle is sought.

SUMMARY OF THE INVENTION The present specification aims to solve the above-mentioned needs and/or problems.

An object of the present specification is to perform remote driving in a situation where a vehicle needs to perform remote driving auxiliary in order to solve the above problems, and to provide an accurate and readable image to a remote control center performing remote driving there is a purpose to

Another object of the present specification is to provide a corrected image so that a controller of a remote control center can accurately recognize a driving environment by correcting a delay of image information according to a communication state.

In addition, an object of the present specification is to correct the delay of image information by the speed of the vehicle when driving in a rotation section, so that the controller of the remote control center recognizes this, and guides the vehicle occupants to the movement of the center of gravity of the vehicle. The purpose is to provide comfort to passengers.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clear to those of ordinary skill in the art to which the present invention belongs from the detailed description of the invention below. will be able to be understood

A remote driving assistance method of an autonomous vehicle according to an embodiment of the present specification includes: receiving communication information of a region in which the vehicle travels from a server; obtaining at least one of a driving speed and sensing information from the vehicle; determining an image delay indicating a difference between a road viewed from the vehicle and a road image displayed by a remote control center based on at least one of the driving speed and communication information; determining the perspective distance distortion of the image due to the image delay; Compensating for distortion of the image and displaying the corrected image on a display of a remote control center, guiding the driving situation of the vehicle obtained through the sensing information; includes.

In addition, the step of obtaining the sensing information includes: obtaining image information on at least one of a driving direction, left or right side by using a camera of the vehicle; Using a gyro sensor of the vehicle for movement of the vehicle's center of gravity It may include; obtaining equilibrium information.

In addition, the image delay, the remote driving assistance method of the autonomous vehicle, characterized in that it is generated based on at least one of the communication information and the driving speed.

In addition, the step of determining the distortion of the image due to the image delay, extracting feature values from the sensing information; and inputting the feature values to an artificial neural network (DNN) classifier trained to discriminate image distortion caused by the image delay. analyzing an output value of the artificial neural network; It may include; determining the distortion of the image due to the image delay based on the output value of the artificial neural network.

In addition, correcting the image and displaying the image on three or more displays of the remote control center may include: determining a preset threshold correction range (Scaling) based on the perspective distance distortion determination of the image; correcting an image based on the correction range (Scaling); It may include; calculating the difference between the horizontal length and vertical length of the image before and after correction based on the first display of the remote control center.

In addition, the step of displaying the difference in the horizontal length and the vertical length on the three or more displays of the remote control center may include at least one of a first display, a second display to the left of the first display, and a third display to the right of the first display. You can do;

The displaying on the display may include: adjusting a disposition angle between at least two of the first display, the second display, and the third display; may include.

In addition, in the step of guiding the driving situation of the vehicle, if the result of determining the perspective distance distortion of the image is greater than or equal to a preset threshold correction range, warning the remote control center, but guiding the driving situation of the vehicle through AR may include

In addition, the preset threshold value may be based on at least one of a specific time or a specific rotational angular velocity.

In addition, the step of warning when the preset threshold value is a specific time may include sound output to the remote control center.

In addition, the warning step when the preset threshold value is a specific rotational angular velocity may include a vibration output of the seat of the remote control center.

In addition, the step of determining the distance distortion of the image due to the image delay, Controlling the communication unit to transmit the sensing information to the AI processor; Controlling the communication unit to receive the AI-processed information from the AI processor; further comprising, the AI-processed information, the road and remote control viewed from the vehicle based on at least one of the driving speed and communication information a result of judging the perspective distance distortion of an image representing a difference in the road image displayed at the center; Perspective distance distortion determination result according to the difference between the movement of the center of gravity of the vehicle occupant in the rotation section and the rotation angular velocity of the controller of the remote control center; It can be characterized in that it can be determined.

In accordance with another embodiment of the present specification, there is provided an apparatus for assisting remote driving of an autonomous vehicle, comprising: a communication unit; Memory; and a processor functionally connected to the communication unit and the memory, wherein the communication unit receives at least one or more vehicle information of the vehicle traveling speed, sensing information, and communication information through the communication unit, wherein the processor includes the traveling speed or communication information Determine an image delay representing a difference between the road viewed from the vehicle and the displayed road image of the remote control center based on at least one of, determine the perspective distance distortion of the image due to the image delay, and correct the distortion of the image and displaying the corrected image on a display of a remote control center, guiding the driving situation of the vehicle obtained through the sensing information.

In order to achieve the above object, an autonomous driving vehicle according to an embodiment of the present invention includes a display unit, and vehicle driving information obtained when the vehicle enters within a set distance from a preset destination while driving or when a stop request is received and a control unit that searches for at least one stopable area based on , and displays information on the searched stopable area on the display unit or transmits the information to a preset terminal.

The autonomous vehicle according to an embodiment of the present invention may determine a recommended area within the searched stop possible area based on the vehicle driving information, and display information on the recommended area on the display unit, and a control unit for transmitting to the preset terminal.

The details of other embodiments are included in the detailed description and drawings.

A method for correcting an image in order to perform remote driving in a situation in which the vehicle needs to perform remote driving auxiliary according to an embodiment of the present invention and effects of an apparatus therefor will be described as follows.

The present specification can effectively collect vehicle information and vehicle surrounding information.

In addition, according to the present specification, communication delay may be reduced by directly transmitting/receiving information around a vehicle through an information collecting device rather than a vehicle based on the location of the vehicle.

In addition, the present specification may perform remote driving in a situation where remote driving is required, and provide an accurate and readable image to a remote control center performing remote driving.

In addition, the present specification may provide a corrected image so that the controller of the remote control center can accurately recognize the driving environment by correcting the delay of image information according to communication.

In addition, the present specification corrects the delay of image information by the speed of the vehicle when driving in a rotating section so that the controller of the remote control center recognizes this, and guides the vehicle occupants on the movement of the center of gravity of the vehicle to provide comfort to the passengers. can provide

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention pertains from the description below. .

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.
1 illustrates a block diagram of a wireless communication system to which the methods proposed in the present specification can be applied.
2 shows an example of a signal transmission/reception method in a wireless communication system.
3 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.
4 shows an example of a vehicle-to-vehicle basic operation using 5G communication.
5 is a diagram illustrating a vehicle according to an embodiment of the present specification.
6 is a block diagram of an AI device that can be applied to an embodiment of the present specification.
7 is a diagram for explaining a system in which an autonomous driving vehicle and an AI device are linked according to an embodiment of the present specification.
8 is a flowchart of a method for correcting image delay according to an embodiment of the present invention.
9 is a view for explaining a remote operation system according to an embodiment of the present invention.
10 is a diagram illustrating an example of a remote control center.
11 is a flowchart for briefly explaining the operation of the remote operation system according to an embodiment of the present invention.
12 is a view for explaining another example of determining an image delay error value according to an embodiment of the present invention.
13 to 14 are diagrams for explaining a method of measuring a distance between the vehicle 10 and an object according to an embodiment of the present invention.
13A to 13B are diagrams for explaining a method of correcting an image based on an image delay error value according to an embodiment of the present invention.
14 is a view for explaining a method of displaying a corrected image on a display of a remote control center according to an embodiment of the present specification.
15 is a view for explaining a method of displaying a corrected image in the case of magnification correction according to an embodiment of the present specification.
16 is a diagram illustrating an arrangement of a display of a remote control center according to an embodiment of the present specification.
17 is a view showing a screen for guiding the driving situation of the vehicle through AR according to an embodiment of the present specification.
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description for better understanding of the present invention, provide embodiments of the present invention, and together with the detailed description, explain the technical features of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present specification , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as "comprises" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, 5G communication required by a device requiring AI-processed information and/or an AI processor will be described through paragraphs A to G.

A. Example UE and 5G network block diagram

1 illustrates a block diagram of a wireless communication system to which the methods proposed in the present specification can be applied.

Referring to FIG. 1 , a device (autonomous driving device) including an autonomous driving module may be defined as a first communication device ( 910 in FIG. 1 ), and the processor 170 may perform a detailed autonomous driving operation.

A 5G network including another vehicle communicating with the autonomous driving device may be defined as a second communication device ( 920 in FIG. 1 ), and the processor 921 may perform a detailed autonomous driving operation.

The 5G network may be represented as the first communication device and the autonomous driving device may be represented as the second communication device.

For example, the first communication device or the second communication device may be a base station, a network node, a transmitting terminal, a receiving terminal, a wireless device, a wireless communication device, an autonomous driving device, or the like.

For example, a terminal or user equipment (UE) includes a vehicle, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, personal digital assistants (PDA), and a portable multimedia player (PMP). , navigation, slate PC, tablet PC, ultrabook, wearable device, for example, watch-type terminal (smartwatch), glass-type terminal (smart glass), HMD ( head mounted display) and the like. For example, the HMD may be a display device worn on the head. For example, an HMD may be used to implement VR, AR or MR. Referring to FIG. 1 , a first communication device 910 and a second communication device 920 include a processor 911,921, a memory 914,924, and one or more Tx/Rx RF modules (radio frequency module, 915,925). , including Tx processors 912 and 922 , Rx processors 913 and 923 , and antennas 916 and 926 . Tx/Rx modules are also called transceivers. Each Tx/Rx module 915 transmits a signal via a respective antenna 926 . The processor implements the functions, processes and/or methods salpinned above. The processor 921 may be associated with a memory 924 that stores program code and data. Memory may be referred to as a computer-readable medium. More specifically, in DL (communication from a first communication device to a second communication device), the transmit (TX) processor 912 implements various signal processing functions for the L1 layer (ie, the physical layer). The receive (RX) processor implements the various signal processing functions of L1 (ie, the physical layer).

The UL (second communication device to first communication device) is handled in the first communication device 910 in a manner similar to that described with respect to the receiver function in the second communication device 920 . Each Tx/Rx module 925 receives a signal via a respective antenna 926 . Each Tx/Rx module provides an RF carrier and information to the RX processor 923 . The processor 921 may be associated with a memory 924 that stores program code and data. Memory may be referred to as a computer-readable medium.

B. Signal transmission/reception method in wireless communication system

2 is a diagram illustrating an example of a signal transmission/reception method in a wireless communication system.

Referring to FIG. 2 , the UE performs an initial cell search operation such as synchronizing with the BS when the power is turned on or a new cell is entered ( S201 ). To this end, the UE receives a primary synchronization channel (P-SCH) and a secondary synchronization channel (S-SCH) from the BS, synchronizes with the BS, and acquires information such as cell ID can do. In the LTE system and the NR system, the P-SCH and the S-SCH are called a primary synchronization signal (PSS) and a secondary synchronization signal (SSS), respectively. After the initial cell discovery, the UE may receive a physical broadcast channel (PBCH) from the BS to obtain broadcast information in the cell. Meanwhile, the UE may check the downlink channel state by receiving a downlink reference signal (DL RS) in the initial cell search step. After the initial cell search, the UE receives a physical downlink control channel (PDCCH) and a physical downlink shared channel (PDSCH) according to information carried on the PDCCH to obtain more specific system information. It can be done (S202).

On the other hand, when there is no radio resource for the first access to the BS or signal transmission, the UE may perform a random access procedure (RACH) to the BS (steps S203 to S206). To this end, the UE transmits a specific sequence as a preamble through a physical random access channel (PRACH) (S203 and S205), and a random access response to the preamble through the PDCCH and the corresponding PDSCH (random access response, RAR) message may be received (S204 and S206). In the case of contention-based RACH, a contention resolution procedure may be additionally performed.

After performing the process as described above, the UE receives PDCCH/PDSCH (S207) and a physical uplink shared channel (PUSCH)/physical uplink control channel as a general uplink/downlink signal transmission process. Uplink control channel, PUCCH) transmission (S208) may be performed. In particular, the UE receives downlink control information (DCI) through the PDCCH. The UE monitors a set of PDCCH candidates in monitoring opportunities set in one or more control element sets (CORESETs) on a serving cell according to corresponding search space configurations. The set of PDCCH candidates to be monitored by the UE is defined in terms of search space sets, which may be a common search space set or a UE-specific search space set. CORESET consists of a set of (physical) resource blocks with a time duration of 1 to 3 OFDM symbols. The network may configure the UE to have multiple CORESETs. The UE monitors PDCCH candidates in one or more search space sets. Here, monitoring means trying to decode PDCCH candidate(s) in the search space. If the UE succeeds in decoding one of the PDCCH candidates in the search space, the UE determines that the PDCCH is detected in the corresponding PDCCH candidate, and performs PDSCH reception or PUSCH transmission based on the DCI in the detected PDCCH. The PDCCH may be used to schedule DL transmissions on PDSCH and UL transmissions on PUSCH. Here, the DCI on the PDCCH is a downlink assignment (i.e., downlink grant; DL grant) including at least modulation and coding format and resource allocation information related to the downlink shared channel, or uplink It includes an uplink grant (UL grant) including a modulation and coding format and resource allocation information related to a shared channel.

Referring to FIG. 2 , an initial access (IA) procedure in a 5G communication system will be additionally described.

The UE may perform cell search, system information acquisition, beam alignment for initial access, DL measurement, and the like based on the SSB. The SSB is mixed with an SS/PBCH (Synchronization Signal/Physical Broadcast channel) block.

SSB consists of PSS, SSS and PBCH. The SSB is configured in four consecutive OFDM symbols, and PSS, PBCH, SSS/PBCH or PBCH are transmitted for each OFDM symbol. PSS and SSS consist of 1 OFDM symbol and 127 subcarriers, respectively, and PBCH consists of 3 OFDM symbols and 576 subcarriers.

Cell discovery refers to a process in which the UE acquires time/frequency synchronization of a cell, and detects a cell ID (Identifier) (eg, Physical layer Cell ID, PCI) of the cell. PSS is used to detect a cell ID within a cell ID group, and SSS is used to detect a cell ID group. PBCH is used for SSB (time) index detection and half-frame detection.

There are 336 cell ID groups, and there are 3 cell IDs for each cell ID group. There are a total of 1008 cell IDs. Information on the cell ID group to which the cell ID of the cell belongs is provided/obtained through the SSS of the cell, and information on the cell ID among 336 cells in the cell ID is provided/obtained through the PSS.

The SSB is transmitted periodically according to the SSB period (periodicity). The SSB basic period assumed by the UE during initial cell discovery is defined as 20 ms. After cell access, the SSB period may be set to one of {5ms, 10ms, 20ms, 40ms, 80ms, 160ms} by the network (eg, BS).

Next, the acquisition of system information (SI) will be described.

The SI is divided into a master information block (MIB) and a plurality of system information blocks (SIB). SI other than MIB may be referred to as Remaining Minimum System Information (RMSI). The MIB includes information/parameters for monitoring the PDCCH scheduling the PDSCH carrying the System Information Block1 (SIB1) and is transmitted by the BS through the PBCH of the SSB. SIB1 includes information related to availability and scheduling (eg, transmission period, SI-window size) of the remaining SIBs (hereinafter, SIBx, where x is an integer of 2 or more). SIBx is included in the SI message and transmitted through the PDSCH. Each SI message is transmitted within a periodically occurring time window (ie, an SI-window).

Referring to FIG. 2 , a random access (RA) process in a 5G communication system will be additionally described.

The random access process is used for a variety of purposes. For example, the random access procedure may be used for network initial access, handover, and UE-triggered UL data transmission. The UE may acquire UL synchronization and UL transmission resources through a random access procedure. The random access process is divided into a contention-based random access process and a contention free random access process. The detailed procedure for the contention-based random access process is as follows.

The UE may transmit the random access preamble through the PRACH as Msg1 of the random access procedure in the UL. Random access preamble sequences having two different lengths are supported. The long sequence length 839 applies for subcarrier spacings of 1.25 and 5 kHz, and the short sequence length 139 applies for subcarrier spacings of 15, 30, 60 and 120 kHz.

When the BS receives the random access preamble from the UE, the BS sends a random access response (RAR) message (Msg2) to the UE. The PDCCH scheduling the PDSCH carrying the RAR is CRC-masked and transmitted with a random access (RA) radio network temporary identifier (RNTI) (RA-RNTI). The UE detecting the PDCCH masked by the RA-RNTI may receive the RAR from the PDSCH scheduled by the DCI carried by the PDCCH. The UE checks whether the random access response information for the preamble it has transmitted, that is, Msg1, is in the RAR. Whether or not random access information for Msg1 transmitted by itself exists may be determined by whether a random access preamble ID for the preamble transmitted by the UE exists. If there is no response to Msg1, the UE may retransmit the RACH preamble within a predetermined number of times while performing power ramping. The UE calculates the PRACH transmit power for the retransmission of the preamble based on the most recent path loss and power ramping counter.

The UE may transmit UL transmission on the uplink shared channel as Msg3 of the random access procedure based on the random access response information. Msg3 may include the RRC connection request and UE identifier. As a response to Msg3, the network may send Msg4, which may be treated as a contention resolution message on DL. By receiving Msg4, the UE can enter the RRC connected state.

C. Beam Management (BM) Procedure of 5G Communication System

The BM process may be divided into (1) a DL BM process using SSB or CSI-RS, and (2) a UL BM process using a sounding reference signal (SRS). In addition, each BM process may include Tx beam sweeping to determine a Tx beam and Rx beam sweeping to determine an Rx beam.

Let's look at the DL BM process using SSB.

A configuration for a beam report using the SSB is performed during channel state information (CSI)/beam configuration in RRC_CONNECTED.

- The UE receives from the BS a CSI-ResourceConfig IE including a CSI-SSB-ResourceSetList for SSB resources used for BM. The RRC parameter csi-SSB-ResourceSetList indicates a list of SSB resources used for beam management and reporting in one resource set. Here, the SSB resource set may be set to {SSBx1, SSBx2, SSBx3, SSBx4, ??}. The SSB index may be defined from 0 to 63.

- UE receives signals on SSB resources from the BS based on the CSI-SSB-ResourceSetList.

- When the CSI-RS reportConfig related to reporting on SSBRI and reference signal received power (RSRP) is configured, the UE reports the best SSBRI and RSRP corresponding thereto to the BS. For example, when the reportQuantity of the CSI-RS reportConfig IE is set to 'ssb-Index-RSRP', the UE reports the best SSBRI and the corresponding RSRP to the BS.

If the CSI-RS resource is configured in the same OFDM symbol(s) as the SSB, and 'QCL-TypeD' is applicable, the UE has the CSI-RS and the SSB similarly located in the 'QCL-TypeD' viewpoint ( quasi co-located, QCL). Here, QCL-TypeD may mean QCL between antenna ports in terms of spatial Rx parameters. When the UE receives signals of a plurality of DL antenna ports in a QCL-TypeD relationship, the same reception beam may be applied.

Next, a DL BM process using CSI-RS will be described.

The Rx beam determination (or refinement) process of the UE using the CSI-RS and the Tx beam sweeping process of the BS will be described in turn. In the UE Rx beam determination process, the repetition parameter is set to 'ON', and in the BS Tx beam sweeping process, the repetition parameter is set to 'OFF'.

First, a process of determining the Rx beam of the UE will be described.

- The UE receives the NZP CSI-RS resource set IE including the RRC parameter for 'repetition' from the BS through RRC signaling. Here, the RRC parameter 'repetition' is set to 'ON'.

- The UE repeats signals on the resource(s) in the CSI-RS resource set in which the RRC parameter 'repetition' is set to 'ON' in different OFDM symbols through the same Tx beam (or DL spatial domain transmission filter) of the BS receive

- The UE determines its own Rx beam.

- The UE omits CSI reporting. That is, the UE may omit the CSI report when the multi-RRC parameter 'repetition' is set to 'ON'.

Next, the Tx beam determination process of the BS will be described.

- The UE receives the NZP CSI-RS resource set IE including the RRC parameter for 'repetition' from the BS through RRC signaling. Here, the RRC parameter 'repetition' is set to 'OFF' and is related to the Tx beam sweeping process of the BS.

- The UE receives signals on resources in the CSI-RS resource set in which the RRC parameter 'repetition' is set to 'OFF' through different Tx beams (DL spatial domain transmission filter) of the BS.

- The UE selects (or determines) the best beam.

- The UE reports the ID (eg, CRI) and related quality information (eg, RSRP) for the selected beam to the BS. That is, when the CSI-RS is transmitted for the BM, the UE reports the CRI and the RSRP to the BS.

Next, a UL BM process using SRS will be described.

- The UE receives the RRC signaling (eg, SRS-Config IE) including the (RRC parameter) usage parameter set to 'beam management' from the BS. SRS-Config IE is used for SRS transmission configuration. The SRS-Config IE includes a list of SRS-Resources and a list of SRS-ResourceSets. Each SRS resource set means a set of SRS-resources.

- The UE determines Tx beamforming for the SRS resource to be transmitted based on the SRS-SpatialRelation Info included in the SRS-Config IE. Here, the SRS-SpatialRelation Info is set for each SRS resource and indicates whether to apply the same beamforming as that used in SSB, CSI-RS, or SRS for each SRS resource.

- If SRS-SpatialRelationInfo is configured in the SRS resource, the same beamforming as that used in SSB, CSI-RS, or SRS is applied and transmitted. However, if SRS-SpatialRelationInfo is not configured in the SRS resource, the UE arbitrarily determines Tx beamforming and transmits the SRS through the determined Tx beamforming.

Next, a beam failure recovery (BFR) process will be described.

In a beamformed system, Radio Link Failure (RLF) may frequently occur due to rotation, movement, or beamforming blockage of the UE. Therefore, BFR is supported in NR to prevent frequent RLF from occurring. BFR is similar to the radio link failure recovery process, and can be supported when the UE knows new candidate beam(s). For beam failure detection, the BS sets beam failure detection reference signals to the UE, and the UE determines that the number of beam failure indications from the physical layer of the UE is within a period set by the RRC signaling of the BS. When a threshold set by RRC signaling is reached (reach), a beam failure is declared (declare). after beam failure is detected, the UE triggers beam failure recovery by initiating a random access procedure on the PCell; Beam failure recovery is performed by selecting a suitable beam (if the BS provides dedicated random access resources for certain beams, these are prioritized by the UE). Upon completion of the random access procedure, it is considered that beam failure recovery has been completed.

D. URLLC (Ultra-Reliable and Low Latency Communication)

URLLC transmission defined in NR has (1) relatively low traffic size, (2) relatively low arrival rate, (3) extremely low latency requirements (eg, 0.5, 1ms), (4) a relatively short transmission duration (eg, 2 OFDM symbols), (5) may mean transmission for an urgent service/message, etc. In the case of UL, transmission for a specific type of traffic (eg, URLLC) is multiplexed with other previously scheduled transmissions (eg, eMBB) in order to satisfy a more stringent latency requirement. Needs to be. In this regard, as one method, information to be preempted for a specific resource is given to the previously scheduled UE, and the resource is used by the URLLC UE for UL transmission.

For NR, dynamic resource sharing between eMBB and URLLC is supported. eMBB and URLLC services may be scheduled on non-overlapping time/frequency resources, and URLLC transmission may occur on resources scheduled for ongoing eMBB traffic. The eMBB UE may not know whether the PDSCH transmission of the corresponding UE is partially punctured, and the UE may not be able to decode the PDSCH due to corrupted coded bits. In consideration of this, NR provides a preemption indication. The preemption indication may be referred to as an interrupted transmission indication.

With respect to the preemption indication, the UE receives the DownlinkPreemption IE through RRC signaling from the BS. When the UE is provided with the DownlinkPreemption IE, for monitoring the PDCCH carrying DCI format 2_1, the UE is configured with the INT-RNTI provided by the parameter int-RNTI in the DownlinkPreemption IE. The UE is additionally configured with a set of serving cells by INT-ConfigurationPerServing Cell including a set of serving cell indices provided by servingCellID and a corresponding set of positions for fields in DCI format 2_1 by positionInDCI, dci-PayloadSize It is established with the information payload size for DCI format 2_1 by , and is set with the indicated granularity of time-frequency resources by timeFrequencySect.

The UE receives DCI format 2_1 from the BS based on the DownlinkPreemption IE.

When the UE detects the DCI format 2_1 for the serving cell in the configured set of serving cells, the UE determines that the DCI format of the set of PRBs and the set of symbols of the monitoring period immediately preceding the monitoring period to which the DCI format 2_1 belongs. It can be assumed that there is no transmission to the UE in the PRBs and symbols indicated by 2_1. For example, the UE sees that the signal in the time-frequency resource indicated by the preemption is not the scheduled DL transmission for itself and decodes data based on the signals received in the remaining resource region.

E. mMTC (massive MTC)

mMTC (massive machine type communication) is one of the scenarios of 5G to support hyper-connectivity service that communicates simultaneously with a large number of UEs. In this environment, the UE communicates intermittently with a very low transmission rate and mobility. Therefore, mMTC is primarily aimed at how long the UE can run at a low cost. In relation to mMTC technology, 3GPP deals with MTC and NB (NarrowBand)-IoT.

The mMTC technology has features such as repeated transmission of PDCCH, PUCCH, physical downlink shared channel (PDSCH), PUSCH, and the like, frequency hopping, retuning, and guard period.

That is, a PUSCH (or PUCCH (particularly, long PUCCH) or PRACH) including specific information and a PDSCH (or PDCCH) including a response to specific information are repeatedly transmitted. Repeated transmission is performed through frequency hopping, and for repeated transmission, (RF) retuning is performed in a guard period from a first frequency resource to a second frequency resource, and specific information And a response to specific information may be transmitted/received through a narrowband (ex. 6 RB (resource block) or 1 RB).

F. Basic operation between autonomous vehicles using 5G communication

3 shows an example of basic operations of an autonomous vehicle and a 5G network in a 5G communication system.

The autonomous vehicle transmits specific information transmission to the 5G network (S1). The specific information may include autonomous driving-related information. Then, the 5G network may determine whether to remotely control the vehicle (S2). Here, the 5G network may include a server or module for performing remote control related to autonomous driving. In addition, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle (S3).

G. Application operation between autonomous vehicle and 5G network in 5G communication system

Hereinafter, the operation of the autonomous vehicle using 5G communication will be described in more detail with reference to FIGS. 1 and 2 and the above salpin wireless communication technology (BM procedure, URLLC, Mmtc, etc.).

First, the method proposed in this specification, which will be described later, and the basic procedure of the application operation to which the eMBB technology of 5G communication is applied will be described.

As in steps S1 and S3 of FIG. 3 , in order for the autonomous vehicle to transmit/receive signals and information to/from the 5G network, the autonomous vehicle performs an initial access procedure with the 5G network before step S1 of FIG. and a random access procedure.

More specifically, the autonomous vehicle performs an initial access procedure with the 5G network based on the SSB to obtain DL synchronization and system information. A beam management (BM) process and a beam failure recovery process may be added to the initial access procedure, and in the process of the autonomous vehicle receiving a signal from the 5G network, QCL (quasi-co location) ) relationship can be added.

In addition, the autonomous vehicle performs a random access procedure with the 5G network for UL synchronization acquisition and/or UL transmission. In addition, the 5G network may transmit a UL grant for scheduling transmission of specific information to the autonomous vehicle. Accordingly, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. In addition, the 5G network transmits a DL grant for scheduling transmission of a 5G processing result for the specific information to the autonomous vehicle. Accordingly, the 5G network may transmit information (or signals) related to remote control to the autonomous vehicle based on the DL grant.

Next, the method proposed in the present specification, which will be described later, and the basic procedure of the application operation to which the URLLC technology of 5G communication is applied will be described.

As described above, after the autonomous vehicle performs an initial access procedure and/or a random access procedure with the 5G network, the autonomous vehicle may receive a DownlinkPreemption IE from the 5G network. Then, the autonomous vehicle receives DCI format 2_1 including a pre-emption indication from the 5G network based on the DownlinkPreemption IE. In addition, the autonomous vehicle does not perform (or expect or assume) the reception of eMBB data in the resource (PRB and/or OFDM symbol) indicated by the pre-emption indication. Thereafter, the autonomous vehicle may receive a UL grant from the 5G network when it is necessary to transmit specific information.

Next, the method proposed in the present specification, which will be described later, and the basic procedure of the application operation to which the mMTC technology of 5G communication is applied will be described.

Among the steps of FIG. 3, the parts that are changed by the application of the mMTC technology will be mainly described.

In step S1 of FIG. 3 , the autonomous vehicle receives a UL grant from the 5G network to transmit specific information to the 5G network. Here, the UL grant includes information on the number of repetitions for the transmission of the specific information, and the specific information may be repeatedly transmitted based on the information on the number of repetitions. That is, the autonomous vehicle transmits specific information to the 5G network based on the UL grant. In addition, repeated transmission of specific information may be performed through frequency hopping, transmission of the first specific information may be transmitted in a first frequency resource, and transmission of the second specific information may be transmitted in a second frequency resource. The specific information may be transmitted through a narrowband of 6RB (Resource Block) or 1RB (Resource Block).

H. Autonomous vehicle-to-vehicle operation using 5G communication

4 illustrates an example of a vehicle-to-vehicle basic operation using 5G communication.

The first vehicle transmits specific information to the second vehicle (S61). The second vehicle transmits a response to the specific information to the first vehicle (S62).

On the other hand, depending on whether the 5G network is directly (sidelink communication transmission mode 3) or indirectly (sidelink communication transmission mode 4) involved in the resource allocation of the specific information and the response to the specific information, the vehicle-to-vehicle application operation Configuration may vary.

Next, a vehicle-to-vehicle application operation using 5G communication will be examined.

First, how the 5G network is directly involved in resource allocation of vehicle-to-vehicle signal transmission/reception will be described.

The 5G network may transmit DCI format 5A to the first vehicle for scheduling of mode 3 transmission (PSCCH and/or PSSCH transmission). Here, a physical sidelink control channel (PSCCH) is a 5G physical channel for scheduling specific information transmission, and a physical sidelink shared channel (PSSCH) is a 5G physical channel for transmitting specific information. Then, the first vehicle transmits SCI format 1 for scheduling of transmission of specific information to the second vehicle on the PSCCH. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

Next, how the 5G network is indirectly involved in resource allocation of signal transmission/reception will be examined.

The first vehicle senses a resource for mode 4 transmission in the first window. Then, the first vehicle selects a resource for mode 4 transmission in the second window based on the sensing result. Here, the first window means a sensing window, and the second window means a selection window. The first vehicle transmits SCI format 1 for scheduling of transmission of specific information to the second vehicle on the PSCCH based on the selected resource. Then, the first vehicle transmits specific information to the second vehicle on the PSSCH.

The above salpin 5G communication technology may be applied in combination with the methods proposed in the present specification to be described later, or may be supplemented to specify or clarify the technical characteristics of the methods proposed in the present specification.

5 is a diagram illustrating a vehicle according to an embodiment of the present specification.

Referring to FIG. 5 , a vehicle 10 according to the embodiment of the present specification is defined as a transportation means traveling on a road or track. The vehicle 10 is a concept including a car, a train, and a motorcycle. The vehicle 10 may be a concept including all of an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, and an electric vehicle having an electric motor as a power source. The vehicle 10 may be a vehicle owned by an individual. The vehicle 10 may be a shared vehicle. The vehicle 10 may be an autonomous vehicle.

In the diagram illustrating operations in the vehicle 10 and the server 20 of FIG. 6 , the AI device 20 may include an AI processor 21 , a memory 25 , and/or a communication unit 27 .

The AI device 20 is a computing device capable of learning a neural network, and may be implemented in various electronic devices such as a server, a desktop PC, a notebook PC, and a tablet PC.

The AI processor 21 may learn the neural network using a program stored in the memory 25 . In particular, the AI processor 21 may learn a neural network for recognizing vehicle-related data. Here, the neural network for recognizing vehicle-related data may be designed to simulate a human brain structure on a computer, and may include a plurality of network nodes having weights that simulate neurons of the human neural network. The plurality of network modes may transmit and receive data according to a connection relationship, respectively, so as to simulate a synaptic activity of a neuron in which a neuron sends and receives a signal through a synapse. Here, the neural network may include a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes may exchange data according to a convolutional connection relationship while being located in different layers. Examples of neural network models include deep neural networks (DNN), convolutional deep neural networks (CNN), Recurrent Boltzmann Machine (RNN), Restricted Boltzmann Machine (RBM), deep trust It includes various deep learning techniques such as neural networks (DBN, deep belief networks) and deep Q-networks, and can be applied to fields such as computer vision, speech recognition, natural language processing, and voice/signal processing.

Meanwhile, the processor performing the above-described function may be a general-purpose processor (eg, CPU), but may be an AI-only processor (eg, GPU) for artificial intelligence learning.

The memory 25 may store various programs and data necessary for the operation of the AI device 20 . The memory 25 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SDD). The memory 25 is accessed by the AI processor 21 , and reading/writing/modification/deletion/update of data by the AI processor 21 may be performed. In addition, the memory 25 may store a neural network model (eg, deep learning model 26) generated through a learning algorithm for data classification/recognition according to an embodiment of the present specification.

Meanwhile, the AI processor 21 may include a data learning unit 22 that learns a neural network for data classification/recognition. The data learning unit 22 may learn a criterion regarding which training data to use to determine data classification/recognition and how to classify and recognize data using the training data. The data learning unit 22 may learn the deep learning model by acquiring learning data to be used for learning and applying the acquired learning data to the deep learning model.

The data learning unit 22 may be manufactured in the form of at least one hardware chip and mounted on the AI device 20 . For example, the data learning unit 22 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of a general-purpose processor (CPU) or a graphics-only processor (GPU) to the AI device 20 . may be mounted. Also, the data learning unit 22 may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. In this case, the at least one software module may be provided by an operating system (OS) or may be provided by an application.

The data learning unit 22 may include a training data acquiring unit 23 and a model learning unit 24 .

The training data acquisition unit 23 may acquire training data required for a neural network model for classifying and recognizing data. For example, the training data acquisition unit 23 may acquire vehicle data and/or sample data to be input to the neural network model as training data.

The model learning unit 24 may use the acquired training data to learn so that the neural network model has a criterion for determining how to classify predetermined data. In this case, the model learning unit 24 may train the neural network model through supervised learning using at least a portion of the training data as a criterion for determination. Alternatively, the model learning unit 24 may learn the neural network model through unsupervised learning for discovering a judgment criterion by self-learning using learning data without guidance. Also, the model learning unit 24 may train the neural network model through reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct. Also, the model learning unit 24 may train the neural network model by using a learning algorithm including an error back-propagation method or a gradient decent method.

When the neural network model is trained, the model learning unit 24 may store the learned neural network model in a memory. The model learning unit 24 may store the learned neural network model in the memory of the server connected to the AI device 20 through a wired or wireless network.

The data learning unit 22 further includes a training data preprocessing unit (not shown) and a training data selection unit (not shown) to improve the analysis result of the recognition model or to save resources or time required for generating the recognition model You may.

The learning data preprocessor may preprocess the acquired data so that the acquired data can be used for learning for situation determination. For example, the training data preprocessor may process the acquired data into a preset format so that the model learning unit 24 may use the acquired training data for image recognition learning.

In addition, the learning data selection unit may select data required for learning from among the learning data acquired by the learning data acquiring unit 23 or the training data preprocessed by the preprocessing unit. The selected training data may be provided to the model learning unit 24 . For example, the learning data selector may select only data about an object included in the specific region as the learning data by detecting a specific region among images acquired through a vehicle camera.

In addition, the data learning unit 22 may further include a model evaluation unit (not shown) in order to improve the analysis result of the neural network model.

The model evaluator may input evaluation data to the neural network model and, when an analysis result output from the evaluation data does not satisfy a predetermined criterion, may cause the model learning unit 22 to learn again. In this case, the evaluation data may be predefined data for evaluating the recognition model. As an example, the model evaluator may evaluate as not satisfying a predetermined criterion when, among the analysis results of the learned recognition model for the evaluation data, the number or ratio of evaluation data for which the analysis result is not accurate exceeds a preset threshold. have.

The communication unit 27 may transmit the AI processing result by the AI processor 21 to an external electronic device.

Here, the external electronic device may be defined as an autonomous vehicle. Also, the AI device 20 may be defined as another vehicle or a 5G network communicating with the autonomous driving module vehicle. Meanwhile, the AI device 20 may be implemented by being functionally embedded in an autonomous driving module provided in a vehicle. In addition, the 5G network may include a server or module that performs autonomous driving-related control.

On the other hand, although the AI device 20 shown in FIG. 6 has been functionally divided into the AI processor 21, the memory 25, the communication unit 27, and the like, the above-described components are integrated into one module and the AI module Note that it may also be called

6 is a block diagram of an AI device according to an embodiment of the present specification.

The AI device 20 may include an electronic device including an AI model capable of performing AI processing, or a server including the AI model. Also, the AI device 20 may be included in at least a part of the vehicle 10 shown in FIG. 1 to perform at least a part of AI processing together.

The AI processing may include all operations related to driving of the vehicle 10 illustrated in FIG. 5 . For example, an autonomous vehicle may process sensing data or driver data by AI processing to perform processing/judgment and control signal generation operations. Also, for example, the autonomous driving vehicle may perform autonomous driving control by AI-processing data obtained through interaction with other electronic devices provided in the vehicle.

In the diagram illustrating operations in the vehicle 10 and the server 20 of FIG. 7 , the AI device 20 may include an AI processor 21 , a memory 25 , and/or a communication unit 27 .

The AI device 20 is a computing device capable of learning a neural network, and may be implemented in various electronic devices such as a server, a desktop PC, a notebook PC, and a tablet PC.

The AI processor 21 may learn the neural network using a program stored in the memory 25 . In particular, the AI processor 21 may learn a neural network for recognizing vehicle-related data. Here, the neural network for recognizing vehicle-related data may be designed to simulate a human brain structure on a computer, and may include a plurality of network nodes having weights that simulate neurons of the human neural network. The plurality of network modes may transmit and receive data according to a connection relationship, respectively, so as to simulate a synaptic activity of a neuron in which a neuron sends and receives a signal through a synapse. Here, the neural network may include a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes may exchange data according to a convolutional connection relationship while being located in different layers. Examples of neural network models include deep neural networks (DNN), convolutional deep neural networks (CNN), Recurrent Boltzmann Machine (RNN), Restricted Boltzmann Machine (RBM), deep trust It includes various deep learning techniques such as neural networks (DBN, deep belief networks) and deep Q-networks, and can be applied to fields such as computer vision, speech recognition, natural language processing, and voice/signal processing.

Meanwhile, the processor performing the above-described function may be a general-purpose processor (eg, CPU), but may be an AI-only processor (eg, GPU) for artificial intelligence learning.

The memory 25 may store various programs and data necessary for the operation of the AI device 20 . The memory 25 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SDD). The memory 25 is accessed by the AI processor 21 , and reading/writing/modification/deletion/update of data by the AI processor 21 may be performed. In addition, the memory 25 may store a neural network model (eg, deep learning model 26) generated through a learning algorithm for data classification/recognition according to an embodiment of the present specification.

Meanwhile, the AI processor 21 may include a data learning unit 22 that learns a neural network for data classification/recognition. The data learning unit 22 may learn a criterion regarding which training data to use to determine data classification/recognition and how to classify and recognize data using the training data. The data learning unit 22 may learn the deep learning model by acquiring learning data to be used for learning and applying the acquired learning data to the deep learning model.

The data learning unit 22 may be manufactured in the form of at least one hardware chip and mounted on the AI device 20 . For example, the data learning unit 22 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of a general-purpose processor (CPU) or a graphics-only processor (GPU) to the AI device 20 . may be mounted. Also, the data learning unit 22 may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. In this case, the at least one software module may be provided by an operating system (OS) or may be provided by an application.

The data learning unit 22 may include a training data acquiring unit 23 and a model learning unit 24 .

The training data acquisition unit 23 may acquire training data required for a neural network model for classifying and recognizing data. For example, the training data acquisition unit 23 may acquire vehicle data and/or sample data to be input to the neural network model as training data.

The model learning unit 24 may use the acquired training data to learn so that the neural network model has a criterion for determining how to classify predetermined data. In this case, the model learning unit 24 may train the neural network model through supervised learning using at least a portion of the training data as a criterion for determination. Alternatively, the model learning unit 24 may learn the neural network model through unsupervised learning for discovering a judgment criterion by self-learning using learning data without guidance. Also, the model learning unit 24 may train the neural network model through reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct. Also, the model learning unit 24 may train the neural network model by using a learning algorithm including an error back-propagation method or a gradient decent method.

When the neural network model is trained, the model learning unit 24 may store the learned neural network model in a memory. The model learning unit 24 may store the learned neural network model in the memory of the server connected to the AI device 20 through a wired or wireless network.

The data learning unit 22 further includes a training data preprocessing unit (not shown) and a training data selection unit (not shown) to improve the analysis result of the recognition model or to save resources or time required for generating the recognition model You may.

The learning data preprocessor may preprocess the acquired data so that the acquired data can be used for learning for situation determination. For example, the training data preprocessor may process the acquired data into a preset format so that the model learning unit 24 may use the acquired training data for image recognition learning.

In addition, the learning data selection unit may select data required for learning from among the learning data acquired by the learning data acquiring unit 23 or the training data preprocessed by the preprocessing unit. The selected training data may be provided to the model learning unit 24 . For example, the learning data selector may select only data about an object included in the specific region as the learning data by detecting a specific region among images acquired through a vehicle camera.

In addition, the data learning unit 22 may further include a model evaluation unit (not shown) in order to improve the analysis result of the neural network model.

The model evaluator may input evaluation data to the neural network model and, when an analysis result output from the evaluation data does not satisfy a predetermined criterion, may cause the model learning unit 22 to learn again. In this case, the evaluation data may be predefined data for evaluating the recognition model. As an example, the model evaluator may evaluate as not satisfying a predetermined criterion when, among the analysis results of the learned recognition model for the evaluation data, the number or ratio of evaluation data for which the analysis result is not accurate exceeds a preset threshold. have.

The communication unit 27 may transmit the AI processing result by the AI processor 21 to an external electronic device.

Here, the external electronic device may be defined as an autonomous vehicle. Also, the AI device 20 may be defined as another vehicle or a 5G network communicating with the autonomous driving module vehicle. Meanwhile, the AI device 20 may be implemented by being functionally embedded in an autonomous driving module provided in a vehicle. In addition, the 5G network may include a server or module that performs autonomous driving-related control.

On the other hand, although the AI device 20 shown in FIG. 6 has been functionally divided into the AI processor 21, the memory 25, the communication unit 27, and the like, the above-described components are integrated into one module and the AI module Note that it may also be called

7 is a diagram for explaining a system in which an autonomous driving vehicle and an AI device are linked according to an embodiment of the present specification.

Referring to FIG. 7 , the autonomous vehicle 10 may transmit data requiring AI processing to the AI device 20 through the communication unit, and the AI device 20 including the deep learning model 26 is the deep learning AI processing results using the model 26 may be transmitted to the autonomous vehicle 10 . The AI device 20 may refer to the contents described in FIG. 2 .

The autonomous driving vehicle 10 may include a memory 140 , a processor 170 , and a power supply unit 190 , and the processor 170 may further include an autonomous driving module 260 and an AI processor 261 . can Also, the autonomous driving vehicle 10 may include an interface unit that is connected to at least one electronic device provided in the vehicle by wire or wirelessly to exchange data required for autonomous driving control. At least one electronic device connected through the interface unit may include an object detection unit 210 , a communication unit 220 , a driving control unit 230 , a main ECU 240 , a vehicle driving unit 250 , a sensing unit 270 , and location data generation. part 280 may be included.

The interface unit may be composed of at least one of a communication module, a terminal, a pin, a cable, a port, a circuit, an element, and a device.

The memory 140 is electrically connected to the processor 170 . The memory 140 may store basic data for the unit, control data for operation control of the unit, and input/output data. The memory 140 may store data processed by the processor 170 . The memory 140 may be configured as at least one of ROM, RAM, EPROM, flash drive, and hard drive in terms of hardware. The memory 140 may store various data for the overall operation of the autonomous vehicle 10 , such as a program for processing or controlling the processor 170 . The memory 140 may be implemented integrally with the processor 170 . According to an embodiment, the memory 140 may be classified into a sub-configuration of the processor 170 .

The power supply unit 190 may supply power to the autonomous driving device 10 . The power supply unit 190 may receive power from a power source (eg, a battery) included in the autonomous vehicle 10 to supply power to each unit of the autonomous vehicle 10 . The power supply unit 190 may be operated according to a control signal provided from the main ECU 240 . The power supply unit 190 may include a switched-mode power supply (SMPS).

The processor 170 may be electrically connected to the memory 140 , the interface unit 280 , and the power supply unit 190 to exchange signals. The processor 170 is, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors (processors), controller It may be implemented using at least one of controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

The processor 170 may be driven by power provided from the power supply 190 . The processor 170 may receive data, process data, generate a signal, and provide a signal while power is supplied by the power supply unit 190 .

The processor 170 may receive information from another electronic device in the autonomous vehicle 10 through the interface unit. The processor 170 may provide a control signal to another electronic device in the autonomous vehicle 10 through the interface unit.

The autonomous vehicle 10 may include at least one printed circuit board (PCB). The memory 140 , the interface unit, the power supply unit 190 , and the processor 170 may be electrically connected to the printed circuit board.

Hereinafter, other electronic devices in the vehicle connected to the interface unit, the AI processor 261, and the autonomous driving module 260 will be described in more detail. Hereinafter, for convenience of description, the autonomous vehicle 10 will be referred to as a vehicle 10 .

First, the object detector 210 may generate information about an object outside the vehicle 10 . The AI processor 261 applies a neural network model to the data obtained through the object detector 210, so that at least one of the presence or absence of an object, location information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object. You can create one.

The object detection unit 210 may include at least one sensor capable of detecting an object outside the vehicle 10 . The sensor may include at least one of a camera, a radar, a lidar, an ultrasonic sensor, and an infrared sensor. The object detector 210 may provide data on an object generated based on a sensing signal generated by the sensor to at least one electronic device included in the vehicle.

Meanwhile, the vehicle 10 transmits the data acquired through the at least one sensor to the AI device 20 through the communication unit 220 , and the AI device 20 adds a neural network model 26 to the transmitted data. AI processing data generated by applying may be transmitted to the vehicle 10 . The vehicle 10 may recognize information on the detected object based on the received AI processing data, and the autonomous driving module 260 may perform an autonomous driving control operation using the recognized information.

The communication unit 220 may exchange signals with a device located outside the vehicle 10 . The communication unit 220 may exchange signals with at least one of an infrastructure (eg, a server, a broadcasting station), another vehicle, and a terminal. The communication unit 220 may include at least one of a transmit antenna, a receive antenna, a radio frequency (RF) circuit capable of implementing various communication protocols, and an RF element to perform communication.

By applying the neural network model to the data obtained through the object detector 210, at least one of the existence of an object, location information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object can be generated. .

The driving operation unit 230 is a device for receiving a user input for driving. In the manual mode, the vehicle 10 may be driven based on a signal provided by the driving control unit 230 . The driving manipulation unit 230 may include a steering input device (eg, a steering wheel), an acceleration input device (eg, an accelerator pedal), and a brake input device (eg, a brake pedal).

Meanwhile, in the autonomous driving mode, the AI processor 261 may generate an input signal of the driving manipulation unit 230 according to a signal for controlling the movement of the vehicle according to the driving plan generated through the autonomous driving module 260 . have.

Meanwhile, the vehicle 10 transmits data required for control of the driver manipulation unit 230 to the AI device 20 through the communication unit 220 , and the AI device 20 transmits the neural network model 26 to the transmitted data. AI processing data generated by applying may be transmitted to the vehicle 10 . The vehicle 10 may use the input signal of the driver manipulation unit 230 to control the movement of the vehicle based on the received AI processing data.

The main ECU 240 may control the overall operation of at least one electronic device included in the vehicle 10 .

The vehicle driving unit 250 is a device for electrically controlling various vehicle driving devices in the vehicle 10 . The vehicle driving unit 250 may include a power train drive control device, a chassis drive control device, a door/window drive control device, a safety device drive control device, a lamp drive control device, and an air conditioning drive control device. The power train drive control device may include a power source drive control device and a transmission drive control device. The chassis drive control device may include a steering drive control device, a brake drive control device, and a suspension drive control device. Meanwhile, the safety device drive control device may include a safety belt drive control device for seat belt control.

The vehicle driving unit 250 includes at least one electronic control device (eg, a control electronic control unit (ECU)).

The vehicle driving unit 250 may control a power train, a steering device, and a brake device based on a signal received from the autonomous driving module 260 . The signal received from the autonomous driving module 260 may be a driving control signal generated by applying a neural network model to vehicle-related data in the AI processor 261 . The driving control signal may be a signal received from the external AI device 20 through the communication unit 220 .

The sensing unit 270 may sense the state of the vehicle. The sensing unit 270 may include an inertial measurement unit (IMU) sensor, a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensor, a heading sensor, a position module, and a vehicle. It may include at least one of a forward/reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor, a temperature sensor, a humidity sensor, an ultrasonic sensor, an illuminance sensor, and a pedal position sensor. Meanwhile, an inertial measurement unit (IMU) sensor may include at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor.

The sensing unit 270 may generate state data of the vehicle based on a signal generated by at least one sensor. The vehicle state data may be information generated based on data sensed by various sensors provided inside the vehicle. The sensing unit 270 may include vehicle attitude data, vehicle motion data, vehicle yaw data, vehicle roll data, vehicle pitch data, vehicle collision data, vehicle direction data, vehicle angle data, and vehicle speed. data, vehicle acceleration data, vehicle inclination data, vehicle forward/reverse data, vehicle weight data, battery data, fuel data, tire pressure data, vehicle interior temperature data, vehicle interior humidity data, steering wheel rotation angle data, vehicle exterior illumination Data, pressure data applied to the accelerator pedal, pressure data applied to the brake pedal, and the like may be generated.

The AI processor 261 may generate vehicle state data by applying a neural network model to sensing data generated by at least one sensor. AI processing data generated by applying the neural network model includes vehicle attitude data, vehicle motion data, vehicle yaw data, vehicle roll data, vehicle pitch data, vehicle crash data, vehicle direction data, Vehicle angle data, vehicle speed data, vehicle acceleration data, vehicle inclination data, vehicle forward/reverse data, vehicle weight data, battery data, fuel data, tire pressure data, vehicle interior temperature data, vehicle interior humidity data, steering wheel rotation It may include angle data, external illumination data of the vehicle, pressure data applied to the accelerator pedal, pressure data applied to the brake pedal, and the like.

The autonomous driving module 260 may generate a driving control signal based on the AI-processed state data of the vehicle.

On the other hand, the vehicle 10 transmits the sensing data acquired through the at least one sensor to the AI device 20 through the communication unit 22, and the AI device 20 uses the transmitted sensing data as a neural network model 26 ), the generated AI processing data may be transmitted to the vehicle 10 .

The location data generator 280 may generate location data of the vehicle 10 . The location data generator 280 may include at least one of a Global Positioning System (GPS) and a Differential Global Positioning System (DGPS).

The AI processor 261 may generate more accurate vehicle location data by applying a neural network model to location data generated by at least one location data generating device.

According to an embodiment, the AI processor 261 performs a deep learning operation based on at least one of an Inertial Measurement Unit (IMU) of the sensing unit 270 and a camera image of the object detection device 210 , and the generated Position data may be corrected based on AI processing data.

On the other hand, the vehicle 10 transmits the location data obtained from the location data generating unit 280 to the AI device 20 through the communication unit 220, and the AI device 20 adds the received location data to the neural network model ( 26), the generated AI processing data may be transmitted to the vehicle 10 .

The vehicle 10 may include an internal communication system 50 . A plurality of electronic devices included in the vehicle 10 may exchange signals via the internal communication system 50 . Signals may contain data. The internal communication system 50 may use at least one communication protocol (eg, CAN, LIN, FlexRay, MOST, Ethernet).

The autonomous driving module 260 may generate a path for autonomous driving based on the obtained data, and may generate a driving plan for driving along the generated path.

The autonomous driving module 260 may implement at least one Advanced Driver Assistance System (ADAS) function. ADAS includes Adaptive Cruise Control (ACC), Autonomous Emergency Braking (AEB), Forward Collision Warning (FCW), Lane Keeping Assist (LKA), ), Lane Change Assist (LCA), Target Following Assist (TFA), Blind Spot Detection (BSD), Adaptive High Beam Control (HBA) , Auto Parking System (APS), Pedestrian Collision Warning System (PD Collision Warning System), Traffic Sign Recognition (TSR), Trafffic Sign Assist (TSA), Night Vision System At least one of a Night Vision (NV), a Driver Status Monitoring (DSM), and a Traffic Jam Assist (TJA) may be implemented.

The AI processor 261 may transmit at least one sensor provided in the vehicle and an executable control signal received from an external device to the autonomous driving module 260 .

In addition, the vehicle 10 transmits at least one data for performing ADAS functions to the AI device 20 through the communication unit 220 , and the AI device 20 adds a neural network model 260 to the received data. By applying, a control signal capable of performing an ADAS function may be transmitted to the vehicle 10 .

The autonomous driving module 260 acquires the driver's state information and/or the vehicle's state information through the AI processor 261 , and based on this, a switching operation from the autonomous driving mode to the manual driving mode or autonomous in the manual driving mode A switching operation to the driving mode may be performed.

Meanwhile, the vehicle 10 may use AI processing data for passenger support for driving control. For example, as described above, the state of the driver and the passenger may be checked through at least one sensor provided inside the vehicle.

Alternatively, the vehicle 10 may recognize a driver's or passenger's voice signal through the AI processor 261 , perform a voice processing operation, and perform a voice synthesis operation.

The above salpin 5G communication technology may be applied in combination with the methods proposed in the present specification to be described later, or may be supplemented to specify or clarify the technical characteristics of the methods proposed in the present specification.

Hereinafter, various embodiments of the present specification will be described in detail with reference to the accompanying drawings.

8 is a flowchart of a method for correcting image delay according to an embodiment of the present invention.

Referring to FIG. 8 , the processor 170 may acquire vehicle information including driving speed or sensing information from the vehicle and communication information from the server ( S800 ).

The object detection unit 210 may include at least one sensor capable of detecting an object outside the vehicle 10 . The sensor may include at least one of a camera, a radar, a lidar, an ultrasonic sensor, and an infrared sensor. The sensing information may be information acquired through at least one of the camera, radar, lidar, ultrasonic sensor, and infrared sensor.

Among the sensors, a camera may generate information about an object outside the vehicle 10 by using an image. The camera may include at least one lens, at least one image sensor, and at least one processor that is electrically connected to the image sensor to process a received signal, and generate data about the object based on the processed signal.

The camera may be mounted at a position where a field of view (FOV) can be secured in the vehicle in order to photograph the outside of the vehicle. The camera may be disposed adjacent to the front windshield in the interior of the vehicle to acquire an image of the front of the vehicle. The camera may be placed around the front bumper or radiator grill. The camera may be disposed adjacent to the rear glass in the interior of the vehicle to acquire an image of the rear of the vehicle. The camera may be placed around the rear bumper, trunk or tailgate. The camera may be disposed adjacent to at least one of the sides in the interior of the vehicle in order to acquire an image of the side of the vehicle. Alternatively, the camera may be disposed around a side mirror, a fender or a door.

According to an embodiment of the present specification, the vehicle 10 may acquire communication information between the vehicle and the server.

The communication information may include communication speed and transmission delay information.

The communication speed may be obtained by measuring the time it takes for data transmitted from the vehicle to the server to return to the vehicle.

Transmission delay means the time it takes to transmit (or receive from the network) data to the network line. The communication speed between the vehicle and the server can be measured through the transmission and reception of information between the vehicle and the server. The processor 170 may control the communication unit to transmit the sensing information of the autonomous vehicle to the AI processor included in the 5G network. In addition, the processor 170 may control the communication unit to receive AI-processed information from the processor.

Meanwhile, the vehicle 10 may perform an initial connection procedure with the 5G network in order to transmit the driver's state information to the 5G network. The vehicle 10 may perform an initial access procedure with the 5G network based on a synchronization signal block (SSB).

In addition, the vehicle 10 may receive from the network DCI (Downlink Control Information) used to schedule transmission of the sensing information obtained from at least one sensor provided inside the vehicle through a wireless communication unit.

The processor 170 may transmit the sensing information to the network based on the DCI.

The processor 170 may receive vehicle location information and communication speed information. A physical broadcast channel (PBCH) may be transmitted to the UE.

Data extracted from the sensing information is transmitted to the network through a PUSCH, and the SSB and the DM-RS of the PUSCH may be QCLed for QCL type D.

According to the embodiment of the present specification, the autonomous vehicle transmits the image by reflecting the delay of the image information provided according to the communication speed of the area through which the vehicle passes while providing image information to the controller of the remote control center using the output device. can be corrected.

Specifically, when the autonomous vehicle checks the communication state according to the driving region and provides sensing information including image information using output devices (eg, display, audio, etc.) of the remote control center, the communication speed As a result, the provision of images may be delayed or cut off.

Therefore, in order to provide continuous sensing information even in an area where the communication speed changes while the autonomous vehicle is driving, the size and type of information that the vehicle can process can be classified and information on the delay of sensing information can be provided according to the communication speed. . In addition, image information to be applied to the AI model may be acquired from a sensing device of the vehicle.

The processor 170 may determine the image delay based on at least one of the driving speed and the communication state (S810).

The image delay may be generated based on at least one of a traveling speed and a transmission delay.

Latency refers to the time it takes for an information or data packet to be transmitted from one point to another.

The network latency may be caused by at least one of a transmission delay, a propagation delay, a node processing delay, and a queuing delay.

Transmission Delay is the time it takes to transmit (or receive from) data on a network line, which may be caused by the transmission speed of network equipment. For example, if the network equipment is EtherNet, the transmission speed corresponds to 100Mbps. Transmission delay is defined as the ratio of bandwidth to data transmission rate.

Transmission delay = L/R (bandwidth = L(bit), data transmission rate = R(bit/sec))

A data packet called propagation delay may occur due to the influence of the time value used for transmission over the network and the transmission speed of the electronic signal of the network line itself. Propagation delay is defined as the ratio of transmission distance to signal transmission speed.

Propagation delay = d/s (transmission distance = d, transmission speed = s)

Nodal processing delay refers to the time required for data packet header processing, bit data error, and distribution path search, which are tasks of the router itself.

The queuing delay refers to the time it takes when a packet temporarily stays in a queue when a router cannot transmit a data packet to the network.

The image delay refers to the occurrence of distortion of an image different from the road image viewed by the passenger in the vehicle 10 and the road image displayed on the display of the remote control center obtained by the camera of the vehicle 10 .

The image delay may occur due to the delay of image transmission in the process of transmitting the image (data) acquired by the camera between the vehicle 10 and the network. That is, the image delay may occur due to a transmission delay in the process of transmitting the image between the vehicle 10 and the network.

In addition, the image delay may occur due to an error occurring in the process of measuring the relative distance between the camera of the vehicle 10 and the object when the vehicle travels at a high speed.

In other words, the camera may be a method of calculating the distance by measuring the time for the light source to be reflected from the object and return to the object (TOP, Time Of Flight). Since the camera acquires distance information and relative speed information with respect to an object based on the change in object size over time, in the case of a moving autonomous vehicle, it is affected by the driving speed, and in the case of high-speed driving, the distance information from the object Due to the difference, distortion of the distance with respect to the image may occur.

In addition, the image delay may cause distortion of the distance to the image due to the difference in the rotation angular velocity of the controller of the remote control center due to the movement of the center of gravity of the passenger in the actual vehicle in the rotation section.

Based on the image delay, the image may be corrected based on the distortion determination result of the distance to the image (S820).

The result of the distance distortion determination for the image is compared with the output value obtained by learning through an artificial neural network trained to discriminate distance distortion due to image delay obtained from the AI processor to determine a predetermined range of correction (scaling) do. In other words, the AI processor may acquire an output value by training to distinguish distortion caused by image delay. The output value is set to correspond to a predetermined correction range (Scaling).

The scaling range may be preset based on the difference between the perspective distance between the road viewed from the vehicle 10 and the road image displayed on the display of the remote control center due to the image delay.

The difference in the perspective distance refers to a difference between a far and near relative distance between an object on a road and a vehicle.

When the difference between the perspective distance between the vehicle 10 and the object is further increased due to the image delay, correction can be made to reduce the distance by enlarging the accumulation of the image, and when the difference between the perspective distance is closer, the accumulation of the image is reduced and corrected can do.

The size of the image corrected by enlarging or reducing the accumulation of the image may be different from the size of the output screen of the display on the remote control center. When the size of the corrected image is different from the size of the output screen of the display, the processor 170 may change the display method on the display (S830).

When the accumulation of the image is enlarged, the image to be displayed on the display may have a horizontal length and a vertical length exceeding the output screen size of the display based on a preset output screen size of the display. In this case, the excess horizontal length may be displayed on the left display or the right display of the display.

When the accumulation of the image is reduced, the image to be displayed on the display based on the preset output screen size of the display is smaller than the output screen size of the display, so that a horizontal length and a vertical length of a blank may occur in the display. In this case, the image to be displayed on the left display or the right display may be displayed on the horizontal and vertical lengths of the blank space.

The human driver or the machine driver of the remote control center may control the remote driving based on the display on the screen of the display ( S840 ).

The remote control center may control the vehicle by the human driver 1221 or the machine driver 1222 based on the image shown on the display of the remote control center.

For example, the processor 170 may store an image captured in a state in which image delay does not occur, and correct the image based on the image delay error value using the stored image. The processor 170 may store or temporarily store a frame for a certain amount of time from the moment the camera is operated for sensing. The image captured in a state in which the image delay does not occur may be an image captured within a predetermined time before the image delay occurs.

The processor 170 obtains information on an image delay error value through a stored image captured in a state in which no image delay occurs, and uses the obtained image delay error value to obtain information on the length of the screen requiring correction. . For example, the processor 170 may calculate a difference portion using stored image frames captured in a state in which no image delay occurs in order to reduce an area in which an image delay error occurs.

9 is a view for explaining a remote operation system according to an embodiment of the present invention.

10 is a diagram illustrating an example of a remote control center.

9 and 10 , the remote driving system may include a vehicle 1210 , one or more surrounding vehicles 1230 , a remote control center 1300 , and one or more server devices 1400 . The devices 1210 , 1230 , 1300 , and 1400 may exchange information with each other through wireless communication (eg, 5G communication). The devices 1210 , 1230 , 1300 , and 1400 may exchange information through the initial access and/or random access procedure described with reference to FIG. 2 .

The vehicle 1210 includes a sensor ECU (Electronic Control Unit), various sensors (eg, camera, gyro sensor, etc.), a driving ECU (Control ECU), a steering ECU (Steering ECU), and an accelerator/brake ECU ( Acc/brake ECU) and a communication control unit (TCU). ECUs in the vehicle can transmit information such as the vehicle's current location, speed, and direction to the remote control center through the TCU. In addition, these devices may be electrically and/or functionally connected within the vehicle. The sensor ECU may be a device that controls sensors such as a camera inside a vehicle. The driving ECU may be a device that controls vehicle driving, such as a steering wheel, acceleration and/or stopping. The TCU may be a device that transmits and receives information to and from the remote control center 1300 . For example, the vehicle 1210 may receive control information through a communication control device. The sensor electronic control device and/or the driving electronic control device in the vehicle 1210 may be controlled by the received control information to implement remote driving. And/or, the vehicle 1210 may transmit vehicle internal information (eg, diagnostic information) to the controller 1340 of the remote control center 1300 through the communication control device at the request of the remote control center 1300 . . Also, the vehicle may operate including the in-vehicle devices described above with reference to FIGS. 1 to 6 .

The controller 1340 of the remote control center 1300 may receive information from a vehicle, one or more server devices 1400 , one or more surrounding vehicles 1230 , and the like. Based on the location of the vehicle, the controller 1340 of the remote control center 1300 may collect information by connecting communication with the RSU, the surrounding vehicle, and/or the ITS around the vehicle. The control unit 1340 of the remote control center 1300 reconstructs the environment around the vehicle based on the collected information (eg, navigation, the highest/lowest speed of the road, the front traffic lights, obstacles, traffic congestion, emergency vehicles (ambulance, 119, Police car), etc.) can be operated.

In addition, the control unit 1340 of the remote control center 1300 classifies the cause of the remote driving request for each problem situation of the vehicle, and based on this, can differentially request necessary information, and classifies the remote driver into a human and a machine to provide necessary information. can be requested separately. In addition, the controller 1340 of the remote control center 1300 may variably apply the information request range for each vehicle speed and road complexity situation (eg, intersection) when requesting vehicle surrounding information for a remote driving vehicle.

As shown in FIG. 10 , the controller 1340 of the remote control center 1300 may control the vehicle by the human driver 1221 or the machine driver 1222 through the corresponding information. Through the received information, the human driver 1221 can hear sounds and warning sounds around the vehicle as if driving in real life, feel vibrations according to the vehicle situation, and view AR rendering of vehicle surroundings and road information through a camera, etc. can be seen as In this case, the machine driver 1222 may monitor and assist the human driver 1221 .

Alternatively, the machine operator 1222 may control the remote operation in case of a user's request or a specific situation. The machine operator 1222 may be located separately from the remote control device 1220 . For example, the machine driver 1222 may be one or more surrounding vehicles 1230 or an external server device 1400 , and performs remote driving by receiving information from the remote control device 1220 , or using a human driver 1221 . Assist and/or monitor. In case of an emergency, the controller 1340 of the remote control center 1300 may change the human driver 1221 to the machine driver 1222 . The controller 1340 of the remote control center 1300 may transmit control information for remote control to the vehicle based on the corresponding information.

The server device 1400 may transmit navigation information of the vehicle 1210 , information on surrounding accident vehicles, etc. to the controller 1340 of the remote control center 1300 according to a request.

One or more surrounding vehicles 1230 may transmit information collected through devices such as sensors included in the corresponding vehicle 1210 to the controller 1340 of the remote control center 1300 .

11 is a flowchart for briefly explaining the operation of the remote operation system according to an embodiment of the present invention.

Referring to FIG. 11 , the vehicle may transmit a remote driving request (request information) to the remote control center (or remote cockpit) (S1101). For example, the vehicle may request a remote operation from a remote control center by detecting a failure of a sensor or autonomous driving system in the vehicle in real time. Alternatively, the vehicle may transmit a remote driving request to the remote control center at the request of the driver. The remote driving request may include information indicating the type of failure of the vehicle, such as a sensor, an autonomous driving system, and/or information indicating the vehicle status.

Next, the remote control center may determine whether remote driving is performed according to the remote driving request received from the vehicle (S1102). In other words, the remote control center may determine whether remote driving is performed based on the remote driving request and request necessary information. The present invention may request and/or receive information for determining whether to drive remotely from a vehicle and/or an information collection device.

Next, the remote control center may request and receive vehicle information necessary to determine whether to perform remote driving to the vehicle (S1103, S1104).

Next, it is possible to request and receive the communication state information of the vehicle through the server (S1120, S1121).

Next, the remote control center may determine whether to delay the image based on the request information, communication information, and/or vehicle information (S1105).

Next, the remote control center may request and/or receive the result of determining the perspective distance distortion of the image from the AI device of the server based on the remote driving request, vehicle information, and the determined remote driving and/or image delay determination (S1106) , S1107).

Next, the remote control center may request and/or receive the corrected image from the server based on the remote driving request, vehicle information, determined remote driving and/or the distance distortion due to image delay (S1108, S1109) .

Next, the remote control center may display the corrected image on the display of the remote control center based on the remote driving request, vehicle information, determined remote driving and/or distance distortion due to image delay (S1110).

Next, the remote control center may receive vehicle information and/or surrounding vehicle information in real time and perform remote driving based thereon ( S1111 ). The remote control center configures a user interface based on vehicle information and/or vehicle surroundings information, and a human driver may control the vehicle through the user interface.

12 is a view for explaining another example of determining perspective distance distortion due to image delay in an embodiment of the present invention.

Referring to FIG. 12 , a feature value extracted from information obtained by the vehicle 10 may be transmitted to the 5G network ( S1200 ).

The information obtained by the vehicle 10 may be a traveling speed of the vehicle, or sensing information obtained from a sensor of the vehicle 10 .

Here, the 5G network may include an AI processor 21 or an AI processor, and the AI processor of the 5G network may perform AI processing based on the received vehicle information (S1210).

The AI processor may input the feature values received from the vehicle 10 into the DNN classifier (S1211). The AI processor may analyze the DNN output value (S1212). The AI processor may acquire an output value by training to distinguish distortion caused by image delay. The output value is set to correspond to a predetermined correction range (Scaling).

From the DNN output value, it is possible to determine the perspective distance distortion of the image due to the image delay (S1213). That is, it means determining a predetermined range of correction (scaling) by comparing it with an output value obtained by learning through an artificial neural network trained to discriminate distance distortion due to image delay obtained from an AI processor.

The scaling range may be preset based on the difference between the perspective distance between the road viewed from the vehicle 10 and the road image displayed on the display of the remote control center due to the image delay.

The 5G network may transmit the result of the distance distortion of the image due to the image delay determined by the AI processor to the vehicle 10 through the wireless communication unit (S1215).

When it is determined that perspective distance distortion has occurred in the image, the image may be corrected based on the image correction range ( S1214 and S1216 ).

When the AI processor determines the perspective distance distortion of the image due to the image delay, the AI processor may determine correction (S1214).

The perspective distance distortion of the image may be generated based on a difference between the perspective distance between the road viewed from the vehicle 10 and the road image displayed on the display of the remote control center due to the image delay.

The difference in the perspective distance refers to a difference between a far and near relative distance between an object on a road and a vehicle. When the difference between the perspective distance between the vehicle 10 and the object is further increased due to the image delay, correction can be made to reduce the distance by enlarging the accumulation of the image, and when the difference between the perspective distance is closer, the accumulation of the image is reduced and corrected can do.

The size of the image corrected by enlarging or reducing the accumulation of the image may be different from the size of the output screen of the display on the remote control center. When the size of the corrected image is different from the size of the output screen of the display, the processor 170 may change the display method on the display.

When the accumulation of the image is enlarged, the image may have a horizontal length and a vertical length exceeding the output screen size of the display based on a preset output screen size of the display. In this case, the excess horizontal length may be displayed on the left display or the right display of the display.

When the accumulation of the image is reduced, the image may be smaller than the output screen size of the display based on a preset output screen size of the display, so that a horizontal length and a vertical length of a blank may occur in the display. In this case, the image to be displayed on the left display or the right display may be displayed on the horizontal and vertical lengths of the blank space.

If no correction is required for the perspective distance distortion of the image, the image may be transmitted to the remote control center (S1217).

13A to 13B are diagrams illustrating a method of correcting an image based on the determination of the distortion of the perspective distance due to the image delay according to an embodiment of the present invention.

Referring to FIG. 13A , for example, ( 1 ) of FIG. 13A may represent an image in which a traveling speed of the vehicle 10 is 50 km and a transmission delay of 30 msec is generated. (2) of FIG. 13A may represent an AI processor that obtains and corrects an output value by inputting a feature value extracted from sensing information to an artificial neural network (DNN) classifier trained to distinguish image distortion.

(3) of FIG. 13A may represent an image corrected based on an output value AS obtained from an artificial neural network (DNN) classifier.

According to an embodiment of the present specification, the output value AS may have any one value between 0 and 5.

(3) of FIG. 13A and (1) of FIG. 13A and the two screens may have the same scale. That is, the image may be corrected based on a preset threshold correction range (Scaling) based on a result of determining the perspective distance distortion of the image due to the image delay based on the output value AS of the artificial neural network. In other words, when the result of determining the perspective distance distortion of the image is 0, it may mean distortion to a degree that does not need to be corrected. In this case, the image information transmitted from the vehicle may be displayed on the display of the remote control center.

Referring to FIG. 13B , for example, ( 1 ) of FIG. 13B may represent an image in which the vehicle 10 has a traveling speed of 70 km and a transmission delay of 50 msec occurs. In (2) of FIG. 13B, the feature value extracted from the sensing information is input to an artificial neural network classifier trained to distinguish the distortion of the perspective distance of the image, and the output value AS is obtained and the image based on the judgment result on the distortion of the perspective distance. may represent an AI processor that calibrates

Comparing the two screens of (3) of FIG. 13B and (1) of FIG. 13B, the enlarged screen of (1) of FIG. 13B may be (3) of FIG. 13B. That is, when the distortion of the image due to the image delay is determined based on the output value AS of the artificial neural network, the accumulation of the image may be corrected based on the scaling range.

That is, when the image distortion determination result value due to the image delay is obtained based on the output value AS of the artificial neural network, the image is corrected based on the preset threshold correction range (Scaling) of (3) of FIG. 13B. can When the correction range is not 0, the image may be corrected based on a preset threshold correction range. In other words, FIG. 13(b) is a view in which the image is corrected based on a preset correction range in this case where the result of determining the perspective distance distortion of the image corresponds to 3 and the result of determining the perspective distance distortion of the image is 3 can be

According to the embodiment of the present specification, even in the case of transmission delay depending on the driving speed and/or communication state, the same screen as that seen by the passenger of the actual vehicle 10 to the human driver of the remote control center can be displayed on the display of the remote control center. have.

14 is a view for explaining a method of displaying a corrected image on three displays of a remote control center according to an embodiment of the present specification.

Referring to FIG. 14 , a portion indicated by an oblique line in FIG. 14 (a) is a view of the three displays from a top view. Displayed on the back side of the display means an image to be played back on the display.

14 (b) shows that when the image to be displayed on the middle display screen among the three displays is enlarged due to the image delay, the enlarged horizontal and/or vertical length of the middle screen is displayed on the left and/or right display screen. can

The rectangle indicated by the double slanted line in contact with the left vertex (LV) and the right vertex (RV) of the center display of FIG. can indicate The right side of the rectangle indicated by the double diagonal line may be displayed on the screen of the right display D3 of the center display D1. Similarly, the left of the rectangles indicated by the double diagonal line may be displayed on the screen of the center display D1 and the left display D2.

15 is a view for explaining a method of displaying a corrected image in the case of magnification correction according to an embodiment of the present specification.

Referring to FIG. 15 , it may indicate that the enlarged horizontal length is enlarged and displayed on the left display part (a) and the right display part (b) based on the center display.

16 is a diagram illustrating an arrangement of a display of a remote control center according to an embodiment of the present specification.

Referring to FIG. 16 , three consecutive displays are symmetrically arranged at a predetermined angle.

In the method of displaying a corrected image when the display of the remote control center is configured with three displays according to the embodiment of the present specification, the arrangement angle of the display is adjusted so that the controller can understand the exact road environment.

16 (a) has an angle (x) between the first display and the second display and an angle (x) between the second display and the third display, and may be a view before the arrangement angle of the display is adjusted. have.

16 (b) is a display arrangement angle in which the angle y between the first display and the second display and the angle y between the second display and the third display are relatively large compared to that of FIG. 16 (a). indicates

According to an embodiment of the present specification, image perspective distance distortion may occur based on image delay. When the horizontal and vertical lengths of the first display exceed the horizontal length and vertical length of the first display by enlarging the screen size based on the image perspective distance distortion determination result, the area that can be expressed on the first display before correction is set to the second display and the third display. It can be moved and displayed. In this case, as shown in (b) of FIG. 16 , by widening the angle y between the respective displays, the controller can intuitively grasp the road environment.

17 is a view showing a screen for guiding the driving situation of the vehicle through AR according to an embodiment of the present specification. AR technology provides a CG image made virtually on top of an image of a real object, and MR technology is applied to the real world. It is a computer graphics technology that provides by mixing and combining virtual objects.

Referring to FIG. 17 , when the corrected screen is displayed on the display of the remote control center, the AI processing result value obtained based on the image delay may be displayed together with the corrected screen. That is, in (a) of FIG. 17 , a preset threshold correction range (Scaling) based on a result of the determination of perspective distance distortion of an image may be displayed and provided to the controller as an AR guide.

When the calibrated screen is displayed on the display of the remote control center, it is possible to display the actual ride comfort of the occupants of the vehicle, such as the difference in rotational angular velocity between the occupant and the controller due to the movement of the center of gravity of the vehicle, as AR guide and/or vibration output. Also, in FIG. 17(b) , movement of the center of gravity due to inertia felt by the passenger may be provided to the controller as an AR guide.

Hereinafter, embodiments applied in the present specification are as follows.

The remote driving assistance method of a free-driving vehicle according to the first embodiment of the present specification includes receiving communication information of a region in which the vehicle travels from a server, acquiring at least one of driving speed and sensing information from the vehicle, and Determine an image delay representing a difference between a road viewed from the vehicle and a road image displayed by a remote control center based on at least one of driving speed or communication information, and determine perspective distance distortion of an image due to the image delay Compensating for image distortion and displaying the corrected image on a display of a remote control center, including guiding the driving situation of the vehicle obtained through the sensing information.

Second Embodiment: The method of Embodiment 1, wherein the acquiring of the sensing information includes acquiring image information about at least one of a driving direction, left or right side using a vehicle camera, and using a gyro sensor of the vehicle. can be used to obtain equilibrium information on the movement of the vehicle's center of gravity.

Third embodiment: The first embodiment, wherein the image delay may occur based on at least one of the communication information and the traveling speed.

Fourth Embodiment: In the first embodiment, the determining of the image distortion due to the image delay includes extracting feature values from sensing information and using the feature values to distinguish image distortion due to the image delay. It is possible to input to a trained artificial neural network (DNN) classifier, analyze an output value of the artificial neural network, and determine image distortion due to the image delay based on the output value of the artificial neural network.

Fifth embodiment: The method of Embodiment 1, wherein the step of correcting the image and displaying it on the display of the remote control center determines a preset threshold correction range (Scaling) based on the perspective distance distortion determination of the image, The image may be corrected based on the calibration range (Scaling), and the difference between the horizontal length and the vertical length of the image before and after the correction may be calculated based on the first display of the remote control center.

Example 6: The method of Example 1,

The displaying on the display of the remote control center includes displaying the difference between the horizontal length and the vertical length on at least one of the first display, the second display on the left side of the first display, and the third display on the right side of the first display may include doing

Seventh embodiment: The method of Embodiment 1, wherein the displaying on the display may include adjusting a disposition angle between at least two of the first display, the second display, and the third display.

Eighth embodiment: The method of Embodiment 1, wherein the step of guiding the driving condition of the vehicle comprises:

When the result of determining the distance distortion of the image is greater than or equal to a preset threshold correction range, warning the remote control center may include, but may include guiding the driving situation of the vehicle through AR.

Ninth embodiment: The first embodiment, wherein the preset threshold value may be based on at least one of a specific time or a specific rotational angular velocity.

Tenth embodiment: The first embodiment, wherein the warning when the preset threshold value is a specific time may include sound output to the remote control center.

Eleventh embodiment: The first embodiment, wherein the warning when the preset threshold value is a specific rotational angular velocity may include a vibration output of the seat of the remote control center.

Twelfth embodiment: The method of Embodiment 1, wherein the determining of the result value of the error state due to the image delay includes controlling a communication unit to transmit the sensing information to an AI processor, and receiving AI-processed information from the AI processor control the communication unit to do so, and the AI-processed information determines perspective distance distortion of an image representing a difference between a road viewed from the vehicle and a road image displayed by the remote control center based on at least one of the driving speed or communication information result; Perspective distance distortion determination result according to the difference between the movement of the center of gravity of the vehicle occupant in the rotation section and the rotation angular velocity of the controller of the remote control center; A remote driving assistance method of an autonomous vehicle, characterized in that

A remote driving assistance apparatus for an autonomous vehicle according to a thirteenth embodiment of the present specification includes a communication unit, a memory, and a processor functionally connected to the communication unit and the memory, and the driving speed of the vehicle, sensing information or communication through the communication unit Receive at least one or more vehicle information of the state information, and the processor determines an image delay indicating a difference between a road viewed from the vehicle and a road image displayed by the remote control center based on at least one of the driving speed and communication information and determining the perspective distance distortion of the image due to the image delay, correcting the distortion of the image, and displaying the corrected image on the display of the remote control center, The driving condition of the vehicle obtained through the sensing information guide

14th Embodiment: The 14th embodiment: The sensing information according to Embodiment 13, wherein the sensing information includes image information on at least one of a driving direction, left or right side using a camera of the vehicle or a weight of the vehicle using a gyro sensor of the vehicle It may be equilibrium information for center movement.

Fifteenth embodiment: The thirteenth embodiment, wherein the image delay may occur based on at least one of the communication information and the traveling speed.

Sixteenth embodiment: The method of embodiment 13, wherein the processor extracts feature values from the acquired sensing information, and applies the feature values to an artificial neural network (DNN) classifier trained to discriminate image distortion caused by the image delay. input, analyze the output value of the artificial neural network, and determine the distortion of the image due to the image delay based on the output value of the artificial neural network.

Seventeenth embodiment: The 13th embodiment, wherein the processor determines a preset threshold correction range (Scaling) based on the perspective distance distortion determination of the image, corrects the image based on the correction range (Scaling), , it is possible to calculate the difference between the horizontal length and the vertical length of the image before and after correction based on the first display of the remote control center.

18th Embodiment: The 17th embodiment, wherein the processor is configured to display the difference between the horizontal length and the vertical length to at least one of a first display, a second display to the left of the first display, and a third display to the right of the first display. can be displayed

19th embodiment: The 18th embodiment, wherein the processor may adjust an angle disposed between at least two of the first display, the second display, and the third display.

Embodiment 20: The method of embodiment 13, wherein the processor comprises:

If the result of determining the perspective distance distortion of the image is greater than or equal to a preset threshold correction range, the remote control center may be warned, but the driving situation of the vehicle may be guided through AR.

Twenty-first embodiment: The twentieth embodiment, wherein the preset threshold value may be based on at least one of a specific time or a specific rotational angular velocity.

The twenty-second embodiment: the twenty-first embodiment, when the preset threshold value is a specific time, a sound output may be included in the remote control center.

23rd embodiment: The 21st embodiment, when the preset threshold value is a specific rotational angular velocity, the vibration output of the seat of the remote control center may be included.

24th embodiment: The method of embodiment 13, wherein the processor controls the communication unit to transmit the sensing information to an AI processor included in the network, and controls the communication unit to receive AI-processed information from the AI processor, , The AI-processed information may include: a perspective distance distortion determination result of an image representing a difference between a road viewed from the vehicle and a road image displayed by the remote control center based on at least one of the driving speed or communication information; Perspective distance distortion determination result according to the difference between the movement of the center of gravity of the vehicle occupant in the rotation section and the rotation angular velocity of the controller of the remote control center; It can be characterized as being

The above-described specification can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of this specification should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalents of this specification are included in the scope of this specification.

1210: vehicle 1230: surrounding vehicle
1300: remote control center 1400: server

Claims (20)

In the remote driving assistance method of an autonomous driving vehicle for remote driving in an autonomous driving system,
Receiving communication information of a region in which the vehicle travels from a server;
acquiring at least one of a driving speed and sensing information from the vehicle;
determining an image delay indicating a difference between a road viewed from the vehicle and a road image displayed by a remote control center based on at least one of the driving speed and communication information;
determining the perspective distance distortion of the image due to the image delay;
correcting the distortion of the image and displaying the corrected image on a display of a remote control center; and
guiding the driving condition of the vehicle acquired through the sensing information;
A remote driving assistance method of an autonomous vehicle, comprising: The method of claim 1,
The step of obtaining the sensing information includes:
obtaining image information on at least one of a driving direction, a left side, or a right side using a camera of the vehicle;
obtaining balance information on the movement of the center of gravity of the vehicle by using a gyro sensor of the vehicle;
A remote driving assistance method of an autonomous vehicle, comprising: The method of claim 1,
The video delay is
The remote driving assistance method of an autonomous vehicle, characterized in that it is generated based on at least one of the communication information and the driving speed. The method of claim 1,
The step of determining the perspective distance distortion of the image due to the image delay,
extracting feature values from the sensing information; and
inputting the feature values to an artificial neural network (DNN) classifier trained to discriminate image distortion caused by the image delay;
analyzing an output value of the artificial neural network;
determining the perspective distance distortion of the image due to the image delay based on the output value of the artificial neural network;
A remote driving assistance method of an autonomous vehicle, comprising: The method of claim 1,
Correcting the image and displaying it on the display of the remote control center,
determining a preset threshold correction range (Scaling) based on the perspective distance distortion determination of the image;
correcting an image based on the correction range (Scaling);
calculating a difference between a horizontal length and a vertical length of an image before and after correction based on the first display of the remote control center;
A remote driving assistance method of an autonomous vehicle, comprising: 6. The method of claim 5,
The step of displaying on the display of the remote control center comprises:
displaying the difference between the horizontal length and the vertical length on at least one of the first display, a second display to the left of the first display, and a third display to the right of the first display;
A remote driving assistance method of an autonomous vehicle, comprising: 7. The method of claim 6,
The step of displaying on the display is
adjusting a disposition angle between at least two of the first display, the second display, or the third display;
A remote driving assistance method of an autonomous vehicle, comprising: The method of claim 1,
The step of guiding the driving situation of the vehicle,
If the result of determining the perspective distance distortion of the image is greater than a preset threshold correction range, a warning signal is transmitted to the remote control center,
A remote driving assistance method for an autonomous vehicle, comprising guiding the driving situation of the vehicle through AR. 9. The method of claim 8,
The preset threshold value is based on at least one of a specific time or a specific rotational angular velocity. 10. The method of claim 9,
The step of warning when the preset threshold value is a specific time,
The remote driving assistance method of an autonomous vehicle, characterized in that it includes a sound output to the remote control center. 10. The method of claim 9,
Transmitting a warning signal when the preset threshold value is a specific rotational angular velocity comprises:
The remote driving assistance method of the autonomous vehicle, characterized in that it comprises the vibration output of the seat of the remote control center. According to claim 1,
The step of determining the perspective distance distortion of the image due to the image delay,
controlling the communication unit to transmit the sensing information to the AI processor; and
Controlling the communication unit to receive the AI-processed information from the AI processor; further comprising,
The AI-processed information is
a perspective distance distortion determination result of an image representing a difference between a road viewed from the vehicle and a road image displayed by a remote control center based on at least one of the driving speed or communication information;
In the rotation section, the center of gravity movement of the vehicle occupant and the difference in the rotation angular velocity of the controller of the remote control center include the result of the judgment of perspective distance distortion.
A remote driving assistance method of an autonomous vehicle, characterized in that. In the remote driving assistance device of an autonomous vehicle,
communication department;
Memory; and
a processor operatively connected to the communication unit and the memory; including,
Receive at least one or more vehicle information of the driving speed of the vehicle, sensing information, and communication information through the communication unit,
The processor is
Determining an image delay representing a difference between a road viewed from the vehicle and a road image displayed by a remote control center based on at least one of the driving speed or communication information, and determining a perspective distance distortion of an image due to the image delay, Comprising correcting the distortion of the image and displaying the corrected image on a display of a remote control center,
A remote driving assistance device, characterized in that it guides the driving situation of the vehicle acquired through the sensing information. 14. The method of claim 13,
The sensing information is
The remote driving assistance device, characterized in that it is image information on at least any one of the driving direction, left or right side using the camera of the vehicle, or balance information on the movement of the center of gravity of the vehicle using the gyro sensor of the vehicle. 14. The method of claim 13,
The video delay is
The remote driving assistance device, characterized in that it is generated based on at least one of the communication information and the driving speed. 14. The method of claim 13,
The processor is
The step of determining the perspective distance distortion of the image due to the image delay,
Extracting feature values from the acquired sensing information, inputting the feature values to an artificial neural network (DNN) classifier trained to distinguish image distortion due to the image delay, analyzing the output value of the artificial neural network, and the artificial neural network Remote driving assistance device, characterized in that for determining the distance distortion of the image due to the image delay based on the output value of. 14. The method of claim 13,
The processor is
Determine a preset threshold correction range (Scaling) based on the perspective distance distortion determination of the image, correct the image based on the correction range (Scaling), before and after correction based on the first display of the remote control center A remote driving assistance device, characterized in that it calculates the difference between the horizontal and vertical lengths of the image. 18. The method of claim 17,
The processor is
The remote driving assistance device, characterized in that the difference between the horizontal and vertical length is displayed on at least one of the first display, a second display on the left side of the first display, and a third display on the right side of the first display. 19. The method of claim 18,
The processor is
The remote driving assistance device, characterized in that adjusting an angle disposed between at least two of the first display, the second display and the third display. 14. The method of claim 13,
The processor is
If the result of determining the distance distortion of the image is greater than or equal to a preset threshold correction range, a warning signal is transmitted to the remote control center,
Remote driving assistance method, characterized in that guiding the driving situation of the vehicle through AR.

Images