KR20220071513A - A method and a device for determining latency that occurs when synchronizing sensors in autonomous vehicles and converting protocols - Google Patents

Abstract

The present invention relates to a method and apparatus for determining latency that occurs during sensor synchronization of autonomous vehicle and protocol conversion. According to the present invention, the present invention has an effect of increasing precision and reliability of vehicle control by determining latency occurring at a data sensing and transmission time point and a transmission completion time point according to a data transmission feature (a transmission delay, transmission method and the like) between sensors to control time synchronization when an autonomous vehicle collects data from heterogeneous sensors (slaves).

Description

{A method and a device for determining latency that occurs when synchronizing sensors in autonomous vehicles and converting protocols}

The present invention relates to a method and apparatus for determining the latency that occurs during sensor synchronization and protocol conversion of an autonomous vehicle, and more specifically, a first latency according to a transmission delay of data between a slave and a master for sensor time synchronization; , to a method and apparatus capable of calculating and determining a second latency according to a communication protocol conversion between the slave and the master.

Domain Control Unit (DCU) implements integrated control of various ECUs, sensors and actuators to satisfy large data processing and high vehicle control performance required by autonomous driving vehicles, that is, ADAS (Advanced Driver Assistance Systems) applied vehicles. It is a next-generation integrated control system.

Data required for recognition, judgment, and control of autonomous driving is required in DCU, and various types of sensors such as radar, camera, lidar V2X, digital map, and ADR are used to acquire large-capacity data necessary for recognition, judgment, and control. and it is an essential data element for operating autonomous vehicles.

When the heterogeneous sensors transmit data, one gateway integrates the data and transmits it to the vehicle control device, and the data analysis device recognizes and analyzes each data of the received heterogeneous sensors and transmits the analysis result to the vehicle control device to control the vehicle According to the determination of the device, a command for controlling the vehicle may be transmitted to each electric component (steering device, throttle valve, brake, etc.) of the vehicle. At this time, referring to FIG. 4 showing a DCU block diagram of an autonomous vehicle, the above-described data analysis device may be an application processor (AP) using a Linux platform and Adaptive AUTOSAR, and the vehicle control device is a Classic AUTOSAR may be a microcontroller unit (MCU) to which is applied. The AP may play a role of recognizing, analyzing, and judging data sensed by the sensor, and the MCU may play a role in directly controlling the vehicle based on the information the AP determines and transmits. For example, the AP may recognize an obstacle in front of it and determine that it should be avoided based on data from a number of different sensors, and the MCU may control the vehicle to run avoiding the obstacle based on the AP's determination.

On the other hand, if there is a discrepancy in the sensing timing between these heterogeneous sensors and the transmission time when the heterogeneous sensors transmit sensing data to the gateway, the data analysis device cannot perform accurate analysis, and as a result, the vehicle control device stops the vehicle. Since an error may occur in the control, it is necessary to develop a method for calculating and determining the latency caused by the above-described cause before data transmission from the sensors.

In addition, the communication protocols based on the heterogeneous sensors may be different from each other. Therefore, the communication protocol of the gateway for acquiring and integrating the data of the heterogeneous sensors and transferring the data to the vehicle control device is different from the communication protocol of each of the heterogeneous sensors. The same protocol may be used or a different protocol may be used. Timing shift due to conversion and data transmission between different protocols is also a problem to be improved. When the communication protocol of the gateway and the communication protocol between the sensor are different, it is necessary to develop a method for calculating and determining the latency generated during data transmission.

Korean Patent Registration No. 10-1619628

According to the present invention, a shift occurs in the sensing timing between heterogeneous sensors (which may be referred to as a 'slave') and a transmission time when the heterogeneous sensors transmit sensing data to a gateway (which may be referred to as a 'master'). An object of the present invention is to provide an apparatus and method for determining latency for calculating and determining latency.

In addition, the present invention provides a latency determining device and method for calculating and determining latency generated while transmitting data sensed from the sensor to a data analysis device when the communication protocol of the gateway and the communication protocol between the sensors are different The purpose.

In order to achieve the above object, an embodiment of the present invention is configured in a master that receives data sensed by a slave through a network of an autonomous vehicle and transmits it to a data analysis device, and determines the latency for time synchronization. An apparatus, comprising: a transmitter configured to transmit time information of the master to the slave; a receiving unit for receiving an ACK signal transmitted from the slave after receiving the time information of the master; and a processor for calculating a first latency according to a transmission delay of data between the slave and the master based on the ACK signal and a second latency according to a communication protocol conversion between the slave and the master.

Here, the processor may calculate the first latency based on a difference between the time information of the master and the reception time of the ACK signal.

In addition, when the communication protocol of the slave is Ethernet and the communication protocol of the master is CAN, the transmission unit divides the data transmitted from the slave, stores it in a buffer, and sequentially transmits it to the data analysis device, the processor , the second latency may be calculated based on a difference between a transmission time of first transmitted data among the divided data and a transmission time of last transmitted data among the divided data.

On the other hand, in order to achieve the above object, an embodiment of the present invention determines the latency for time synchronization, configured in a master that receives data sensed by a slave through a network of an autonomous vehicle and transmits it to a data analysis device. A method for determining a latency, comprising: a step of calculating a first latency according to a data transmission delay between the slave and the master before data is transmitted from the slave and transmitting it to the slave; and a step b of calculating a second latency according to the communication protocol conversion between the master and the slave and transmitting it to the data analysis device.

In this case, the step a may include transmitting time information of the master to the slave; receiving an ACK signal from the slave whose time is synchronized to match the time information of the master; calculating the first latency based on the ACK signal; and transmitting the first latency to the slave.

In addition, the step b may include: determining the necessity of converting the communication protocol of the slave; As a result of the determination, when the communication protocol of the slave is an Ethernet protocol and the communication protocol of the master is a CAN protocol, dividing and storing the data transmitted from the slave in a buffer and sequentially transmitting the data to the data analysis device; calculating a second latency based on a transmission time difference between first transmitted data and last transmitted data among data sequentially transmitted by the buffer; and transmitting the second latency to the data analysis device.

According to the present invention, when the autonomous vehicle collects data from heterogeneous sensors (slave), it occurs at the time of sensing, transmission, and transmission completion of data according to the data transmission characteristics (transmission delay, transmission method, etc.) between each sensor. There is an effect of increasing the precision and reliability of vehicle control by determining the latency to control the time synchronization.

1 is a conceptual diagram of an apparatus for determining a latency according to an embodiment of the present invention.
2 is a flowchart illustrating a flow of a method for determining latency according to an embodiment of the present invention.
3 is a flowchart illustrating a detailed flow of step S100 in the flowchart of FIG. 2 .
4 shows a DCU block diagram of an autonomous vehicle.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. This is not intended to limit the present invention to specific embodiments, and should be construed to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, terms such as first, second, etc. may be used to describe various components, but the components may not be limited by the terms. The above terms are only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

The term “and/or” may include a combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in between. can be understood On the other hand, when an element is referred to as being “directly connected” or “directly connected” to another element, it may be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression may include the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and one or more other features It may be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary may be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, it is interpreted in an ideal or excessively formal meaning. it may not be

In addition, the following embodiments are provided to more completely explain to those with average knowledge in the art, and the shapes and sizes of elements in the drawings may be exaggerated for more clear explanation.

1 is a conceptual diagram of an apparatus for determining a latency according to an embodiment of the present invention.

Referring to FIG. 1 , the latency determining device 100 according to an embodiment of the present invention is a device configured in a master that receives data sensed by a slave 200 through a network of an autonomous vehicle and transmits it to the data analysis device 300 . As such, it may include a transmitter 110 , a receiver 120 , and a processor 130 .

Here, the slave 200 may refer to a variety of heterogeneous sensors that collect data necessary to operate the autonomous vehicle. For example, the slave 200 may be a radar, a camera, a lidar, a GPS, an inertial measurement sensor (IMU), or the like. For location recognition, IMU, GPS, camera, lidar, etc. are used, and for object recognition such as people and obstacles, a camera, lidar, radar, etc. may be used. In other words, it is difficult to realize autonomous driving only with data sensed by a single sensor, so location recognition and object recognition must be performed through a method of collecting and analyzing data from different types of sensors. Time synchronization should be performed to compensate for latency caused by various reasons.

The master may perform a function of receiving data from the slave 200 and integrating the data. For example, it may serve as a medium for receiving data from sensors based on different communication protocols, converting and matching the communication protocols, and transmitting data to the data analysis device 300 . That is, the master may be formed of a gateway or a switch. The latency determining apparatus 100 according to an embodiment of the present invention may be a device included in the gateway or may be the gateway itself.

The transmitter 110 may transmit time information of the master to the slave 200 . More specifically, the transmitter 110 may transmit a message including its own time information to the slave 200 through the timer 140 of the master. In general, each device (master, slave, data analysis device, vehicle control device, etc.) is provided with an independent timer, and the timer includes time information of each device. In this case, the time information of the master may be aligned with the global time, and the time information of the slave 200 may be aligned with the respective local time, so that the timing may be shifted due to different time information. The plurality of slaves 200 receiving the time information of the master by the transmitter 110 may time-synchronize the time information set to their respective local time to match the time information of the master.

The receiver 120 may receive an acknowledgment (ACK) signal transmitted after the slave 200 receives the time information of the master. More specifically, each of the plurality of slaves 200 may transmit the ACK signal as a message indicating that time synchronization is achieved and ready to transmit data after performing time synchronization to match the master's time information and its own time information . The reception time of each ACK signal received by the receiver 120 from the plurality of slaves 200 may be stored in the memory 150 .

Here, the memory 150 may include a volatile storage device and a non-volatile storage device. The volatile storage device may be, for example, random access memory (RAM), and the non-volatile storage device may be, for example, a read only memory (ROM), a hard disk drive (HDD), or a flash memory.

The processor 130 may be electrically connected to the transmitting unit 110 , the receiving unit 120 , the timer 140 , the memory 150 , and the like, and may electrically control each component, and an electrical circuit for executing software commands can be, thereby performing various data processing and calculations to be described later. To implement this, the processor 130 may be configured as an IC (Integrated Circuit) that performs arithmetic processing. For example, the processor 130 is a CPU, a Digital Signal Processor (DSP), or a Graphics Processing Unit (GPU). ) can be

The processor 130 may calculate the first latency according to the data transmission delay between the slave 200 and the master based on the ACK signal sent by each of the slaves 200 .

More specifically, the processor 130 may calculate the first latency of each of the plurality of slaves 200 based on a difference between the time information of the master and the reception time of the ACK signal received by the master. In this case, the reception times of the respective ACK signals of the slaves 200 stored in the memory 150 may all be different, and all corresponding first latency values may be calculated.

The processor 130 controls the transmitter 110 to transmit a plurality of first latencies calculated in order for the slave 200 to perform first synchronization correcting the first latency to each of the corresponding slaves 200 . can do.

With such a configuration, the slave 200 can precisely complete the first synchronization according to each characteristic. In other words, the plurality of slaves 200, that is, heterogeneous sensors, receive the master's time information before data transmission starts, undergo time synchronization to match their time information and the master's time information, and then Up to a first latency caused by having a transmission delay time can be individually compensated for. That is, when the slaves 200 transmit their sensed data to the master, the slave 200 may include time information for which the first latency is corrected. Therefore, the master can accurately classify and collect data having the same sensing timing and transmit it to the data analysis device 300 , and the data analysis device 300 can analyze the data more precisely and further increase the precision of vehicle control will have

Meanwhile, the processor 130 may calculate the second latency according to the communication protocol conversion between the slave 200 and the master.

More specifically, the processor 130 may determine whether it is necessary to convert the communication protocol between the slave 200 and the master. The communication protocol may be, for example, a controller area network (CAN) protocol or an Ethernet protocol, and each of the slaves 200 may transmit data to the master based on any one communication protocol among the above protocols. In addition, when the master transmits data to the data analysis device 300 , it may be based on any one of the above protocols. That is, the communication protocol between the slave 200 and the master may be CAN/Ethernet, CAN/CAN, Ethernet/CAN, or Ethernet/Ethernet.

At this time, when data is transmitted between the slave 200 and the master and between the master and the data analysis device 300 based on the same communication protocol, it is not a big problem. The problem is that data is transmitted from the slave 200 to the master using Ethernet communication and data is transmitted from the master to the data analysis device 300 using CAN communication. At this time, the problem is the difference in the size of the data that can be transmitted. In the case of CAN communication, data is 8 bytes or less, whereas in the case of Ethernet communication, it has a minimum of 46 bytes and a maximum of 1500 bytes. Therefore, data based on Ethernet communication is CAN Communication cannot be transmitted at once and must be transmitted in divisions. Here, a latency (second latency) occurs due to a transmission time difference between the first transmitted data and the last transmitted data among the divided data. On the other hand, since Ethernet has a larger frame structure than CAN, the entire CAN message can be included in the payload of the Ethernet frame, so that the above-described problem does not occur.

In addition, the processor 130 can identify the communication protocol based on the slave 200 by identifying the protocol type from the port number delivered in the transport layer header, and can determine whether conversion of the communication protocol is required through this. .

In addition, when a communication protocol conversion between the slave 200 and the master is required, the processor 130 may perform the communication protocol conversion. For example, the processor 130 may receive a CAN frame having a format consisting of a CAN header and diagnostic data and convert it into an Ethernet frame having a format consisting of a TCP header, an EthDiag header, and diagnostic data. Here, the processor 130 may indicate to the EthDiag header that data following the frame is CAN diagnostic data. Similarly in the opposite case, the processor 130 may take data after the header in the Ethernet frame format and then combine the CAN header to create a CAN frame.

When the communication protocol of the slave 200 is an Ethernet protocol and the communication protocol of the master is a CAN protocol, the processor 130 divides the data transmitted from the slave 200 and stores it in a buffer, and the data analysis device 300 It is possible to control the transmitter 130 to transmit sequentially.

In this case, the processor 130 may calculate the second latency based on a difference between a transmission time of first transmitted data among the divided data and a transmission time of last transmitted data among the divided data.

Meanwhile, the processor 130 may determine whether the difference between the transmission times exceeds a preset threshold time. Transmission time of each segment of data may be further delayed due to various reasons. In this case, the divided and transmitted data may be data with no utility value that cannot be precisely analyzed (inappropriate data). Since data is continuously transmitted from the slave 200, it may be effective for accurate judgment and control to discard inappropriate data. Accordingly, the processor 130 generates a message for determining inappropriate data when the second latency exceeds a preset threshold time (threshold value), and transmits it to the data analysis device together with the last data through the transmitter 110 This can prevent nonconforming data from being used for cognitive judgment and analysis of autonomous vehicles. For example, when the latency is greater than or equal to the transmission period of the slave 200, a message for determining inappropriate data may be generated.

Through this configuration, the data analysis apparatus 300 can analyze data based on only highly reliable data, and furthermore, the reliability of vehicle control is increased.

The processor 130 controls the transmitter 110 to transmit the calculated second latency to the data analysis device 300 so that the data analysis device 300 performs second synchronization to correct the second latency. can do.

With this configuration, the data analysis device 300 corrects the second latency generated by the communication protocol difference between the slave 200 and the master (more specifically, when converting from the CAN communication protocol to the Ethernet communication protocol) 2 Synchronization can be performed. Accordingly, the data analysis apparatus 300 can analyze data more precisely, and furthermore, has the effect of increasing the precision of vehicle control.

Hereinafter, a latency determination method according to an embodiment of the present invention will be described.

The method of determining the latency according to an embodiment of the present invention may be performed through the apparatus 100 for determining the latency described with reference to FIG. 1 .

2 is a flowchart illustrating a flow of a method for determining latency according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a detailed flow of step S100 in the flowchart of FIG. 2 .

Referring to FIG. 2 , the method for determining latency, which is an embodiment of the present invention, is a method configured in a master that receives data sensed by a slave 200 through a network of an autonomous vehicle and transmits it to the data analysis device 300 . , before the data from the slave 200 is transmitted, step a of calculating the first latency according to the data transmission delay between the slave 200 and the master is performed (S100).

Thereafter, when the master transmits the data received from the slave 200 to the data analysis device 300 , step b of calculating the second latency that may occur according to the communication protocol conversion between the slave 200 and the master is performed. (S200)

At this time, as will be described later, step b (S200) includes a step (S210) of determining the need for conversion of the communication protocol, and the communication protocol underlying the master and the slave 200 is the same or the slave 200 is CAN communication. When data is transmitted and the second latency does not occur, such as when the master transmits data through Ethernet communication, step b may be omitted.

Meanwhile, looking at the detailed flow of step a ( S100 ) with reference to FIG. 3 , first, the step ( S110 ) of transmitting time information of the master to the slave 200 is performed. More specifically, the transmitter 110 may transmit a message including its own time information to the slave 200 through the timer 140 of the master. The slave 200 may prepare to transmit data by correcting its own time information to match the master's time information.

Thereafter, an ACK message is received from the slave 200. (S120) More specifically, the receiving unit 120 performs time synchronization so that the slave 200 matches its own time information with the master's time information based on the master's time information. When the process proceeds and generates an ACK signal and transmits it to the master, the master receives the ACK signal transmitted by the slave 200 .

The master calculates the first latency based on the ACK signal (S130) and transmits the first latency to the slave 200 so that the first synchronization is performed by the slave 200 (S140). More specifically, the processor The 130 may calculate the first latency of each of the plurality of slaves 200 based on a difference between the time information of the master and the reception time of the ACK signal received by the master. In this case, the reception times of the respective ACK signals of the slaves 200 stored in the memory 150 may all be different, and all corresponding first latency values may be calculated.

In addition, the processor 130 transmits the plurality of first latencies calculated in order to cause the slave 200 to perform first synchronization correcting the first latency to the corresponding slaves 200, respectively. Transmission unit 110 can control

On the other hand, referring back to FIG. 2 and looking at the detailed flow of step b (S200), first, the processor 130 determines the necessity of converting the communication protocol between the slave 200 and the master. (S210) At this time, the master performs CAN communication and the slave 200 performs Ethernet communication, the subsequent steps (S210 to S250) of calculating the second latency are performed, and the master and the slave 200 are based on the same communication protocol, or the master communicates over Ethernet and the slave 200 ) transmits data without calculating the second latency when performing CAN communication. (S260)

When the second latency calculation step is described in detail, the processor 130 divides the data received from the slave 200 when the communication protocol of the slave 200 is the Ethernet protocol and the communication protocol of the master is the CAN protocol after dividing the buffer ( buffer) and control the transmission unit 130 to sequentially transmit the divided data to the data analysis device 300 (S220).

At this time, the processor 130 calculates a second latency based on a difference between a transmission time of the first transmitted data among the divided data and a transmission time of the last transmitted data among the divided data (S230).

Meanwhile, the processor 130 may determine whether the transmitted data is nonconforming data. (S240) More specifically, the processor 130 determines whether the calculated second latency exceeds a preset threshold time (threshold value), the nonconforming data It is possible to prevent non-conforming data from being used for cognitive determination and analysis of the autonomous vehicle by generating a message for determining .

Thereafter, the processor 130 controls the transmission unit 110 to use the calculated second latency to the data analysis apparatus 300 so that the data analysis apparatus 300 performs a second synchronization for correcting the second latency. ) to (S250)

In addition, a detailed description of the latency determining method is omitted here because it overlaps with the description of the aforementioned latency determining apparatus 100 .

As described above, according to the present invention, when an autonomous vehicle collects data from heterogeneous sensors (slave), the sensing, transmission timing and There is an effect of increasing the precision and reliability of vehicle control by determining the latency occurring at the time of completion of transmission to control time synchronization.

Although the present invention has been described in detail through specific examples, it is intended to describe the present invention in detail, and the present invention is not limited thereto. It is clear that the transformation or improvement is possible by the person.

All simple modifications or changes of the present invention fall within the scope of the present invention, and the specific scope of protection of the present invention will be made clear by the appended claims.

100: time synchronizer
110: transmission unit
120: receiver
130: processor
140: timer
150: memory
200: slave
300: data analysis device

Claims (6)

A latency determining device for determining latency for time synchronization, configured in a master that receives data sensed by a slave through a network of an autonomous vehicle and transmits it to a data analysis device, comprising:
a transmitter for transmitting time information of the master to the slave;
a receiving unit for receiving an ACK signal transmitted from the slave after receiving the time information of the master; and
A processor for calculating a first latency according to a transmission delay of data between the slave and the master based on the ACK signal and a second latency according to a communication protocol conversion between the slave and the master;
According to claim 1,
The processor is
The device for determining the latency, characterized in that for calculating the first latency based on a difference between the time information of the master and the reception time of the ACK signal.
3. The method of claim 2,
When the communication protocol of the slave is Ethernet and the communication protocol of the master is CAN,
The transmission unit divides the data received from the slave, stores it in a buffer, and sequentially transmits it to the data analysis device,
The processor is configured to calculate the second latency based on a difference between a transmission time of first transmitted data among the divided data and a transmission time of last transmitted data among the divided data. .
A method for determining latency for determining latency for time synchronization, configured in a master configured to receive data sensed by a slave through a network of an autonomous vehicle and transmit it to a data analysis device, the method comprising:
a step of calculating a first latency according to a data transmission delay between the slave and the master before data is transmitted from the slave and transmitting it to the slave; and
A method for determining latency comprising: calculating a second latency according to the communication protocol conversion between the master and the slave and transmitting the second latency to the data analysis device.
5. The method of claim 4,
In step a,
transmitting time information of the master to the slave;
receiving an ACK signal from the slave whose time is synchronized to match the time information of the master;
calculating the first latency based on the ACK signal; and
and transmitting the first latency to the slave.
6. The method of claim 5,
In step b,
determining the necessity of converting the communication protocol of the slave;
As a result of the determination, when the communication protocol of the slave is an Ethernet protocol and the communication protocol of the master is a CAN protocol, dividing and storing the data received from the slave in a buffer and sequentially transmitting the data to the data analysis device;
calculating a second latency based on a transmission time difference between first transmitted data and last transmitted data among data sequentially transmitted by the buffer; and
Transmitting the second latency to the data analysis device; Latency determination method comprising a.

Images