WO2023123416A1 - 同步的方法、装置以及车辆 - Google Patents

同步的方法、装置以及车辆 Download PDF

Info

Publication number
WO2023123416A1
WO2023123416A1 PCT/CN2021/143829 CN2021143829W WO2023123416A1 WO 2023123416 A1 WO2023123416 A1 WO 2023123416A1 CN 2021143829 W CN2021143829 W CN 2021143829W WO 2023123416 A1 WO2023123416 A1 WO 2023123416A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
data
camera device
laser radar
synchronization
Prior art date
Application number
PCT/CN2021/143829
Other languages
English (en)
French (fr)
Inventor
黄梓亮
潘杨杰
赖龙珍
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2021/143829 priority Critical patent/WO2023123416A1/zh
Priority to CN202180029658.3A priority patent/CN116685871A/zh
Publication of WO2023123416A1 publication Critical patent/WO2023123416A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders

Definitions

  • the embodiments of the present application relate to the field of intelligent driving, and more specifically, relate to a synchronization method, device and vehicle.
  • the time synchronization of lidar and camera is particularly important.
  • the existing technology uses the data time stamps of the two to determine whether the two are time-synchronized.
  • the lidar is mechanically rotated and scanned, and the camera is exposed instantaneously, this leads to the time overlap between the two.
  • the number of frames is very small, the accuracy and reliability of data fusion between different sensors are poor, and it is difficult to achieve temporal and spatial synchronization between multiple sensors.
  • the embodiment of the present application provides a synchronization method, device and vehicle, which can control the initial azimuth angle of the lidar rotor and then control the radar rotor under the condition that the resources of the complex programmable logic device CPLD or the field programmable logic gate array FPGA are sufficient.
  • the scanning angle of the corresponding range is triggered to trigger the shooting of the camera device in the corresponding range, so as to realize the time synchronization and/or space-time synchronization of the radar and the camera device; in the case of insufficient resources of the complex programmable logic device CPLD or the field programmable logic gate array FPGA, according to the laser
  • the periodic matching of the radar and the camera device realizes the space-time matching of the radar and the camera device as much as possible. In this way, the synchronization of sampling frequency and phase between the radar and the camera can be achieved, and the accuracy and reliability of data fusion between the radar and the camera can be improved.
  • a time synchronization method includes: determining the synchronization mode of the laser radar and the camera according to the resource state of the complex programmable logic device CPLD or the field programmable logic gate array FPGA, and the laser radar and the camera
  • the synchronization mode of the device includes a first synchronization mode or a second synchronization mode; according to the synchronization mode of the laser radar and the camera device, the laser radar and the camera device are synchronized.
  • the camera device may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc.
  • the image information acquired by the camera device may include still images or video stream information.
  • Determining the synchronization mode of the laser radar and the camera device according to the resource status of the CPLD or FPGA is specifically: when the CPLD or FPGA resources are insufficient, the first time synchronization mode is adopted, and when the CPLD or FPGA resources are sufficient, the second time synchronization mode is adopted.
  • the first synchronous mode may mean that the camera device is triggered by the CPLD or FPGA according to the original frequency.
  • the second synchronization mode can mean that the internal rotor of the lidar rotates to a certain angle or several angles, triggering the CPLD or FPGA to trigger the exposure of the camera device.
  • sufficient CPLD or FPGA resources can mean that CPLD or FPGA can mobilize more resources to process tasks, that is, CPLD or FPGA has strong computing power and current idle capacity, and insufficient CPLD or FPGA resources can mean that CPLD or FPGA can handle tasks. There are relatively few resources that can be scheduled, that is, the computing power and current idle capacity are weak.
  • the basic principle of sensor data fusion is to carry out multi-level and multi-space information complementation and optimal combination processing of various sensors, and finally produce a consistent interpretation of the observation environment.
  • the ultimate goal of information fusion is to derive more useful information based on the separate observation information obtained by each sensor, through multi-level and multi-faceted combinations of information.
  • the data fusion on the lidar and the camera is relatively difficult.
  • the lidar is a slow-scanning device, while the camera is exposed instantaneously.
  • the radar and the camera it is very difficult for the radar and the camera to synchronize time.
  • the accuracy and reliability of data fusion are required to be improved by the radar and the camera, but also the time-space synchronization of the radar and the camera is required.
  • the traditional time synchronization method only mechanically time-stamps the data collected by the radar and the camera based on the same time base. Furthermore, motion compensation can be performed on the data collected by the radar and the camera, because the traditional method just hand over the data to the algorithm application to judge the time difference , if the sampling frequency and phase synchronization between the radar and the camera are not achieved, the time-space synchronization cannot be achieved, and the driving safety of the vehicle cannot be improved.
  • the synchronization mode of the laser radar and the camera device can be determined according to the resource status of the CPLD or FPGA, and the second synchronization mode can be adopted when the resources of the CPLD or FPGA are sufficient to control the initial azimuth angle of the laser radar rotor, and then Control the scanning angle of the radar rotor to trigger the shooting of the camera device in the corresponding range, and realize the time synchronization and/or space-time synchronization of the radar and the camera device; when the CPLD or FPGA resources are insufficient, the first synchronization mode is adopted, according to the lidar and camera Periodic matching of the device, as far as possible to achieve the spatio-temporal matching of the radar and the camera device. In this way, the synchronization of sampling frequency and phase between the radar and the camera can be achieved, and the accuracy and reliability of data fusion between the radar and the camera can be improved.
  • the method further includes: dividing the lidar and camera devices into sensor groups.
  • any number of laser radars and camera devices in any orientation on the vehicle can be divided into the same sensor group according to the layout of the laser radar and camera devices on the vehicle. Furthermore, time synchronization or space-time synchronization is performed on the lidar and the camera device located in the same sensor group.
  • the laser radar and the camera can be divided into sensor groups, and the radar and the camera in the same sensor group can be time-synchronized or time-space-synchronized, thereby improving the Efficiency of data fusion between radar and camera devices.
  • the synchronization mode of the lidar and the imaging device is the first synchronization mode
  • the method further includes: according to the scanning period of the lidar and the imaging device to determine the exposure frequency of the camera; according to the exposure frequency of the first camera, trigger the exposure of the first camera to obtain the first data, the camera includes the first camera; obtain the When the first camera device is exposed, the second data collected by the laser radar; if the difference between the time stamp of the first data and the time stamp of the second data is less than or equal to the first threshold, then The first data and the second data are synchronously processed.
  • the first threshold may be a preset value, which may be determined according to the actual application of the method for synchronizing the laser radar and the camera device. For example, if the synchronization accuracy of the lidar and the camera device is required to be high, the value of the first threshold can be set to a larger value; if the accuracy of the synchronization of the laser radar and the camera device is not high, the value of the first threshold can be set to The numerical value is set to be small.
  • the exposure frequency of the camera can be determined according to the following formula:
  • fc represents the exposure frequency of each camera
  • n represents the number of one or more cameras and n is a positive integer
  • TL represents the scanning period of the lidar.
  • the exposure frequency of the imaging device may be determined according to the scanning period of the lidar and the number of imaging devices, and the exposure of the imaging device may be triggered according to the exposure frequency of the imaging device.
  • judge whether the laser radar and the camera device meet the requirements of time synchronization or space-time synchronization by judging whether the difference between the data time stamps of the first data and the second data collected by the laser radar and the camera device is less than or equal to the first threshold. Only when the requirements of time synchronization or space-time synchronization are met, the obtained results are output to the algorithm application for processing. In this way, lidar and camera devices can match more frame data in time, reduce the workload of algorithm application, and improve the accuracy and reliability of data fusion between radar and camera devices.
  • the rotation period of the lidar is a maximum value within a preset interval.
  • the preset interval may refer to a preset range that the scanning period of the radar can reach, and the value of the preset interval may be determined according to inherent properties of the radar.
  • the inherent scanning period of the radar is divided into two gears, namely 50ms and 100ms, and the preset interval of the radar is 50ms-100ms.
  • the scanning period of the laser radar is set to the maximum value of the preset interval. According to the above example, the scanning period of the laser radar is set to 100ms.
  • the lidar can be set to the slowest working mode suitable for the needs of the working scene, and the exposure frequency of the camera can be determined according to the scanning cycle of the lidar and the number of cameras in the sensor group, so as to trigger Camera exposure.
  • the lidar can better match the exposure characteristics of the camera device, so that the lidar and the camera can be further matched in time to more frame data, and the accuracy of data fusion between the radar and the camera device can be further improved. sex and reliability.
  • the synchronization mode of the lidar and the camera device is the second synchronization mode
  • the method further includes: at the first moment, the initial synchronization mode of the lidar The azimuth angle is set to a first azimuth angle, and a first camera device is provided in the direction of the first azimuth angle, and the camera device includes the first camera device; at the first moment, the laser radar The first data collected and the second data collected by the first camera device; synchronously processing the first data and the second data.
  • the method further includes: determining the second The exposure moment of the camera device is the second moment, and the camera device includes the second camera device; at the second moment, the third data collected by the lidar and the fourth data collected by the second camera device are acquired. data; synchronizing the third data and the fourth data.
  • the exposure time of the second camera can be determined as at the second moment, and trigger the exposure of the second camera device at the second moment.
  • the second camera device may refer to a camera device located behind the first camera device for exposure in the camera device group, or may refer to multiple camera devices located behind the first camera device for exposure.
  • the initial azimuth angles of several laser radars can be set to be the same, and the laser radar and the first camera device in the camera device group can be triggered synchronously, and the laser radar and the second camera device can be triggered according to the calculated sequence time , the time or space-time synchronization between multiple laser radars and camera devices is realized, which ensures the high requirements for time-space synchronization of radar and camera devices in high-speed driving scenarios.
  • the CPLD only performs angle detection once, that is, the CPLD only detects the angle between the first camera device and the laser radar, and the subsequent time is based on the set exposure sequence time, which can reduce the resource consumption of the CPLD.
  • timing triggering of the lidar and the second camera device according to the calculated sequence may mean that the second camera device is triggered sequentially from the beginning of the time to the end of the time according to the calculated time sequence.
  • a synchronization device which includes: a processing unit; the processing unit is used to determine the synchronization mode of the laser radar and the camera according to the resource state of the CPLD or FPGA, and the synchronization mode of the laser radar and the camera
  • the synchronization mode includes a first synchronization mode or a second synchronization mode; the processing unit is further configured to synchronize the lidar and the camera device according to the synchronization mode.
  • the processing unit is further configured to divide the lidar and camera devices into sensor groups.
  • the synchronization mode of the lidar and the camera device includes a first synchronization mode
  • the processing unit is specifically configured to Determine the exposure frequency of the imaging device according to the number of the imaging devices
  • the processing unit is also used to trigger the exposure of the first imaging device according to the exposure frequency of the first imaging device to obtain the first data, the imaging device Including the first camera
  • the device also includes an acquisition unit, configured to acquire the second data collected by the lidar when the first camera is exposed; if the time stamp of the first data and the A difference between time stamps of the second data is less than or equal to a first threshold, and the processing unit is configured to perform synchronization processing on the first data and the second data.
  • the processing unit is specifically configured to determine the exposure frequency of the camera device according to the following formula:
  • fc represents the exposure frequency of each camera
  • n represents the number of one or more cameras and n is a positive integer
  • TL represents the scanning period of the lidar.
  • the synchronization mode of the lidar and the camera device is the second synchronization mode
  • the processing unit is specifically configured to set the initial azimuth of the laser radar as a first azimuth at the first moment, and a first camera is provided in the direction of the first azimuth, and the camera includes the The first camera device; the device also includes an acquisition unit configured to acquire the first data collected by the lidar and the second data collected by the first camera device at the first moment; The processing unit is further configured to perform synchronous processing on the first data and the second data.
  • the processing unit is further configured to determine the The exposure moment of the second camera device is the second moment, and the camera device includes the second camera device; the acquisition unit is also used to acquire the third data collected by the lidar at the second moment and the fourth data collected by the second camera device; the processing unit is further configured to perform synchronous processing on the third data and the fourth data.
  • the processing unit is specifically configured to connect the position of the lidar to the position of the first camera and the position of the lidar to The angle between the lines connecting the positions of the second camera device determines the exposure moment of the second camera device as the second moment.
  • a synchronization device which includes: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory, and the device is used for The methods in each of the above aspects are performed.
  • a computer-readable medium stores program codes, and when the computer program codes are run on a computer, the computer is made to execute the methods in the above aspects.
  • a chip in a fifth aspect, includes: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory, and the device is used to execute Methods in any of the above aspects.
  • a vehicle includes: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory, and the processing in the vehicle A device is used to perform the method in each of the above aspects.
  • Fig. 1 is a functional schematic diagram of a vehicle provided by an embodiment of the present application.
  • Fig. 2 is a schematic diagram of synchronizing a radar and a camera according to an embodiment of the present application.
  • FIG. 3 is a system architecture for synchronizing a radar and a camera device according to an embodiment of the present application.
  • FIG. 4 is another system architecture for synchronizing a radar and a camera device according to an embodiment of the present application.
  • Fig. 5 is a schematic diagram of division of sensor groups provided by the embodiment of the present application.
  • FIG. 6 is a synchronizing method 600 for a radar and a camera provided in an embodiment of the present application.
  • FIG. 7 is a synchronization method 700 for a radar and a camera in a first synchronization mode provided by an embodiment of the present application.
  • FIG. 8 is a method 800 for synchronizing a radar and a camera in a second synchronization mode according to an embodiment of the present application.
  • FIG. 9 is a radar and camera synchronization device 900 provided by an embodiment of the present application.
  • Fig. 10 is a radar and camera synchronization device 1000 provided by an embodiment of the present application.
  • Fig. 1 is a functional schematic diagram of a vehicle 100 provided by an embodiment of the present application.
  • Vehicle 100 may be configured in a fully or partially autonomous driving mode.
  • the vehicle 100 can obtain its surrounding environment information through the perception system 120, and obtain an automatic driving strategy based on the analysis of the surrounding environment information to realize fully automatic driving, or present the analysis results to the user to realize partially automatic driving.
  • Vehicle 100 may include various subsystems such as infotainment system 110 , perception system 120 , computing platform 130 , and display device 140 .
  • vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components.
  • each subsystem and component of the vehicle 100 may be interconnected in a wired or wireless manner.
  • the infotainment system 110 may include a communication system 111 , an entertainment system 112 and a navigation system 113 .
  • Communication system 111 may include a wireless communication system that may wirelessly communicate with one or more devices, either directly or via a communication network.
  • a wireless communication system may use 3G cellular communications, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communications, such as LTE. Or 5G cellular communications.
  • the wireless communication system can use Wi-Fi to communicate with a wireless local area network (wireless local area network, WLAN).
  • the wireless communication system may communicate directly with the device using an infrared link, Bluetooth, or ZigBee.
  • Other wireless protocols, such as various vehicle communication systems, for example, a wireless communication system may include one or more dedicated short range communications (DSRC) devices, which may include communication between vehicles and/or roadside stations Public and/or Private Data Communications.
  • DSRC dedicated short range communications
  • the entertainment system 112 can include a central control screen, a microphone and a sound system. Users can listen to the radio and play music in the car based on the entertainment system; Touch type, users can operate by touching the screen. In some cases, the user's voice signal can be acquired through the microphone, and the user can control the vehicle 100 based on the analysis of the user's voice signal, such as adjusting the temperature inside the vehicle. In other cases, music may be played to the user via a speaker.
  • the navigation system 113 may include a map service provided by a map provider, so as to provide navigation for the driving route of the vehicle 100 , and the navigation system 113 may cooperate with the global positioning system 121 and the inertial measurement unit 122 of the vehicle.
  • the map service provided by the map provider can be a two-dimensional map or a high-definition map.
  • the perception system 120 may include several kinds of sensors that sense information about the environment around the vehicle 100 .
  • the perception system 120 may include a positioning system 121, the positioning system may be a global positioning system (global positioning system, GPS), it may also be a Beidou system or other positioning systems, an inertial measurement unit (inertial measurement unit, IMU) 122, a laser radar 123 , one or more of millimeter wave radar 124 , ultrasonic radar 126 , and camera device 126 .
  • the perception system 120 may also include sensors of the interior systems of the monitored vehicle 100 (eg, interior air quality monitors, fuel gauges, oil temperature gauges, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding properties (position, shape, orientation, velocity, etc.). Such detection and identification is a critical function for safe operation of the vehicle 100 .
  • the positioning system 121 may be used to estimate the geographic location of the vehicle 100 .
  • the inertial measurement unit 122 is used to sense the position and orientation changes of the vehicle 100 based on inertial acceleration.
  • inertial measurement unit 122 may be a combination accelerometer and gyroscope.
  • the lidar 123 may utilize laser light to sense objects in the environment in which the vehicle 100 is located.
  • lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.
  • the millimeter wave radar 124 may utilize radio signals to sense objects within the surrounding environment of the vehicle 100 .
  • radar 126 may be used to sense the velocity and/or heading of objects.
  • the ultrasonic radar 125 may sense objects around the vehicle 100 using ultrasonic signals.
  • the camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100 .
  • the camera device 126 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc., and the image information acquired by the camera device 126 may include still images or video stream information.
  • the computing platform 130 may include processors 131 to 13n (n is a positive integer), the processor is a circuit with signal processing capabilities, in one implementation, the processor may be a circuit with instruction reading and execution capabilities, For example, a central processing unit (central processing unit, CPU), a microprocessor, a graphics processing unit (graphics processing unit, GPU) (which can be understood as a microprocessor), or a digital signal processor (digital signal processor, DSP), etc.
  • CPU central processing unit
  • microprocessor graphics processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • the processor can realize a certain function through the logical relationship of the hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, for example, the processor is an application-specific integrated circuit, ASIC) or a hardware circuit implemented by a programmable logic device (programmable logic device, PLD), such as FPGA.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • FPGA field-programmable gate array
  • the computing platform 130 can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (neural network processing unit, NPU), tensor processing unit (tensor processing unit, TPU), deep learning processing Unit (deep learning processing unit, DPU), etc.
  • the computing platform 130 may further include a memory for storing instructions, and some or all of the processors 131 to 13n may call the instructions in the memory and execute them to implement corresponding functions.
  • the computing platform 130 may control functions of the vehicle 100 based on input received from various subsystems (eg, the perception system 120 ). In some embodiments, computing platform 130 is operable to provide control over many aspects of vehicle 100 and its subsystems.
  • FIG. 1 should not be construed as limiting the embodiment of the present application.
  • An autonomous vehicle traveling on a road may identify objects within its surroundings to determine adjustments to its current speed.
  • the objects may be other vehicles, traffic control devices, or other types of objects.
  • each identified object may be considered independently and based on the object's respective characteristics, such as its current speed, acceleration, distance to the vehicle, etc., may be used to determine the speed at which the autonomous vehicle is to be adjusted.
  • the vehicle 100 or a sensing and computing device (e.g., computing platform 130) associated with the vehicle 100 may base on the identified characteristics of the object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.) etc.) to predict the behavior of the identified object.
  • each identified object is dependent on the behavior of the other, so all identified objects can also be considered together to predict the behavior of a single identified object.
  • the vehicle 100 is able to adjust its speed based on the predicted behavior of the identified object.
  • the autonomous vehicle is able to determine what steady state the vehicle will need to adjust to (eg, accelerate, decelerate, or stop) based on the predicted behavior of the object.
  • other factors may also be considered to determine the speed of the vehicle 100 , such as the lateral position of the vehicle 100 in the traveling road, the curvature of the road, the proximity of static and dynamic objects, and the like.
  • the computing device may provide instructions to modify the steering angle of the vehicle 100 such that the self-driving vehicle follows a given trajectory and/or maintains contact with objects in the vicinity of the self-driving vehicle (e.g., , the safe lateral and longitudinal distances of cars in adjacent lanes on the road.
  • objects in the vicinity of the self-driving vehicle e.g., , the safe lateral and longitudinal distances of cars in adjacent lanes on the road.
  • the above-mentioned vehicle 100 may be a car, truck, motorcycle, public vehicle, boat, airplane, helicopter, lawn mower, recreational vehicle, playground vehicle, construction equipment, tram, golf cart, train, etc., the embodiment of the present application There is no particular limitation.
  • Complex programmable logic device complex programmable logic device, CPLD
  • CPLD complex programmable logic device
  • a CPLD is equivalent to containing several programmable array logics, and the interconnection lines between each logic block can also be programmatically planned and recorded.
  • CPLD uses this all-in-one integration method to make one Circuits that can only be formed with thousands or even hundreds of thousands of logic gates can be realized.
  • Field programmable logic gate array (field programmable gate array, FPGA), which is developed on the basis of programmable logic devices.
  • FPGA Field programmable logic gate array
  • Micro control unit also known as processing unit, control unit or single chip microcomputer (single chip microcomputer, SCM) refers to the emergence and development of large-scale integrated circuits.
  • the advanced CPU, RAM, ROM, timer counter and various I/O interfaces are integrated on one chip to form a chip-level computer, which can be used for different combination controls for different applications.
  • Internet service provider also known as Internet service provider, Internet service provider, network service provider, refers to a company that provides Internet access services.
  • GMSL gigabit multimedia serial links
  • Local area network switch refers to a device used for data exchange in a switched local area network.
  • Pulse per second Pulse per second, PPS: the abbreviation of pulse per second, used in the communication industry.
  • Motion compensation It is a method to describe the difference between adjacent frames, specifically how each small block of the previous frame moves to a certain position in the current frame.
  • Laser radar also known as light detection and ranging (LDR), is a sensing technology that uses light sources and receivers for remote object detection and ranging. In the vehicle field, the role of lidar is to detect and model obstacles around the vehicle.
  • Time synchronization The unified host provides the reference time for each sensor, and each sensor adds time stamp information to the data collected independently according to the respective time after calibration, so that the time stamps of all sensors can be synchronized, but due to the The collection periods of the sensors are independent of each other, and it may not be possible to guarantee that the same information is collected at the same time.
  • Hard synchronization Use the same hardware to issue trigger acquisition commands at the same time to realize time synchronization of each sensor acquisition and measurement. Acquire the same information at the same time.
  • FIG. 2 is a schematic diagram of radar and camera synchronization provided by an embodiment of the present application.
  • the radar and camera synchronization method in FIG. 2 can be applied to the driving process of the vehicle 100 in FIG. 1 .
  • the basic principle of sensor data fusion is to carry out multi-level and multi-space information complementation and optimal combination processing of various sensors, and finally produce a consistent interpretation of the observation environment.
  • This not only takes advantage of the mutual cooperation of multiple sensors, but also comprehensively processes data from other information sources to improve the intelligence of the entire sensor system.
  • the data fusion of lidar and camera is relatively difficult. Specifically, compared to the camera, the lidar is a slow-scanning device, while the camera is exposed instantaneously. In the scene where the vehicle is driving at a low speed, it is very difficult for the radar and the camera to synchronize time. For the high-speed driving scene of the vehicle, not only the accuracy and reliability of data fusion are required to be improved by the radar and the camera, but also the time-space synchronization of the radar and the camera is required.
  • motion compensation can be performed on the data collected by the radar and the camera, because the traditional method just hand over the data to the algorithm application to judge the time difference , if the sampling frequency and phase synchronization between the radar and the camera are not achieved, the time-space synchronization cannot be achieved, and the driving safety of the vehicle cannot be improved.
  • the embodiment of the present application provides a method for synchronizing radar and camera.
  • the initial azimuth angle of the lidar rotor can be controlled, and then the scanning angle of the radar rotor can be controlled to trigger the camera shooting in the corresponding range, so as to realize the radar and camera.
  • Figure 2(b) after the lidar and the camera are triggered synchronously, the point cloud scanned by the radar can completely cover the actual object to be photographed, realizing the time-space synchronization of the radar and the camera.
  • sufficient CPLD or FPGA resources can mean that CPLD or FPGA can mobilize more resources to process tasks, that is, CPLD or FPGA has strong computing power and current idle capacity, and insufficient CPLD or FPGA resources can mean that CPLD or FPGA can handle tasks. There are relatively few resources that can be scheduled, that is, the computing power and current idle capacity are weak.
  • the lidar in an automatic driving system, after the lidar is powered on, it will rotate periodically according to its own working mode. For example, the lidar rotates according to a 360° cycle;
  • the trigger signal sent by the original working frequency triggers the exposure of the camera module.
  • it can be achieved in the following two ways:
  • the camera is triggered by the CPLD/FPGA according to the original frequency.
  • This method can be called the first synchronization mode or the soft synchronization mode.
  • the internal rotor of the lidar rotates to a certain angle or several angles, triggering the CPLD/FPGA to trigger the camera exposure.
  • This method can be called the second synchronization mode, which can be called the hard synchronization mode.
  • FIG. 3 is a system architecture for synchronizing a radar and a camera device according to an embodiment of the present application.
  • the system architecture for synchronizing a radar and a camera device in FIG. 3 can be applied to the vehicle 100 in FIG. 1 .
  • Figure 3 specifically may include the following steps:
  • different sensors may be divided into different sensor groups according to the layout of the sensors of the radar and the camera on the vehicle, and then the time synchronization of the radar and the camera is completed according to the divided sensor groups.
  • any number of sensors in any orientation can be selected as the sensor group according to actual needs.
  • the camera device may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc.
  • the image information acquired by the camera device may include still images or video stream information.
  • the multi-sensor synchronization mode can be set according to the resource application of the CPLD or FPGA.
  • the hard synchronization mode is adopted when the CPLD or FPGA resources are sufficient, and the soft synchronization mode is adopted when the CPLD or FPGA resources are insufficient.
  • a specific setting manner may be to set the time synchronization mode of multiple sensors according to a switch on the domain controller.
  • the soft synchronization mode may mean that the camera device is triggered by the CPLD/FPGA according to the original frequency.
  • the hard synchronization mode can mean that the internal rotor of the lidar rotates to a certain angle or several angles, triggering the CPLD/FPGA to trigger the exposure of the camera device.
  • the determination may be made according to the result of setting the synchronization mode in step S302. If in step S302, the synchronization mode is set to the hard synchronization mode by the switch of the domain controller, then the synchronization mode judged here is the hard synchronization mode, if the synchronization mode is set to the soft synchronization mode by the switch of the domain controller in the step S302 , the synchronization mode judged here is the soft synchronization mode.
  • step S303 if it is judged in step S303 that the synchronization mode is the hard synchronization mode, then in this step, the initial azimuths of the m lidars are aligned, where m is a positive integer.
  • aligning the initial azimuth angles of the lidars may refer to setting the rotor angles of different lidars to be the same.
  • the rotor of the lidar reaches the first azimuth range of the camera device group, the CPLD sends an exposure signal to trigger the camera device, and records the time t 1 .
  • the laser radar sends a signal to the CPLD, and the CPLD sends an exposure signal to the first camera after receiving the signal of the laser radar , to trigger the exposure of the first camera, and record the time t 1 of the exposure of the camera.
  • the first azimuth angle range of the camera device group may be preset according to parameters of the camera device or requirements of practical applications.
  • the CPLD sets the time t 2 to t n corresponding to the azimuth angle ranges of the second to the nth camera according to the layout of the sensor and the time synchronization framework.
  • the CPLD can set the time t2 to tn when the lidar rotor reaches the azimuth angle range of the second to nth cameras in the camera group, where n is greater than 2, according to the sensor layout and time synchronization architecture integer.
  • the CPLD sequentially sends exposure signals according to time t 2 to t n to trigger exposure of the camera device.
  • the CPLD sends exposure signals to the camera device sequentially according to the time t2 to tn set in step S306 to trigger the camera device exposure.
  • the cameras are time-synchronized or time-space-synchronized.
  • step S307 the time synchronization or time-space synchronization of the radar and the camera device is achieved, it is determined in this step to exit the automatic driving mode, and the synchronization process ends. If in step S307, the time synchronization or space-time synchronization of the radar and the imaging device is not achieved, step S305 is re-executed, and the synchronization process of the radar and the imaging device is performed again.
  • the sampling frequency and phase of the radar and the camera are synchronized by means of synchronous triggering of the radar and the camera, which improves the accuracy and reliability of data fusion between the radar and the camera.
  • FIG. 4 is another system architecture for synchronizing a radar and a camera device according to an embodiment of the present application.
  • the system architecture of synchronizing the radar and the camera device in FIG. 4 can be applied to the vehicle 100 in FIG. 1 .
  • Fig. 4 may specifically include the following steps.
  • different radar and camera devices can be divided into different sensor groups according to the layout of the sensors on the vehicle, and then the synchronization of the radar and camera devices can be completed according to the divided sensor groups.
  • any number of sensors in any orientation can be selected as the sensor group according to actual needs.
  • the camera device may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc.
  • the image information acquired by the camera device may include still images or video stream information.
  • the multi-sensor synchronization mode can be set according to the resource application of the CPLD or FPGA.
  • the hard synchronization mode is adopted when the CPLD or FPGA resources are sufficient, and the soft synchronization mode is adopted when the CPLD or FPGA resources are insufficient.
  • a specific setting manner may be to set the time synchronization mode of multiple sensors according to a switch on the domain controller.
  • the soft synchronization mode may mean that the camera device is triggered by the CPLD/FPGA according to the original frequency.
  • the hard synchronization mode can mean that the internal rotor of the lidar rotates to a certain angle or several angles, triggering the CPLD/FPGA to trigger the exposure of the camera device.
  • the judgment may be made according to the result of setting the time synchronization mode in step S402. If in step S402 the synchronization mode is set to the hard synchronization mode by the switch of the domain controller, then the synchronization mode judged here is the hard synchronization mode, if the synchronization mode is set to the soft synchronization mode by the switch of the domain controller in the step S402 , the synchronization mode judged here is the soft synchronization mode.
  • step S403 judges that the mode of time synchronization is soft synchronization mode, then in this step, the frequency of the radar is set to the lowest gear, that is, the radar duty cycle is the slowest, preferably, the scanning cycle of the radar is set to 100ms.
  • the slowest working cycle of the radar may refer to the gear with the largest scanning period among the selectable working gears of the radar.
  • the laser radar is divided into two gears according to the factory settings, which are 100ms and 50ms.
  • the gear with the largest selection cycle is the gear of 100ms. Therefore, when the scanning cycle of the laser radar The duty cycle of is called "slowest".
  • the frequency fc of the camera device can be calculated by the following formula.
  • n the number of imaging devices, and its value is a positive integer.
  • T L represents the scanning period of the laser radar.
  • one laser radar corresponds to the layout of 6 imaging devices.
  • the lidar rotates normally, and the CPLD triggers the exposure of the camera device according to the PPS of fc frequency times.
  • the exposure frequency fc of the imaging device is calculated through the formula in step S406, and the CPLD triggers the exposure of the imaging device using a PPS that is a multiple of the frequency of fc.
  • the value of fc frequency multiple is a positive integer.
  • the frequency multiplier described in the embodiment of the present application is also the frequency multiplier.
  • the radar and the camera acquire the data timestamps of the lidar and the camera when the camera is exposed, and determine whether the difference between the data timestamps of the lidar and the camera is less than or equal to the first threshold. If the condition is met, it means that the radar and the camera are time-synchronized or In order to meet the requirements of time-space synchronization, the data collected by the radar and camera devices is output to the algorithm application for processing.
  • step S407 if in step S407, the time stamps of the lidar and the camera device are less than or equal to the first threshold, and the time synchronization or space-time synchronization requirements of the radar and camera device are met, then it is judged in this step to exit the automatic driving mode, and the end Synchronization process. If in step S407, the time synchronization requirement of the radar and the camera device is not met, step S406 is executed again, and the CPLD triggers the camera device to perform exposure again according to the PPS of fc frequency times.
  • the lidar can be set to the slowest working mode suitable for the needs of the work scene, and the camera can be set to the corresponding faster working mode, so that when hard time synchronization cannot be achieved due to resource constraints, the radar and camera More frames of data are matched in time, improving the accuracy and reliability of data fusion between radar and camera devices.
  • Fig. 5 is a schematic diagram of a sensor group provided by an embodiment of the present application.
  • the sensor group in FIG. 5 can be applied to the system architecture in which the radar and the camera device are synchronized in FIG. 3 or 4 .
  • the sensor group can include radars and camera devices.
  • sensor group A includes all radars and all camera devices in the figure, so that two radars and 7 camera devices perform Time synchronization.
  • a radar and a camera located in front of the vehicle can be divided into a sensor group A, and a radar and 6 camera devices located on the top of the vehicle can be divided into a sensor group B, such that , the radar and camera devices located in the A synchronization group and the B synchronization group can be time-synchronized respectively in the respective sensor groups.
  • the sensor group described in the embodiment of the present application may also be called a synchronization group, which is not distinguished in the present application.
  • the radar and camera devices can be divided into synchronous groups through graphical configuration software, and the divided results are sent to the time synchronization module of the sensor.
  • the radar and camera devices can be divided into sensor groups, and the radar and camera devices in the same sensor group can be hard-synchronized or soft-synchronized, which can improve the radar and camera Efficiency of data fusion between camera devices.
  • the method for synchronizing the radar and the camera device will be introduced below in conjunction with the process shown in FIG. 6 .
  • FIG. 6 is a synchronizing method 600 for a radar and a camera provided in an embodiment of the present application.
  • the synchronization method 600 is applicable in the vehicle 100 of FIG. 1 .
  • Method 600 may include the following steps.
  • Determining the synchronization mode of the lidar and the camera device can be determined according to the resource status of the CPLD or FPGA.
  • the first time synchronization mode is adopted, and when the CPLD or FPGA resources are sufficient, the second time synchronization mode is adopted.
  • the first synchronous mode may mean that the camera device is triggered by the CPLD or FPGA according to the original frequency.
  • the second synchronization mode can mean that the internal rotor of the lidar rotates to a certain angle or several angles, triggering the CPLD or FPGA to trigger the exposure of the camera device.
  • sufficient CPLD or FPGA resources can mean that CPLD or FPGA can mobilize more resources to process tasks, that is, CPLD or FPGA has strong computing power and current idle capacity, and insufficient CPLD or FPGA resources can mean that CPLD or FPGA can handle tasks. There are relatively few resources that can be scheduled, that is, the computing power and current idle capacity are weak.
  • the synchronization mode of the radar and the camera device may not be selected, and the lidar and the camera device are directly synchronized according to the first synchronization mode or the second synchronization mode.
  • the method before performing step S601, the method further includes: dividing lidar and camera imaging devices into sensor groups.
  • any number of laser radars and camera devices in any orientation on the vehicle can be divided into the same sensor group according to the layout of the laser radar and camera devices on the vehicle. Then, time synchronization or space-time synchronization is performed on the lidar and the camera located in the same sensor group.
  • different sensor groups can be divided in the manner described in FIG. 5 .
  • the laser radar and the camera can be divided into sensor groups, and the radar and the camera in the same sensor group can be time-synchronized or time-space-synchronized, thereby improving the Efficiency of data fusion between radar and camera devices.
  • step S602 synchronizing the lidar and the imaging device includes: The number of devices determines the exposure frequency of the camera; according to the exposure frequency of the first camera, triggers the exposure of the first camera to obtain the first data, the camera includes the first camera; obtains the second When a camera device is exposed, the second data collected by the laser radar; if the difference between the time stamp of the first data and the time stamp of the second data is less than or equal to the first threshold, the first The data and the second data are processed synchronously.
  • the frequency fc of the camera device can be calculated by the following formula:
  • n the number of camera devices, and its value is a positive integer.
  • the CPLD triggers the exposure of the camera device according to the exposure frequency of the camera device, and obtains the first data and the second data respectively collected by the lidar and the camera device at the exposure time of the camera device. Then, the CPLD compares the time stamps of the acquired first data and the second data with the first threshold, and then performs different processing on the data collected by the laser radar and the camera device.
  • the first threshold may be a preset value, which may be determined according to the actual application of the time synchronization method. For example, if the synchronization accuracy of the laser radar and the camera device is required to be high, the value of the first threshold can be set to a larger value; if the accuracy of the synchronization of the laser radar and the camera device is not high, the first threshold can be set to The numerical value is set to be small. If the difference between the data time stamps of the first data and the second data is less than or equal to the first threshold, it means that the lidar and the camera meet the synchronization requirements, and the first data and the second data can be output to the algorithm application for processing. If the difference between the data time stamps of the first data and the second data is greater than the first threshold, it means that the lidar and the camera device do not meet the synchronization requirement, and the first data and the second data need to be discarded.
  • the rotation period of the lidar is a maximum value within a preset interval.
  • the preset interval may refer to a preset range that the scanning period of the radar can reach, and the value of the preset interval may be determined according to inherent properties of the radar.
  • the inherent scanning period of the radar is divided into two gears, namely 50ms and 100ms, and the preset interval of the radar is 50ms-100ms.
  • the scanning period of the laser radar is set to the maximum value of the preset interval. According to the above example, the scanning period of the laser radar is set to 100ms.
  • the laser radar can be set to the slowest working mode suitable for the requirements of the working scene, and the exposure frequency of the camera device can be determined according to the scanning cycle of the laser radar and the number of camera devices corresponding to the laser radar, and the exposure frequency of the camera device can be determined according to the exposure frequency of the camera device.
  • the frequency triggers the exposure of the camera.
  • judge whether the lidar and the camera device meet the requirements of time synchronization by judging whether the difference between the data time stamps of the lidar and the camera device is less than or equal to the first threshold. If the requirements of time synchronization are met, Only then will the obtained results be output to the algorithm application for processing. In this way, the lidar and the camera device can match more frame data in time, reduce the workload of algorithm application, and improve the accuracy and reliability of data fusion between the radar and camera device.
  • step S601 it is determined that the synchronization mode of the lidar and the imaging device is the second synchronization mode
  • step S602 synchronizing the lidar and the imaging device includes: at the first moment, the synchronization mode of the lidar The initial azimuth angle is set to a first azimuth angle, and a first camera device is arranged in the direction of the first azimuth angle, and the camera device includes the first camera device; at the first moment, the laser The first data collected by the radar and the second data collected by the first camera device; synchronously processing the first data and the second data.
  • the initial azimuth angles of m lidars may be set to be equal, where m is an integer greater than 0.
  • the initial azimuth angles of the m lidars can all be set to zero degrees, and the phase angles of the m lidars with the same initial azimuth angles are within the azimuth angle range of the first camera in the camera device group.
  • the m laser radars and the first camera devices in the camera device group are triggered synchronously, and the synchronous triggering time of the laser radars and camera devices is recorded as the first moment, and the laser radars and the first camera devices are respectively collected at the first moment
  • the first data and the second data are processed synchronously.
  • step S602 may further include: determining the exposure moment of the second camera device as the second moment according to the positional relationship of the lidar, the first camera device, and the positional relationship of the second camera device , the imaging device includes the second imaging device; at the second moment, the third data collected by the lidar and the fourth data collected by the second imaging device are obtained; the third data and the The fourth data is processed synchronously.
  • the exposure time of the second camera can be determined as at the second moment, and trigger the exposure of the camera device at the second moment.
  • the second camera device may refer to a camera device located behind the first camera device for exposure in the camera device group, or may refer to multiple camera devices located behind the first camera device for exposure.
  • the second moment of exposure of the second camera can be specifically calculated by the following formula:
  • T L represents the scanning period of the lidar
  • ⁇ n represents the angle between the nth camera and the initial orientation of the radar.
  • the scanning period of the radar is 100ms
  • the angle between the second camera and the radar is ⁇ /3
  • the CPLD triggers the exposure of the second camera device through the ISP.
  • the camera is exposed, the third data collected by the laser radar and the fourth data collected by the second camera device; the CPLD performs synchronous processing on the third data and the fourth data.
  • the initial azimuth angles of several laser radars can be set to be the same, and the laser radar and the first camera device in the camera device group can be triggered synchronously, and the laser radar and the second camera device can be triggered according to the calculated sequence time , the synchronization between multiple radars and camera devices is achieved, which ensures the high requirements for time-space synchronization of radar and camera devices in the scene of high-speed driving of vehicles.
  • the CPLD only performs angle detection once, that is, the CPLD only detects the angle between the first camera device and the laser radar, and the subsequent time is based on the set exposure sequence time, which can reduce the resource consumption of the CPLD.
  • FIG. 7 is a synchronization method 700 for a radar and a camera in a first synchronization mode provided by an embodiment of the present application.
  • the radar and camera time synchronization method 700 in the first synchronization mode in FIG. 7 can be applied to the vehicle 100 in FIG. 1 .
  • Method 700 may include the following steps.
  • S701 set the frequency of the lidar to the lowest gear, that is, the slowest working cycle.
  • the inherent scanning period of the radar is divided into two gears, namely 50ms and 100ms, and setting the frequency of the lidar to the lowest gear is to set the scanning period of the lidar to 100ms.
  • 100 ms is a preferred setting of the radar scanning period in the embodiment of the present application.
  • the frequency fc of the camera can be calculated by the following formula:
  • n the number of cameras, and its value is a positive integer.
  • T L represents the scanning period of the laser radar.
  • one laser radar in the same resource group corresponds to 6 cameras, and the scanning period of the cameras is 100ms.
  • the radar rotates normally, and the CPLD triggers the camera exposure according to the fc multiplier PPS.
  • the CPLD triggers camera exposure using a PPS that is a multiple of the frequency fc.
  • the value of fc frequency multiple is a positive integer.
  • the lidar and camera data time stamps are less than or equal to the first threshold, that is, meet the time synchronization requirement, and output to the algorithm for processing.
  • the time stamps of the lidar and the camera are less than or equal to the first threshold, and if the conditions are met, it means that the time synchronization requirements of the radar and the camera are met, and the result is output to the algorithm application process.
  • the obtained data can be corrected to improve the accuracy of the data.
  • the lidar can be set to the slowest working mode suitable for the needs of the work scene, and the camera can be set to the corresponding faster working mode. More frame data are matched on the camera, which improves the accuracy and reliability of data fusion between radar and camera.
  • FIG. 8 is a method 800 for synchronizing a radar and a camera in a second synchronization mode according to an embodiment of the present application.
  • the radar and camera synchronization method 800 in the second synchronization mode can be applied to the vehicle 100 in FIG. 1 .
  • Method 800 may include the following steps.
  • the initial azimuth angles of the m laser radars are set to 0 degrees.
  • the initial azimuths of the m lidars are set to 0 degrees to ensure that the m radars can be triggered synchronously, and the initial azimuths of the m lidars are located at the first camera of the camera group Azimuth range.
  • m is a positive integer.
  • the CPLD sets the time t 2 to t n corresponding to the azimuth angle ranges of the second to the nth cameras according to the sensor layout and the time synchronization framework.
  • the angle between the radar and each camera in the camera group can be measured, and after the angle is determined, it can be calculated
  • the exposure time t 2 to t n of each camera in the outgoing camera group can be specifically calculated by the following formula:
  • T L represents the scanning period of the lidar
  • ⁇ n represents the angle between the nth camera and the initial orientation of the radar.
  • the scanning period of the radar is 100ms
  • the included angle between the second camera and the radar is ⁇ /3
  • the CPLD triggers the exposure of the second camera through the ISP.
  • the triggering times t 3 to t n of the third camera to the nth camera can be calculated.
  • the CPLD sequentially triggers camera exposure through the ISP according to time t 2 to t n .
  • the CPLD sequentially triggers the exposure of the second to nth cameras through the ISP according to the set time from t2 to tn .
  • the initial azimuth angles of several lidars can be set to 0 degrees, and the lidar and the first camera in the camera group are synchronously triggered, and the radar and the second camera to the nth camera are sequenced according to the calculated sequence time
  • the triggering method realizes the synchronization between multiple radars and cameras, which ensures the high requirements for the time-space synchronization of radars and cameras in the high-speed driving scene.
  • the CPLD only performs angle detection once, that is, the CPLD only detects the angle between the first camera and the lidar, and the subsequent sequence is based on the set sequence, which can reduce the resource consumption of the CPLD.
  • FIG. 9 is a schematic diagram of a radar and camera synchronization device 900 provided by an embodiment of the present application.
  • the device 900 can be used in the vehicle 100 of FIG. 1 .
  • the apparatus 900 may include an acquisition unit 910 , a storage unit 920 and a processing unit 930 .
  • the acquisition unit 910 may implement a corresponding communication function, and the acquisition unit 910 may also be called a communication interface or a communication unit for acquiring data.
  • the storage unit 920 may be used to store corresponding instructions and/or data, and the processing unit 930 is used to perform data processing.
  • the processing unit 930 can read instructions and/or data in the storage unit, so that the device implements the aforementioned method embodiments.
  • the radar and camera synchronization device includes: a processing unit 930; the processing unit 930 is used to determine the synchronization mode of the laser radar and the camera device according to the resource status of the CPLD or FPGA, and the synchronization mode of the laser radar and the camera device includes the first A synchronization mode or a second synchronization mode; the processing unit 930 is further configured to synchronize the lidar and the imaging device according to the synchronization mode of the lidar and the imaging device.
  • the processing unit 930 is further configured to divide the lidar and camera imaging devices into sensor groups.
  • the synchronization mode of the lidar and the imaging device is the first synchronization mode
  • the processing unit 930 is specifically configured to , to determine the exposure frequency of the camera, the camera includes the first camera; the processing unit 930 is further configured to trigger the exposure of the first camera according to the exposure frequency of the first camera, to obtain The first data, the camera includes the first camera; the device also includes an acquisition unit 910, configured to acquire the second data collected by the lidar when the first camera is exposed; if the The difference between the time stamp of the first data and the time stamp of the second data is less than or equal to a first threshold, and the processing unit 930 is configured to perform synchronization processing on the first data and the second data.
  • the processing unit 930 is specifically configured to determine the exposure frequency of the camera device according to the following formula:
  • fc represents the exposure frequency of the imaging device
  • n represents the number of one or more imaging devices and n is a positive integer
  • TL represents the scanning period of the lidar.
  • the synchronization mode of the lidar and the camera device is the second synchronization mode
  • the processing unit 930 is specifically configured to set the initial azimuth of the lidar to the second synchronization mode at the first moment.
  • An azimuth a first camera is provided in the direction of the first azimuth, and the camera includes the first camera;
  • the device also includes an acquisition unit 910, and the acquisition unit 910 is used to acquire At the first moment, the first data collected by the lidar and the second data collected by the first camera device;
  • the processing unit 930 is further configured to process the first data and the second data for synchronization.
  • the processing unit 930 is further configured to determine the exposure of the second camera device according to the lidar, the positional relationship of the first camera device, and the positional relationship of the second camera device. The moment is the second moment; the acquisition unit 910 is also configured to acquire the third data collected by the laser radar and the fourth data collected by the second camera device at the second moment; the processing unit 930 , further configured to perform synchronization processing on the third data and the fourth data.
  • the processing unit 930 is specifically configured to: The angle between the lines connecting the positions determines the exposure moment of the second camera device as the second moment.
  • FIG. 10 is a schematic diagram of a radar and camera synchronization device 1000 provided by an embodiment of the present application.
  • the device 1000 can be used in the vehicle 100 of FIG. 1 .
  • the radar and time synchronization device includes: a memory 1010 , a processor 1020 , and a communication interface 1030 .
  • the memory 1010, the processor 1020, and the communication interface 1030 are connected through an internal connection path, the memory 1010 is used to store instructions, and the processor 1020 is used to execute the instructions stored in the memory 1020 to control the input/output interface 1030 to receive/send At least some parameters of the second channel model.
  • the memory 1010 may be coupled to the processor 1020 via an interface, or may be integrated with the processor 1020 .
  • the above-mentioned communication interface 1030 implements communication between the communication device 1000 and other devices or communication networks by using a transceiver device such as but not limited to a transceiver.
  • the above-mentioned communication interface 1030 may also include an input/output interface (input/output interface).
  • each step of the above method may be implemented by an integrated logic circuit of hardware in the processor 1020 or instructions in the form of software.
  • the methods disclosed in the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.
  • the software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register.
  • the storage medium is located in the memory 1010, and the processor 1020 reads the information in the memory 1010, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.
  • An embodiment of the present application also provides a computer-readable medium, the computer-readable medium stores program codes, and when the computer program codes are run on a computer, the computer is made to perform any of the above-mentioned steps in FIG. 6 to FIG. 8 . a way.
  • An embodiment of the present application also provides a chip, including: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory, so as to execute the above-mentioned steps in FIGS. Either method in Figure 8.
  • the embodiment of the present application also provides a vehicle, including: at least one processor and a memory, the at least one processor is coupled with the memory, and is used to read and execute the instructions in the memory, so as to execute the above-mentioned FIG. 6 to Either method in Figure 8.
  • the embodiment of the present application also provides a vehicle, including any radar and camera device time device shown in FIG. 9 or FIG. 10 .
  • the above-mentioned processor can be a central processing unit (central processing unit, CPU), and the processor can also be other general-purpose processors, digital signal processors (digital signal processor, DSP), dedicated integrated Circuit (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a portion of the processor may also include non-volatile random access memory.
  • the processor may also store device type information.
  • serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be implemented in this application.
  • the implementation of the examples constitutes no limitation.
  • a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer.
  • an application running on a computing device and the computing device can be components.
  • One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers.
  • these components can execute from various computer readable media having various data structures stored thereon.
  • a component may, for example, be based on a signal having one or more packets of data (e.g., data from two components interacting with another component between a local system, a distributed system, and/or a network, such as the Internet via a signal interacting with other systems). Communicate through local and/or remote processes.
  • packets of data e.g., data from two components interacting with another component between a local system, a distributed system, and/or a network, such as the Internet via a signal interacting with other systems.
  • the disclosed systems, devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the functions described are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Abstract

本申请实施例提供了一种同步的方法、装置以及车辆。该方法包括根据复杂可编程逻辑器件CPLD或现场可编程逻辑门阵列FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；根据所述激光雷达和所述摄像装置的同步模式，对所述激光雷达和摄像装置进行同步。通过上述方法，可以根据CPLD或FPGA的资源状态确定激光雷达和相机的同步模式，并根据确定的同步模式对激光雷达和摄像装置进行时间或时空同步，可以提高雷达和摄像装置数据融合的准确性和可靠性。

Description

同步的方法、装置以及车辆 技术领域
本申请实施例涉及智能驾驶领域，并且更具体地，涉及一种同步的方法、装置以及车辆。
背景技术
随着车辆在日常生活中被广泛使用，车辆驾驶的安全性愈发地被重视起来。目前的车辆，尤其是智能车，可以搭载众多的传感器，不同类别的传感器需要使用同一个系统来采集并处理数据。在采集和处理数据的过程中，需要统一这些传感器坐标系和时钟，目的是为了实现同一个目标在同一个时刻出现在不同类别的传感器的同一个世界坐标处。
在众多的传感器中，激光雷达和相机的时间同步尤为重要。目前，现有技术是利用两者的数据时间戳来判断二者是否发生时间同步，但是由于激光雷达是机械式旋转扫描的，而相机是瞬间曝光的，这就导致二者在时间上重合的帧数少之又少，不同传感器之间的数据融合的准确性和可靠性较差，难以做到多个传感器之间的时空同步。
发明内容
本申请实施例提供一种同步的方法、装置以及车辆，可以在复杂可编程逻辑器件CPLD或现场可编程逻辑门阵列FPGA资源充足的情况下，控制激光雷达转子的初始方位角，进而控制雷达转子的扫描角度来触发相应范围的摄像装置拍摄，实现雷达和摄像装置的时间同步和/或时空同步；在复杂可编程逻辑器件CPLD或现场可编程逻辑门阵列FPGA资源不充足的情况下，根据激光雷达和摄像装置的周期匹配，尽可能实现雷达和摄像装置的时空匹配。通过这样的方式，可以做到雷达和摄像装置之间采样频率和相位的同步，提高雷达和摄像装置数据融合的准确性和可靠性。
第一方面，提供了一种时间同步方法，该方法包括：根据复杂可编程逻辑器件CPLD或现场可编程逻辑门阵列FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；根据所述激光雷达和摄像装置的同步模式，对所述激光雷达和所述摄像装置进行同步。
其中，摄像装置可以包括单目相机、双目相机、结构光相机以及全景相机等，摄像装置获取的图像信息可以包括静态图像，也可以包括视频流信息。
根据CPLD或FPGA的资源状态确定激光雷达和摄像装置的同步模式具体为：当CPLD或FPGA资源不充足时，采用第一时间同步模式，当CPLD或FPGA资源充足时，采用第二时间同步模式。其中，第一同步模式可以指摄像装置按照原有频率被CPLD或FPGA被触发。第二同步模式可以指激光雷达内部转子转动到某个角度或者某几个角度，触发CPLD或FPGA从而触发摄像装置曝光。
应理解，CPLD或FPGA资源充足可以指CPLD或FPGA可以调动更多的资源来处理任务，即CPLD或FPGA计算能力和当前空闲能力较强，CPLD或FPGA资源不充足可以指CPLD或FPGA处理任务时能够调度的资源相对较少，即计算能力和当前空闲能力较弱。
传感器的数据融合的基本原理是将各种传感器进行多层次、多空间的信息互补和优化组合处理，最终产生对观测环境的一致性解释。在这个过程中需要充分地利用多源数据进行合理支配与使用，而信息融合的最终目标则是基于各传感器获得的分离观测信息，通过对信息多级别、多方面组合导出更多有用信息。但是，在传感器的数据融合过程中，激光雷达和相机上的数据融合相对困难。具体地，相对于相机，激光雷达是一个慢速扫描设备，而相机是瞬间曝光的，在车辆低速行驶的场景下，雷达和相机要做到时间的同步已经非常困难。而对于车辆的高速行驶场景，不但要求雷达和相机提高数据融合的准确性和可靠性，更要求雷达和相机做到时空同步。
传统时间同步方法只是机械地给雷达和相机采集的数据基于同一时间基准打上时间戳，更进一步，可以给雷达和相机采集的数据做运动补偿，由于传统的方法只是把数据交给算法应用判断时间差，没有做到雷达和相机之间采样频率和相位的同步，也就无法做到时空同步，不能提高车辆的驾驶安全。
本申请实施例中，可以根据CPLD或FPGA的资源状态确定激光雷达和摄像装置的同步模式，可以在CPLD或FPGA资源充足的情况下采用第二同步模式，控制激光雷达转子的初始方位角，进而控制雷达转子的扫描角度来触发相应范围的摄像装置拍摄，实现雷达和摄像装置的时间同步和/或时空同步；在CPLD或FPGA资源不充足的情况下采用第一同步模式，根据激光雷达和摄像装置的周期匹配，尽可能实现雷达和摄像装置的时空匹配。通过这样的方式，可以做到雷达和摄像装置之间采样频率和相位的同步，提高雷达和摄像装置数据融合的准确性和可靠性。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：将所述激光雷达和摄像装置划分至传感器组。
具体地，在划分传感器组时，可以根据激光雷达和摄像装置在车辆上的布局，将车辆上任意方位的任意几个激光雷达和摄像装置划分至同一传感器组内。进而对位于同一传感器组内的激光雷达和摄像装置进行时间同步或时空同步。
本申请实施例中，可以根据车辆上传感器的布局和实际应用的需要，对激光雷达和摄像装置划分传感器组，位于同一传感器组内的雷达和摄像装置进行时间同步或时空同步，由此提高了雷达和摄像装置之间的数据融合的效率。
结合第一方面，在第一方面的某些实现方式中，激光雷达和摄像装置的同步模式为所述第一同步模式，该方法还包括：根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率；根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，则对所述第一数据和所述第二数据进行同步处理。
其中，第一阈值可以是预先设定的数值，可以根据激光雷达和摄像装置同步方法的 实际应用情况进行确定。例如，如果对激光雷达和摄像装置同步的精确性要求较高，可以将第一阈值的数值设置的较大，如果对激光雷达和摄像装置同步的精确性要求不高，可以将第一阈值的数值设置较小。
可选地，可以根据如下公式确定所述摄像装置的曝光频率：
Figure PCTCN2021143829-appb-000001
其中，fc表示所述每一个摄像装置的曝光频率，n表示一个或者多个摄像装置的数量且n为正整数，T L表示所述激光雷达的扫描周期。
本申请实施例中，可以根据激光雷达的扫描周期和摄像装置个数确定摄像装置的曝光频率，并根据摄像装置曝光的频率触发摄像装置曝光。摄像装置曝光后，通过判断激光雷达和摄像装置采集的第一数据和第二数据的数据时间戳的差值是否小于或等于第一阈值来判断激光雷达和摄像装置是否满足时间同步或时空同步的要求，在满足时间同步或时空同步的要求的情况下，才将得到的结果输出给算法应用处理。通过这样的方式，激光雷达和摄像装置可以在时间上匹配更多的帧数据，减少算法应用的工作量，并提高了雷达和摄像装置之间的数据融合的准确性和可靠性。
结合第一方面，在第一方面的某些实现方式中，所述激光雷达的旋转周期为预设区间内的最大值。
其中，预设区间可以指雷达的扫描周期可以达到的预设范围，预设区间的数值可以根据雷达固有的属性进行确定。例如，雷达的固有的扫描周期分为两个档位，分别是50ms和100ms，则雷达的预设区间是50ms-100ms。将激光雷达的扫描周期设置为预设区间的最大值，按照上述所举的例子，是将激光雷达的扫描周期设置为100ms。
本申请实施例中，可以将激光雷达设置成适合工作场景需求的最慢的工作模式，并根据激光雷达的扫描周期和传感器组内的摄像装置个数确定摄像装置的曝光频率，以此来触发摄像装置曝光。通过此种方式，让激光雷达更好的配合了摄像装置的曝光特点，使激光雷达和摄像在时间上进一步匹配到更多的帧数据，进一步提高了雷达和摄像装置之间的数据融合的准确性和可靠性。
结合第一方面，在第一方面的某些实现方式中，所述激光雷达和摄像装置的同步模式为所述第二同步模式，所述方法还包括：在第一时刻，将激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像装置；获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；对所述第一数据和所述第二数据进行同步处理。
本申请实施例中，通过将若干个激光雷达的初始方位角的角度设置一致，以及将若干个激光雷达的初始方位角设置在摄像装置组内第一摄像装置的方位角范围内，使若干个激光雷达和第一摄像装置同步触发，保证了若干个激光雷达和第一摄像装置的时间同步或时空同步。
结合第一方面，在第一方面的某些实现方式中，该方法还包括：根据所述激光雷达、所述第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为第二时刻，所述摄像装置包括所述第二摄像装置；获取在所述第二时刻， 所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；对所述第三数据和所述第四数据进行同步处理。
具体地，可以根据摄像装置组内激光雷达的位置与第一摄像装置的位置的连线以及激光雷达的位置与第二摄像装置位置的连线的夹角，确定第二摄像装置的曝光时刻为第二时刻，并在第二时刻触发第二摄像装置曝光。
应理解，本申请实施例中，第二摄像装置可以指摄像装置组内位于第一摄像装置后进行曝光的一个摄像装置，也可以指位于第一摄像装置后曝光的多个摄像装置。
本申请实施例中，可以将若干个激光雷达初始方位角设置相同，并通过激光雷达和摄像装置组内的第一摄像装置同步触发、激光雷达与第二摄像装置根据计算的序列时刻触发的方式，实现的多个激光雷达和摄像装置之间的时间或时空同步，保证了车辆高速行驶场景下对雷达和摄像装置时空同步的高要求。其中，CPLD仅仅进行一次角度检测，即CPLD仅检测第一摄像装置与激光雷达的角度，后续以设定的曝光序列时刻为准，可以减少CPLD的资源消耗。
应理解，激光雷达与第二摄像装置根据计算的序列时刻触发可以指第二摄像装置按照计算的时间顺序，从时间开始到时间结束依次触发。
第二方面，提供了一种同步装置，该装置包括：处理单元；所述处理单元，用于根据CPLD或FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；所述处理单元，还用于根据所述同步模式，对所述激光雷达和摄像装置进行同步。
结合第二方面，在第二方面的某些实现方式中，所述处理单元，还用于将所述激光雷达和摄像装置划分至传感器组。
结合第二方面，在第二方面的某些实现方式中，所述激光雷达和摄像装置的同步模式包括第一同步模式，所述处理单元，具体用于根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率；所述处理单元还用于，根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；所述装置还包括获取单元，用于获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，所述处理单元用于对所述第一数据和所述第二数据进行同步处理。
结合第二方面，在第二方面的某些实现方式中，所述处理单元，具体用于根据如下公式确定所述摄像装置的曝光频率：
Figure PCTCN2021143829-appb-000002
其中，fc表示所述每一个摄像装置的曝光频率，n表示一个或者多个摄像装置的数量且n为正整数，T L表示所述激光雷达的扫描周期。
结合第二方面，在第二方面的某些实现方式中，所述激光雷达和摄像装置的同步模式为所述第二同步模式，
所述处理单元，具体用于在第一时刻，将激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像 装置；所述装置还包括获取单元，所述获取单元用于获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；所述处理单元，还用于对所述第一数据和所述第二数据进行同步处理。
结合第二方面，在第二方面的某些实现方式中，所述处理单元，还用于根据所述激光雷达、所述第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为第二时刻，所述摄像装置包括所述第二摄像装置；所述获取单元，还用于获取在所述第二时刻，所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；所述处理单元，还用于对所述第三数据和所述第四数据进行同步处理。
结合第二方面，在第二方面的某些实现方式中，所述处理单元，具体用于根据所述激光雷达的位置与所述第一摄像装置的位置连线以及所述激光雷达的位置与所述第二摄像装置的位置的连线的夹角，确定所述第二摄像装置的曝光时刻为第二时刻。
第三方面，提供一种同步装置，该装置包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，该装置用于执行上述各个方面中的方法。
第四方面，提供一种计算机可读介质，所述计算机可读介质存储有程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行上述各个方面中的方法。
第五方面，提供一种芯片，该芯片包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，该装置用于执行上述各个方面中的方法。
第六方面，提供一种车辆，该车辆包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，该车辆中的处理器用于执行上述各个方面中的方法。
附图说明
图1是本申请实施例提供的一种车辆的功能性示意图。
图2是本申请实施例提供的雷达与相机同步的示意图。
图3是本申请实施例提供的一种雷达与摄像装置同步的系统架构。
图4是本申请实施例提供的另一种雷达与摄像装置同步的系统架构。
图5是本申请实施例提供的传感器组的划分示意图。
图6是本申请实施例提供的雷达与摄像装置同步方法600。
图7是本申请实施例提供的第一同步模式下雷达与相机同步方法700。
图8是本申请实施例提供的第二同步模式下雷达与相机同步方法800。
图9是本申请实施例提供的雷达与相机同步装置900。
图10是本申请实施例提供的雷达与相机同步装置1000。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
为了便于理解，下文结合图1，以智能驾驶的场景为例，介绍本申请实施例适用的示 例场景。
图1是本申请实施例提供的车辆100的一个功能性示意图。可以将车辆100配置为完全或部分自动驾驶模式。例如：车辆100可以通过感知系统120获取其周围的环境信息，并基于对周边环境信息的分析得到自动驾驶策略以实现完全自动驾驶，或者将分析结果呈现给用户以实现部分自动驾驶。
车辆100可包括各种子系统，例如信息娱乐系统110、感知系统120、计算平台130和显示装置140。可选地，车辆100可包括更多或更少的子系统，并且每个子系统都可包括多个部件。另外，车辆100的每个子系统和部件可以通过有线或者无线的方式实现互连。
在一些实施例中，信息娱乐系统110可以包括通信系统111，娱乐系统112以及导航系统113。
通信系统111可以包括无线通信系统，无线通信系统可以直接地或者经由通信网络来与一个或多个设备无线通信。例如，无线通信系统可使用3G蜂窝通信，例如CDMA、EVD0、GSM/GPRS，或者4G蜂窝通信，例如LTE。或者5G蜂窝通信。无线通信系统可利用Wi-Fi与无线局域网(wireless local area network，WLAN)通信。在一些实施例中，无线通信系统可利用红外链路、蓝牙或ZigBee与设备直接通信。其他无线协议，例如各种车辆通信系统，例如，无线通信系统可包括一个或多个专用短程通信(dedicated short range communications，DSRC)设备，这些设备可包括车辆和/或路边台站之间的公共和/或私有数据通信。
娱乐系统112可以包括中控屏，麦克风和音响，用户可以基于娱乐系统在车内收听广播，播放音乐；或者将手机和车辆联通，在中控屏上实现手机的投屏，中控屏可以为触控式，用户可以通过触摸屏幕进行操作。在一些情况下，可以通过麦克风获取用户的语音信号，并依据对用户的语音信号的分析实现用户对车辆100的某些控制，例如调节车内温度等。在另一些情况下，可以通过音响向用户播放音乐。
导航系统113可以包括由地图供应商所提供的地图服务，从而为车辆100提供行驶路线的导航，导航系统113可以和车辆的全球定位系统121、惯性测量单元122配合使用。地图供应商所提供的地图服务可以为二维地图，也可以是高精地图。
感知系统120可包括感测关于车辆100周边的环境的信息的若干种传感器。例如，感知系统120可以包括定位系统121，定位系统可以是全球定位系统(global positioning system，GPS)，也可以是北斗系统或者其他定位系统、惯性测量单元(inertial measurement unit，IMU)122、激光雷达123、毫米波雷达124、超声雷达126以及摄像装置126中的一种或者多种。感知系统120还可包括被监视车辆100的内部系统的传感器(例如，车内空气质量监测器、燃油量表、机油温度表等)。来自这些传感器中的一个或多个的传感器数据可用于检测对象及其相应特性(位置、形状、方向、速度等)。这种检测和识别是车辆100的安全操作的关键功能。
定位系统121可用于估计车辆100的地理位置。
惯性测量单元122用于基于惯性加速度来感测车辆100的位置和朝向变化。在一些实施例中，惯性测量单元122可以是加速度计和陀螺仪的组合。
激光雷达123可利用激光来感测车辆100所位于的环境中的物体。在一些实施例中，激光雷达123可包括一个或多个激光源、激光扫描器以及一个或多个检测器，以及其他系统组件。
毫米波雷达124可利用无线电信号来感测车辆100的周边环境内的物体。在一些实施例中，除了感测物体以外，雷达126还可用于感测物体的速度和/或前进方向。
超声雷达125可以利用超声波信号来感测车辆100周围的物体。
摄像装置126可用于捕捉车辆100的周边环境的图像信息。摄像装置126可以包括单目相机、双目相机、结构光相机以及全景相机等，摄像装置126获取的图像信息可以包括静态图像，也可以包括视频流信息。
车辆100的部分或所有功能可以由计算平台130控制。计算平台130可包括处理器131至13n(n为正整数)，处理器是一种具有信号的处理能力的电路，在一种实现中，处理器可以是具有指令读取与运行能力的电路，例如中央处理单元(central processing unit，CPU)、微处理器、图形处理器(graphics processing unit，GPU)(可以理解为一种微处理器)、或数字信号处理器(digital signal processor，DSP)等；在另一种实现中，处理器可以通过硬件电路的逻辑关系实现一定功能，该硬件电路的逻辑关系是固定的或可以重构的，例如处理器为专用集成电路(application-specific integrated circuit，ASIC)或可编程逻辑器件(programmable logic device，PLD)实现的硬件电路，例如FPGA。在可重构的硬件电路中，处理器加载配置文档，实现硬件电路配置的过程，可以理解为处理器加载指令，以实现以上部分或全部单元的功能的过程。此外，还可以是针对人工智能设计的硬件电路，其可以理解为一种ASIC，例如神经网络处理单元(neural network processing unit，NPU)、张量处理单元(tensor processing unit，TPU)、深度学习处理单元(deep learning processing unit，DPU)等。此外，计算平台130还可以包括存储器，存储器用于存储指令，处理器131至13n中的部分或全部处理器可以调用存储器中的指令，执行质量，以实现相应的功能。
计算平台130可基于从各种子系统(例如，感知系统120)接收的输入来控制车辆100的功能。在一些实施例中，计算平台130可操作来对车辆100及其子系统的许多方面提供控制。
可选地，上述组件只是一个示例，实际应用中，上述各个模块中的组件有可能根据实际需要增添或者删除，图1不应理解为对本申请实施例的限制。
在道路行进的自动驾驶车辆，如上面的车辆100，可以识别其周围环境内的物体以确定对当前速度的调整。所述物体可以是其它车辆、交通控制设备、或者其它类型的物体。在一些示例中，可以独立地考虑每个识别的物体，并且基于物体的各自的特性，诸如它的当前速度、加速度、与车辆的间距等，可以用来确定自动驾驶车辆所要调整的速度。
可选地，车辆100或者与车辆100相关联的感知和计算设备(例如，计算平台130)可以基于所识别的物体的特性和周围环境的状态(例如，交通、雨、道路上的冰、等等)来预测所述识别的物体的行为。可选地，每一个所识别的物体都依赖于彼此的行为，因此还可以将所识别的所有物体全部一起考虑来预测单个识别的物体的行为。车辆100能够基于预测的所述识别的物体的行为来调整它的速度。换句话说，自动驾驶车辆能够基于 所预测的物体的行为来确定车辆将需要调整到(例如，加速、减速、或者停止)什么稳定状态。在这个过程中，也可以考虑其它因素来确定车辆100的速度，诸如，车辆100在行驶的道路中的横向位置、道路的曲率、静态和动态物体的接近度等等。
除了提供调整自动驾驶车辆的速度的指令之外，计算设备还可以提供修改车辆100的转向角的指令，以使得自动驾驶车辆遵循给定的轨迹和/或维持与自动驾驶车辆附近的物体(例如，道路上的相邻车道中的轿车)的安全横向和纵向距离。
上述车辆100可以为轿车、卡车、摩托车、公共车辆、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车等，本申请实施例不做特别的限定。
为了便于理解本申请实施例，下面对于本申请实施例所用术语进行介绍。
(1)复杂可编程逻辑器件(complex programmable logic device，CPLD)，CPLD适合用来实现各种运算和组合逻辑。一颗CPLD内等于包含了数颗的可编程数组逻辑，各逻辑区块间的互接连线也可以进行程序性的规划、刻录，CPLD运用这种多合一的集成作法，使其一颗就能实现数千个逻辑门，甚至数十万个逻辑门才能构成的电路。
(2)现场可编程逻辑门阵列(field programmable gate array,FPGA)，它以可编程逻辑器件为技术基础发展而成。作为特殊应用集成电路中的一种半定制电路，它既弥补全定制电路不足，又克服原有可编程逻辑控制器门电路数有限的缺点，而且其内部逻辑可以被设计者反复修改，从而改正程序中的错误。
(3)微控制单元(micro control unit，MCU)，又称处理单元、控制单元或单片微型计算机(single chip microcomputer，SCM)，是指随着大规模集成电路的出现及其发展，将计算机的CPU、RAM、ROM、定时数器和多种I/O接口集成在一片芯片上，形成芯片级的计算机，为不同的应用场合做不同组合控制。
(4)互联网服务供应商(internet service provider，ISP),又称因特网服务提供者、互联网服务提供商、网络服务供应商，即指提供互联网访问服务的公司。
(5)(gigabit multimedia serial links，GMSL)是一种高速串行接口，适用于视频、音频和控制信号的传输。
(6)局域网交换机(local area network switch，LANS):指的是用在交换式局域网内进行数据交换的设备。
(7)每秒脉冲数(pulse per second，PPS):每秒脉冲数的缩写，应用于通信行业中。
(8)运动补偿：是一种描述相邻帧差别的方法，具体来说是描述前面一帧的每个小块怎样移动到当前帧中的某个位置去。
(9)激光雷达：又叫光检测与测距(light detection and ranging，LDR)，是一种使用光源和接收器进行远程物体检测和测距的传感技术。在车辆领域，激光雷达的作用是车辆周围障碍物的探测与建模。
(10)时间同步：通过统一的主机给各个传感器提供基准时间，各传感器根据已经校准后的各自时间为各自独立采集的数据加上时间戳信息，可以做到所有传感器时间戳同步，但由于各个传感器各自采集周期相互独立，可能无法保证同一时刻采集相同的信息。
(11)时空同步：将不同传感器坐标系的测量值转换到同一个坐标系中，其中激光传 感器在高速移动的情况下需要考虑当前速度下的帧内位移校准。
(12)硬同步：使用同一种硬件同时发布触发采集命令，实现各传感器采集、测量的时间同步。做到同一时刻采集相同的信息。
图2是本申请实施例提供的雷达与相机同步的示意图，图2中的雷达与相机的同步方法可应用于图1的车辆100的行驶过程中。
传感器的数据融合的基本原理是将各种传感器进行多层次、多空间的信息互补和优化组合处理，最终产生对观测环境的一致性解释。在这个过程中需要充分地利用多源数据进行合理支配与使用，而信息融合的最终目标则是基于各传感器获得的分离观测信息，通过对信息多级别、多方面组合导出更多有用信息。这不仅是利用了多个传感器相互协同操作的优势，而且也综合处理了其它信息源的数据来提高整个传感器系统的智能化。
但是，在传感器的数据融合过程中，激光雷达和相机的数据融合相对困难。具体地，相对于相机，激光雷达是一个慢速扫描设备，而相机是瞬间曝光的，在车辆低速行驶的场景下，雷达和相机要做到时间的同步已经非常困难。而对于车辆的高速行驶场景，不但要求雷达和相机提高数据融合的准确性和可靠性，更要求雷达和相机做到时空同步。
如图2(a)所示，在车辆高速行驶的场景下，如果雷达和相机做不到时空同步，雷达扫描的点云就会偏离实际需要拍摄的物体，导致雷达扫描的结果和实际的情况存在一定的误差，进而可能危害车辆的驾驶安全。传统时间同步方法只是机械地给雷达和相机采集的数据基于同一时间基准打上时间戳，更进一步，可以给雷达和相机采集的数据做运动补偿，由于传统的方法只是把数据交给算法应用判断时间差，没有做到雷达和相机之间采样频率和相位的同步，也就无法做到时空同步，不能提高车辆的驾驶安全。
本申请实施例提供一种雷达与相机的同步方法，在CPLD资源充足的情况下，可以控制激光雷达转子的初始方位角，进而控制雷达转子的扫描角度来触发相应范围的相机拍摄，实现雷达和相机的时间同步和/或时空同步；在CPLD资源不充足的情况下，可根据激光雷达和相机的周期匹配，尽可能实现雷达和相机的时空匹配。如图2(b)所示，激光雷达和相机同步触发之后，雷达扫描的点云可以完全覆盖实际需要拍摄的物体，实现雷达和相机的时空同步。
应理解，CPLD或FPGA资源充足可以指CPLD或FPGA可以调动更多的资源来处理任务，即CPLD或FPGA计算能力和当前空闲能力较强，CPLD或FPGA资源不充足可以指CPLD或FPGA处理任务时能够调度的资源相对较少，即计算能力和当前空闲能力较弱。
示例地，在自动驾驶系统中，激光雷达通上电以后，会按照自身的工作模式，进行周期转动，如，激光雷达按照360°周期转动；相机需要通过域控制器中的CPLD/FPGA按照相机原有的工作频率发出的触发信号，进而触发相机模组的曝光。具体可以通过以下两种方式来实现：
(1)相机按照原有频率被CPLD/FPGA被触发，这样的方式，可以称为第一同步模式，也可以称为软同步模式。
(2)激光雷达内部转子转动到某个角度或者某几个角度，触发CPLD/FPGA从而触发相机曝光，这样的方式，可以称为第二同步模式，可以称为硬同步模式。
图3是本申请实施例提供的一种雷达与摄像装置同步的系统架构，图3中的雷达与摄像装置同步的系统架构可应用于图1的车辆100中。图3具体可包括如下步骤：
S301，配置传感器布局与同步架构。
具体地，可以根据车辆上的雷达和摄像装置传感器的布局，给不同的传感器划分为不同的传感器组，然后根据划分的传感器组来完成雷达与摄像装置的时间同步。在划分传感器组时，可以根据实际的需要自行选择任意方位上的任意几个传感器作为传感器组。
其中，摄像装置可以包括单目相机、双目相机、结构光相机以及全景相机等，摄像装置获取的图像信息可以包括静态图像，也可以包括视频流信息。
S302，设置多传感器的同步模式。
具体地，可以根据CPLD或FPGA的资源应用情况进行多传感器同步模式的设置，在CPLD或FPGA资源充足的情况下采用硬同步模式，在CPLD或FPGA资源不充足的情况下采用软同步模式。具体的设置方式可以是根据域控制器上的开关来设置多个传感器的时间同步模式。
其中，软同步模式可以指摄像装置按照原有频率被CPLD/FPGA被触发。硬同步模式可以指激光雷达内部转子转动到某个角度或者某几个角度，触发CPLD/FPGA从而触发摄像装置曝光。
S303，判断当前模式为硬同步模式或者软同步模式。
具体地，可以根据步骤S302中设置同步模式的结果来进行判断。如果在步骤S302中通过域控制器的开关将同步模式设置为硬同步模式，则此处判断的同步模式为硬同步模式，如果步骤S302中通过域控制器的开关将同步模式设置为软同步模式，则此处判断的同步模式为软同步模式。
S304，对齐m个激光雷达的初始方位角。
具体地，如果步骤S303判断同步的模式为硬同步模式，则在此步骤中将m个激光雷达的初始方位角对齐，其中，m为正整数。其中，将激光雷达的初始方位角对齐可以指将不同的激光雷达的转子角度设置为相同。
S305，激光雷达的转子达到摄像装置组的第一方位角范围，CPLD发出曝光信号触发摄像装置，记录时刻t 1。
具体地，当激光雷达的转子达到摄像装置组的第一摄像装置的第一方位角范围时，激光雷达向CPLD发送信号，CPLD在接收到激光雷达的信号后，向第一摄像装置发送曝光信号，以触发第一摄像装置曝光，并记录摄像装置曝光的时刻t 1，当第一摄像装置曝光时，表明第一摄像装置和激光雷达进行了同步。其中，摄像装置组的第一方位角范围可以根据摄像装置的参数或者实际应用的需要进行预先设定。
S306，CPLD根据传感器的布局与时间同步架构，设定第2个至第n个摄像装置方位角范围对应的时刻t 2至t n。
具体地，CPLD可以根据传感器的布局与时间同步架构，设置激光雷达转子到达摄像装置组内第2个至第n个摄像装置的方位角范围的时刻t 2至t n，其中n为大于2的整数。
S307，CPLD根据时刻t 2至t n，依次发出曝光信号触发摄像装置曝光。
具体地，CPLD根据步骤S306设定好时刻t 2至t n依次向摄像装置发出曝光信号，来 触发摄像装置曝光，通过这种方式，可以让激光雷达和摄像装置组内第2至第n个摄像装置进行时间同步或时空同步。
S308，判断是否退出自动驾驶模式。
具体地，如果在步骤S307中，做到了雷达和摄像装置的时间同步或时空同步，则在此步骤中判断退出自动驾驶模式，结束同步流程。如果在步骤S307中，没有做到雷达和摄像装置的时间同步或时空同步，则重新执行步骤S305，重新进行雷达和摄像装置的同步流程。
本申请实施例中，通过雷达和摄像装置同步触发的方式，使雷达和摄像装置之间采样频率和相位的同步，提高了雷达和摄像装置之间的数据融合的准确性和可靠性。
图4是本申请实施例提供的另一种雷达与摄像装置同步的系统架构。图4中的雷达与摄像装置同步的系统架构可应用于图1的车辆100中。图4具体可以包括以下步骤。
S401，配置传感器布局与同步架构。
具体地，可以根据车辆上的传感器的布局，将不同的雷达和摄像装置划分为不同的传感器组，然后根据划分的传感器组来完成雷达与摄像装置的同步。在划分传感器组时，可以根据实际的需要自行选择任意方位上的任意几个传感器作为传感器组。
其中，摄像装置可以包括单目相机、双目相机、结构光相机以及全景相机等，摄像装置获取的图像信息可以包括静态图像，也可以包括视频流信息。
S402，设置多传感器的时间同步模式。
具体地，可以根据CPLD或FPGA的资源应用情况进行多传感器同步模式的设置，在CPLD或FPGA资源充足的情况下采用硬同步模式，在CPLD或FPGA资源不充足的情况下采用软同步模式。具体的设置方式可以是根据域控制器上的开关来设置多个传感器的时间同步模式。
其中，软同步模式可以指摄像装置按照原有频率被CPLD/FPGA被触发。硬同步模式可以指激光雷达内部转子转动到某个角度或者某几个角度，触发CPLD/FPGA从而触发摄像装置曝光。
S403，判断当前模式为硬同步模式或者软同步模式。
具体地，可以根据步骤S402中设置时间同步模式的结果来进行判断。如果在步骤S402中通过域控制器的开关将同步模式设置为硬同步模式，则此处判断的同步模式为硬同步模式，如果步骤S402中通过域控制器的开关将同步模式设置为软同步模式，则此处判断的同步模式为软同步模式。
S404，将雷达的频率设置为最低档。
具体地，如果步骤S403判断时间同步的模式为软同步模式，则在此步骤中将雷达的频率设置为最低档，即雷达工作周期最慢，优选地，将雷达的扫描周期设置为100ms。
其中，雷达的工作周期最慢可以指雷达可选择的工作档位中，扫描周期最大的档位。例如，激光雷达根据出厂的设定分为两个档位，分别是100ms和50ms，选择周期最大的档位是选择100ms的档位，因此，当激光雷达的扫描周期为100ms时，可以将雷达的工作周期称作“最慢”。
S405，根据激光雷达的周期和多传感器布局，计算摄像装置的频率。
具体的，摄像装置的频率fc可以通过如下公式进行计算。
Figure PCTCN2021143829-appb-000003
其中，n表示摄像装置的数量，其数值为正整数，T L代表激光雷达的扫描周期，例如，一个激光雷达对应布局6个摄像装置，激光雷达的选择周期为100ms，则摄像装置的频率fc＝6/0.1s＝60HZ。
S406，激光雷达正常旋转工作，CPLD按照fc频倍的PPS触发摄像装置曝光。
具体地，通过步骤S406中的公式计算出摄像装置的曝光频率fc，CPLD使用fc频率倍数的PPS触发摄像装置曝光。其中，fc频倍的数值为正整数。本申请实施例所述的频倍也即是频率倍数。
S407，判断激光雷达和摄像装置的数据时间戳是否小于等于第一阈值。
具体地，获取摄像装置曝光时激光雷达和相机的数据时间戳，判断激光雷达和摄像装置的数据时间戳的差值是否小于等于第一阈值，如果满足条件，说明满足雷达和摄像装置时间同步或时空同步的要求，将雷达和摄像装置的采集的数据输出给算法应用处理。
S408，判断是否退出自动驾驶模式。
具体地，如果在步骤S407中，激光雷达和摄像装置的时间戳小于等于第一阈值，达到了雷达和摄像装置的时间同步或时空同步的要求，则在此步骤中判断退出自动驾驶模式，结束同步流程。如果在步骤S407中，没有达到雷达和摄像装置的时间同步的要求，则重新执行步骤S406，CPLD重新按照fc频倍的PPS触发摄像装置进行曝光。
本申请实施例中，可以将激光雷达设置成适合工作场景需求的最慢工作模式，摄像装置设置到相应的较快工作模式，实现了因资源限制不能实现时间硬同步时，让雷达和摄像装置在时间上匹配了更多的帧数据，提高了雷达和摄像装置之间的数据融合的准确性和可靠性。
图5是本申请实施例提供的传感器组的示意图。图5中的传感器组可应用于图3或图4的雷达与摄像装置同步的系统架构中。
作为一个示意性的例子，如图5所示，可以在车辆上部署两个雷达，和7个摄像装置，图中的①表示雷达，②表示摄像装置。划分传感器组时，可以根据实际的需要选择任意方位的任意几个传感器进行划分。例如，如图5(a)所示，传感器组可以包括雷达和摄像装置，示例地，传感器组A包括图中所有的雷达和所有的摄像装置，，这样，两个雷达和7个摄像装置进行时间同步。又例如，如图5(b)所示，可以将位于车辆前方的一个雷达和一个摄像装置划分为传感器组A，将位于车辆顶部的一个雷达和6个摄像装置划分为一个传感器组B，这样，位于A同步组和B同步组内的雷达和摄像装置可以在各自的传感器组内分别进行时间同步。本申请实施例所述的传感器组也可以称为同步组，本申请对此不做区分。
具体地，在划分不同的传感器组时，可以通过图形化配置软件对雷达和摄像装置划分通同步组，并将划分的结果发送给传感器的时间同步模块。
本申请实施例中，可以根据车辆上传感器的布局和实际应用的需要，对雷达和摄像装置划分传感器组，位于同一传感器组内的雷达和摄像装置可以进行硬同步或软同步， 可以提高雷达和摄像装置之间的数据融合的效率。
下面结合图6所示的过程对雷达和摄像装置的同步方法进行介绍。
图6是本申请实施例提供的雷达与摄像装置的同步方法600。该同步方法600可应用于图1的车辆100中。方法600可以包括以下步骤。
S601，确定激光雷达和摄像装置的时间同步模式。
确定激光雷达和摄像装置的同步模式可以根据CPLD或FPGA的资源状态进行确定。当CPLD或FPGA资源不充足时采用第一时间同步模式，当CPLD或FPGA资源充足时采用第二时间同步模式。其中，第一同步模式可以指摄像装置按照原有频率被CPLD或FPGA被触发。第二同步模式可以指激光雷达内部转子转动到某个角度或者某几个角度，触发CPLD或FPGA从而触发摄像装置曝光。
应理解，CPLD或FPGA资源充足可以指CPLD或FPGA可以调动更多的资源来处理任务，即CPLD或FPGA计算能力和当前空闲能力较强，CPLD或FPGA资源不充足可以指CPLD或FPGA处理任务时能够调度的资源相对较少，即计算能力和当前空闲能力较弱。
一种可能的实现方式中，也可以不选择雷达和摄像装置的同步模式，直接按照第一同步模式或第二同步模式对激光雷达和摄像装置进行同步。
一种可能的实现方式中，在进行步骤S601之前，该方法还包括：将激光雷达和相机摄像装置划分至传感器组。
具体地，在划分传感器组时，可以根据激光雷达和摄像装置在车辆上的布局，将车辆上任意方位的任意几个激光雷达和摄像装置划分至同一传感器组内。进而对位于同一传感器组内的激光雷达和相机进行时间同步或时空同步。例如，可以按照图5所述的方式划分不同的传感器组。
本申请实施例中，可以根据车辆上传感器的布局和实际应用的需要，对激光雷达和摄像装置划分传感器组，位于同一传感器组内的雷达和摄像装置进行时间同步或时空同步，由此提高了雷达和摄像装置之间的数据融合的效率。
S602，根据激光雷达和摄像装置的同步模式，对激光雷达和摄像装置进行同步。
一种可能的实现方式中，在步骤S601确定激光雷达和摄像装置的同步模式为第一同步模式后，在步骤S602中，对激光雷达和摄像装置进行同步包括：根据激光雷达的扫描周期和摄像装置的数量，确定摄像装置的曝光频率；根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，则对所述第一数据和所述第二数据进行同步处理。
具体地，摄像装置的频率fc可以通过如下公式进行计算：
Figure PCTCN2021143829-appb-000004
其中，n表示摄像装置的数量，其数值为正整数，T L代表激光雷达的扫描周期，例如，同一资源组内一个激光雷达对应布局6个摄像装置，摄像装置在预设区间内扫描周 期的最小值为100ms，则摄像装置的频率fc＝6/0.1s＝60HZ。
在确定了摄像装置的曝光频率后，CPLD根据摄像装置的曝光频率，触发摄像装置曝光，并获取摄像装置曝光时刻激光雷达和摄像装置分别采集的第一数据和第二数据。然后，CPLD将获取的第一数据和第二数据的数据时间戳进行差值后和第一阈值进行比较，进而对激光雷达和摄像装置采集的数据进行不同的处理。
其中，第一阈值可以是预先设定的数值，可以根据时间同步的方法实际应用的情况进行确定。例如，如果对激光雷达和摄像装置的同步的精确性要求较高，可以将第一阈值的数值设置的较大，如果对激光雷达和摄像装置同步的精确性要求不高，可以将第一阈值的数值设置较小。如果第一数据和第二数据的数据时间戳的差值小于或等于第一阈值，则说明激光雷达和摄像装置满足同步的要求，可以将第一数据和第二数据输出给算法应用进行处理。如果第一数据和第二数据的数据时间戳的差值大于第一阈值，则说明激光雷达和摄像装置没有达到同步的要求，需要将第一数据和第二数据舍弃。
一种可能的实现方式中，所述激光雷达的旋转周期为预设区间内的最大值。
其中，预设区间可以指雷达的扫描周期可以达到的预设范围，预设区间的数值可以根据雷达固有的属性进行确定。例如，雷达的固有的扫描周期分为两个档位，分别是50ms和100ms，则雷达的预设区间是50ms-100ms。将激光雷达的扫描周期设置为预设区间的最大值，按照上述所举的例子，是将激光雷达的扫描周期设置为100ms。
本申请实施例中，可以将激光雷达设置成适合工作场景需求的最慢的工作模式，根据激光雷达的扫描周期和激光雷达对应的摄像装置个数确定摄像装置的曝光频率，并根据摄像装置曝光的频率触发摄像装置曝光。摄像装置曝光后，通过判断激光雷达和摄像装置的数据时间戳的差值是否小于或等于第一阈值来判断激光雷达和摄像装置是否满足时间同步的要求，在满足时间同步的要求的情况下，才将得到的结果输出给算法应用处理。通过此种方式，可以让激光雷达和摄像装置在时间上匹配更多的帧数据，减少算法应用的工作量，并提高了雷达和摄像装置之间的数据融合的准确性和可靠性。
一种可能的实现方式中，在步骤S601确定激光雷达和摄像装置的同步模式为第二同步模式，在步骤S602中，对激光雷达和摄像装置进行同步包括：在第一时刻，将激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像装置；获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；对所述第一数据和所述第二数据进行同步处理。
具体地，可以设置m个激光雷达的初始方位角的角度相等，其中，m为大于0的整数。例如，可以将m个激光雷达的初始方位角都设置成零度，m个初始方位角相同的激光雷达的相位角位于摄像装置组内第一摄像装置的方位角范围。此时，m个激光雷达和摄像装置组内的第一摄像装置同步触发，记录激光雷达和摄像装置同步触发时刻为第一时刻，并对激光雷达和第一摄像装置在第一时刻分别采集的第一数据和第二数据进行同步处理。
本申请实施例中，通过设置若干个激光雷达的初始方位角的角度相等，以及将若干个激光雷达的初始方位角设置在摄像装置组内第一摄像装置的方位角范围内，使若干个 激光雷达和第一摄像装置同步触发，保证了若干个激光雷达和第一摄像装置的时间同步或时空同步。
一种可能的实现方式中，步骤S602还可以包括：根据所述激光雷达、第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为第二时刻，所述摄像装置包括所述第二摄像装置；获取在所述第二时刻，所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；对所述第三数据和所述第四数据进行同步处理。
具体地，可以根据摄像装置组内激光雷达的位置与第一摄像装置的位置的连线以及激光雷达的位置与第二摄像装置位置的连线的夹角，确定第二摄像装置的曝光时刻为第二时刻，并在第二时刻触发摄像装置曝光。
应理解，本申请实施例中，第二摄像装置可以指摄像装置组内位于第一摄像装置后进行曝光的一个摄像装置，也可以指位于第一摄像装置后曝光的多个摄像装置。
其中，第二相机曝光的第二时刻，具体可以通过以下公式进行计算：
Figure PCTCN2021143829-appb-000005
其中，T L代表激光雷达的扫描周期，θn代表第n摄像装置与雷达初始方位之间的夹角。例如，雷达的扫描周期为100ms，第二摄像装置的与雷达之间的夹角为π/3，则第二摄像装置的触发时刻t 2＝100/2π×π/3＝16.7ms。当时间达到16.7ms时，CPLD通过ISP触发第二摄像装置曝光。相机曝光时，激光雷达采集的第三数据，第二摄像装置采集的第四数据；CPLD对第三数据和第四数据进行同步处理。
本申请实施例中，可以将若干个激光雷达初始方位角设置相同，并通过激光雷达和摄像装置组内的第一摄像装置同步触发、激光雷达与第二摄像装置根据计算的序列时刻触发的方式，实现的多个雷达和摄像装置之间的同步，保证了车辆高速行驶场景下对雷达和摄像装置时空同步的高要求。其中，CPLD仅仅进行一次角度检测，即CPLD仅检测第一摄像装置与激光雷达的角度，后续以设定的曝光序列时刻为准，可以减少CPLD的资源消耗。
下面结合图7和图8所示的过程对雷达和相机的同步方法进行介绍。
图7是本申请实施例提供的第一同步模式下雷达与相机同步方法700。图7中的第一同步模式下雷达与相机时间的同步方法700可应用于图1的车辆100中。方法700可以包括如下步骤。
S701，设置激光雷达的频率为最低档，即工作周期最慢。
例如，雷达的固有的扫描周期分为两个档位，分别是50ms和100ms，将激光雷达的频率设置为最低档，是将激光雷达的扫描周期设置为100ms。其中，100ms为本申请实施例雷达扫描周期的优选的设置。
S702，根据激光雷达扫描周期和多传感器布局计算相机频率。
具体地，相机的频率fc可以通过如下公式进行计算：
Figure PCTCN2021143829-appb-000006
其中，n表示相机的数量，其数值为正整数，T L代表激光雷达的扫描周期，例如，同一资源组内一个激光雷达对应布局6个相机，相机的扫描周期为100ms，则相机的频率fc＝6/0.1s＝60HZ。
S703，雷达正常旋转工作,CPLD按照fc倍频PPS触发相机曝光。
具体地，在步骤S702计算出相机的频率fc后，CPLD使用fc频率倍数的PPS触发相机曝光。其中，fc频倍的数值为正整数。
S704，激光雷达与相机数据时间戳小于或等于第一阈值，即满足时间同步的要求，输出给算法处理。
具体地，判断激光雷达和相机的时间戳是否小于等于第一阈值，如果满足条件，说明满足雷达和相机时间同步的要求，将结果输出给算法应用处理。在输出给算法应用处理之前，可以对得到的数据进行校正，以提高数据的精准度。
本申请实施例中，可以将激光雷达设置成适合工作场景需求的最慢工作模式，相机设置到相应的较快工作模式，实现了因资源限制不能实现时间硬同步时，让雷达和相机在时间上匹配了更多的帧数据，提高了雷达和相机之间的数据融合的准确性和可靠性。
图8是本申请实施例提供的第二同步模式下雷达与相机同步方法800。第二同步模式下雷达与相机同步方法800可应用于图1的车辆100中。方法800可包括如下步骤。
S801，m个激光雷达初始方位角设置为0度。
具体地，如图8(a)所示，将m个激光雷达的初始方位角设置为0度，保证m个雷达可以被同步触发，并且m个激光雷达的初始方位角位于相机组第一相机方位角的范围。其中，m为正整数。
S802，CPLD通过ISP触发相机曝光，时间t 1＝0。
具体地，如图8(a)所示，将m个激光雷达的初始方位角位于相机组第一相机方位角的范围的信息反馈给CPLD，CPLD通过ISP触发相机曝光，记录此时的相机触发时刻t 1＝0。
S803，CPLD根据传感器布局与时间同步架构，设定第2个至第n个相机方位角范围对应的时刻t 2至t n。
如图8(b)和图8(c)所示，在车辆上不同位置部署雷达和相机后，可以测量雷达和相机组内每一个相机之间的夹角，确定好夹角后，可以计算出相机组内每个相机的曝光时刻t 2至t n，具体可以通过以下公式进行计算：
Figure PCTCN2021143829-appb-000007
其中，T L代表激光雷达的扫描周期，θn代表第n相机与雷达初始方位之间的夹角。例如，雷达的扫描周期为100ms，第二相机的与雷达之间的夹角为π/3，则第二相机的触发时刻t 2＝100/2π×π/3＝16.7ms。当时间达到16.7ms时，CPLD通过ISP触发第二相机曝光。同样的，可以计算出第三相机至第n相机的触发时刻t 3至t n。
S804，CPLD根据t 2至t n时刻依次通过ISP触发相机曝光。
具体地，CPLD根据设定的t 2至t n时刻依次通过ISP触发第2至第n相机曝光。
本申请实施例中，可以将若干个激光雷达初始方位角设置为0度，并通过激光雷达 和相机组内的第一相机同步触发、雷达与第二相机至第n相机根据计算的序列时刻依次触发的方式，实现的多个雷达和相机之间的同步，保证了车辆高速行驶场景下对雷达和相机时空同步的高要求。其中，CPLD仅仅进行一次角度检测，即CPLD仅检测第一相机与激光雷达的角度，后续以设定的序列为准，可以减少CPLD的资源消耗。
图9是本申请实施例提供的雷达与相机同步装置900示意图。该装置900可应用于图1的车辆100中。
该装置900可以包括获取单元910、存储单元920和处理单元930。获取单元910可以实现相应的通信功能，获取单元910还可以称为通信接口或通信单元用于获取数据。存储单元920可以用于存储相应的指令和/或数据，处理单元930用于进行数据处理。处理单元930可以读取存储单元中的指令和/或数据，以使得装置实现前述方法实施例。
该雷达与相机同步装置包括：处理单元930；所述处理单元930，用于根据CPLD或FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；所述处理单元930，还用于根据所述激光雷达和所述摄像装置的同步模式，对所述激光雷达和摄像装置进行同步。
一种可能的实现方式中，所述处理单元930，还用于将所述激光雷达和相机摄像装置划分至传感器组。
一种可能的实现方式中，所述激光雷达和摄像装置的同步模式为所述第一同步模式，所述处理单元930，具体用于根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率，所述摄像装置包括所述第一摄像装置；所述处理单元930还用于，根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；所述装置还包括获取单元910，用于获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，所述处理单元930用于对所述第一数据和所述第二数据进行同步处理。
一种可能的实现方式中，所述处理单元930，具体用于根据如下公式确定所述摄像装置的曝光频率：
Figure PCTCN2021143829-appb-000008
其中，fc表示所述摄像装置的曝光频率，n表示一个或者多个摄像装置的数量且n为正整数，T L表示所述激光雷达的扫描周期。
一种可能的实现方式中，所述激光雷达和摄像装置的同步模式为所述第二同步模式，所述处理单元930，具体用于在第一时刻，将激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像装置；所述装置还包括获取单元910，所述获取单元910用于获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；所述处理单元930，还用于对所述第一数据和所述第二数据进行同步处理。
一种可能的实现方式中，所述处理单元930，还用于根据所述激光雷达、所述第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为 第二时刻；所述获取单元910，还用于获取在所述第二时刻，所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；所述处理单元930，还用于对所述第三数据和所述第四数据进行同步处理。
一种可能的实现方式中，所述处理单元930，具体用于根据所述激光雷达的位置与所述第一摄像装置的位置连线以及所述激光雷达的位置与所述第二摄像装置的位置的连线的夹角，确定所述第二摄像装置的曝光时刻为第二时刻。
图10是本申请实施例提供的一种雷达与相机同步装置1000示意图。该装置1000可应用于图1的车辆100中。
该雷达与时间同步装置包括：存储器1010、处理器1020、以及通信接口1030。其中，存储器1010、处理器1020，通信接口1030通过内部连接通路相连，该存储器1010用于存储指令，该处理器1020用于执行该存储器1020存储的指令，以控制输入/输出接口1030接收/发送第二信道模型的至少部分参数。可选地，存储器1010既可以和处理器1020通过接口耦合，也可以和处理器1020集成在一起。
需要说明的是，上述通信接口1030使用例如但不限于收发器一类的收发装置，来实现通信设备1000与其他设备或通信网络之间的通信。上述通信接口1030还可以包括输入/输出接口(input/output interface)。
在实现过程中，上述方法的各步骤可以通过处理器1020中的硬件的集成逻辑电路或者软件形式的指令完成。结合本申请实施例所公开的方法可以直接体现为硬件处理器执行完成，或者用处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1010，处理器1020读取存储器1010中的信息，结合其硬件完成上述方法的步骤。为避免重复，这里不再详细描述。
本申请实施例还提供一种计算机可读介质，所述计算机可读介质存储有程序代码，当所述计算机程序代码在计算机上运行时，使得所述计算机执行上述图6至图8中的任一种方法。
本申请实施例还提供一种芯片，包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，以执行上述图6至图8中的任一种方法。
本申请实施例还提供一种车辆，包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，以执行上述图6至图8中的任一种方法。
本申请实施例还提供一种车辆，包括图9或图10任一种雷达与摄像装置时间装置。
应理解，本申请实施例中，上述处理器可以为中央控制单元(central processing unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
还应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
在本说明书中使用的术语“部件”、“模块”等用于表示计算机相关的实体、硬件、固件、硬件和软件的组合、软件、或执行中的软件。例如，部件可以是但不限于，在处理器上运行的进程、处理器、对象、可执行文件、执行线程、程序和/或计算机。通过图示，在计算设备上运行的应用和计算设备都可以是部件。一个或多个部件可驻留在进程和/或执行线程中，部件可位于一个计算机上和/或分布在2个或更多个计算机之间。此外，这些部件可从在上面存储有各种数据结构的各种计算机可读介质执行。部件可例如根据具有一个或多个数据分组(例如来自与本地系统、分布式系统和/或网络间的另一部件交互的二个部件的数据，例如通过信号与其它系统交互的互联网)的信号通过本地和/或远程进程来通信。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存 储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (20)

  1. 一种同步方法，其特征在于，所述方法包括：
    根据复杂可编程逻辑器件CPLD或现场可编程逻辑门阵列FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；
    根据所述同步模式，对所述激光雷达和所述摄像装置进行同步。
  2. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    将所述激光雷达和所述摄像装置划分至传感器组。
  3. 如权利要求1或2所述的方法，其特征在于，所述激光雷达和摄像装置的同步模式为所述第一同步模式，所述方法还包括：
    根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率；
    根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；
    获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；
    若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，则对所述第一数据和所述第二数据进行同步处理。
  4. 如权利要求3所述的方法，其特征在于，所述激光雷达的扫描周期为预设区间内的最大值。
  5. 如权利要求3或4所述的方法，其特征在于，
    所述根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率，包括：根据如下公式确定所述摄像装置的曝光频率：
    Figure PCTCN2021143829-appb-100001
    其中，fc表示所述摄像装置的曝光频率，n表示一个或者多个摄像装置的数量且n为正整数，T L表示所述激光雷达的扫描周期。
  6. 如权利要求1或2所述的方法，其特征在于，所述激光雷达和摄像装置的同步模式为所述第二同步模式，所述方法还包括：
    在第一时刻，将所述激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像装置；
    获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；
    对所述第一数据和所述第二数据进行同步处理。
  7. 如权利要求6所述的方法，其特征在于，所述方法还包括：
    根据所述激光雷达、所述第一摄像装置和第二摄像装置的位置关系，确定所述第二 摄像装置的曝光时刻为第二时刻，所述摄像装置包括所述第二摄像装置；
    获取在所述第二时刻，所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；
    对所述第三数据和所述第四数据同步进行处理。
  8. 如权利要求7所述的方法，其特征在于，所述根据所述激光雷达、所述第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为第二时刻包括：
    根据所述激光雷达的位置与所述第一摄像装置的位置的连线以及所述激光雷达的位置与所述第二摄像装置的位置的连线的夹角，确定所述第二摄像装置的曝光时刻为第二时刻。
  9. 一种同步装置，其特征在于，所述装置包括：处理单元；
    所述处理单元，用于根据复杂可编程逻辑器件CPLD或现场可编程逻辑门阵FPGA的资源状态确定激光雷达和摄像装置的同步模式，所述激光雷达和摄像装置的同步模式包括第一同步模式或第二同步模式；
    所述处理单元，还用于根据所述同步模式，对所述激光雷达和所述摄像装置进行同步。
  10. 如权利要求9所述的装置，其特征在于，所述处理单元，还用于将所述激光雷达和所述摄像装置划分至传感器组。
  11. 如权利要求9或10所述的装置，其特征在于，所述激光雷达和摄像装置的同步模式为所述第一同步模式，
    所述处理单元，具体用于根据所述激光雷达的扫描周期和所述摄像装置的数量，确定所述摄像装置的曝光频率；
    所述处理单元还用于，根据第一摄像装置的曝光频率，触发所述第一摄像装置曝光，得到第一数据，所述摄像装置包括所述第一摄像装置；
    所述装置还包括获取单元，用于获取所述第一摄像装置曝光时，所述激光雷达采集的第二数据；
    若所述第一数据的时间戳和所述第二数据的时间戳的差值小于或等于第一阈值，所述处理单元用于对所述第一数据和所述第二数据进行同步处理。
  12. 如权利要求11所述的装置，其特征在于，激光雷达的扫描周期为预设区间内的最大值。
  13. 如权利要求11或12所述的装置，其特征在于，所述处理单元，具体用于根据如下公式确定所述摄像装置的曝光频率：
    Figure PCTCN2021143829-appb-100002
    其中，fc表示所述摄像装置的曝光频率，n表示一个或者多个摄像装置的数量且n为正整数，T L表示所述激光雷达的扫描周期。
  14. 如权利要求9或10所述的装置，其特征在于，所述激光雷达和摄像装置的同步模式为所述第二同步模式，
    所述处理单元，具体用于在第一时刻，将激光雷达的初始方位角设置为第一方位角，在所述第一方位角的方向上设置有第一摄像装置，所述摄像装置包括所述第一摄像装置；
    所述装置还包括获取单元，所述获取单元用于获取在所述第一时刻，所述激光雷达采集的第一数据和所述第一摄像装置采集的第二数据；
    所述处理单元，还用于对所述第一数据和所述第二数据进行同步处理。
  15. 如权利要求14所述的装置，其特征在于，
    所述处理单元，还用于根据所述激光雷达、所述第一摄像装置的位置关系和第二摄像装置的位置关系，确定所述第二摄像装置的曝光时刻为第二时刻，所述摄像装置包括所述第二摄像装置；
    所述获取单元，还用于获取在所述第二时刻，所述激光雷达采集的第三数据和所述第二摄像装置采集的第四数据；
    所述处理单元，还用于对所述第三数据和所述第四数据进行同步处理。
  16. 如权利要求15所述的装置，其特征在于，所述处理单元，具体用于根据所述激光雷达的位置与所述第一摄像装置的位置连线以及所述激光雷达的位置与所述第二摄像装置的位置的连线的夹角，确定所述第二摄像装置的曝光时刻为第二时刻。
  17. 一种同步装置，其特征在于，包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，以执行如权利要求1至8中任一项所述的方法。
  18. 一种计算机可读介质，其特征在于，所述计算机可读介质存储有程序代码，当所述计算机程序代码在计算机上运行时，使得所述计算机执行如权利要求1至8中任一项所述的方法。
  19. 一种芯片，其特征在于，包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，以执行如权利要求1至8中任一项所述的方法。
  20. 一种车辆，其特征在于，包括：至少一个处理器和存储器，所述至少一个处理器与所述存储器耦合，用于读取并执行所述存储器中的指令，以执行如权利要求1至8中任一项所述的方法。
PCT/CN2021/143829 2021-12-31 2021-12-31 同步的方法、装置以及车辆 WO2023123416A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2021/143829 WO2023123416A1 (zh) 2021-12-31 2021-12-31 同步的方法、装置以及车辆
CN202180029658.3A CN116685871A (zh) 2021-12-31 2021-12-31 同步的方法、装置以及车辆

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/143829 WO2023123416A1 (zh) 2021-12-31 2021-12-31 同步的方法、装置以及车辆

Publications (1)

Publication Number Publication Date
WO2023123416A1 true WO2023123416A1 (zh) 2023-07-06

Family

ID=86997118

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/143829 WO2023123416A1 (zh) 2021-12-31 2021-12-31 同步的方法、装置以及车辆

Country Status (2)

Country Link
CN (1) CN116685871A (zh)
WO (1) WO2023123416A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2619120A (en) * 2022-05-27 2023-11-29 Motional Ad Llc Image based lidar-camera synchronization

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111435162A (zh) * 2020-03-03 2020-07-21 深圳市镭神智能系统有限公司 激光雷达与相机同步方法、装置、设备和存储介质
US20210025997A1 (en) * 2018-04-09 2021-01-28 Innoviz Technologies Ltd. Lidar systems and methods with internal light calibration
CN112485806A (zh) * 2020-09-27 2021-03-12 浙江众合科技股份有限公司 一种激光雷达和相机时间同步系统及方法
CN113138393A (zh) * 2020-01-17 2021-07-20 阿里巴巴集团控股有限公司 环境感测系统、控制装置以及环境感测数据融合装置
CN113219479A (zh) * 2021-05-13 2021-08-06 环宇智行科技(苏州)有限公司 智能驾驶控制系统的摄像头和激光雷达同步方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210025997A1 (en) * 2018-04-09 2021-01-28 Innoviz Technologies Ltd. Lidar systems and methods with internal light calibration
CN113138393A (zh) * 2020-01-17 2021-07-20 阿里巴巴集团控股有限公司 环境感测系统、控制装置以及环境感测数据融合装置
CN111435162A (zh) * 2020-03-03 2020-07-21 深圳市镭神智能系统有限公司 激光雷达与相机同步方法、装置、设备和存储介质
CN112485806A (zh) * 2020-09-27 2021-03-12 浙江众合科技股份有限公司 一种激光雷达和相机时间同步系统及方法
CN113219479A (zh) * 2021-05-13 2021-08-06 环宇智行科技(苏州)有限公司 智能驾驶控制系统的摄像头和激光雷达同步方法及系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2619120A (en) * 2022-05-27 2023-11-29 Motional Ad Llc Image based lidar-camera synchronization

Also Published As

Publication number Publication date
CN116685871A (zh) 2023-09-01

Similar Documents

Publication Publication Date Title
CN110895147B (zh) 自动驾驶车辆的用相机捕获图像数据的图像数据获取逻辑
EP3614688B1 (en) Trigger logic to trigger sensors of an autonomous driving vehicle for capturing data
US11029165B2 (en) Sensor integration and synchronization unit for an autonomous driving vehicle
WO2021047271A1 (zh) 一种时间同步方法及装置
JP2018060512A (ja) 画像生成システム、プログラム及び方法並びにシミュレーションシステム、プログラム及び方法
EP3891474B1 (en) Synchronizing sensors of autonomous driving vehicles
CN112543876B (zh) 用于自动驾驶车辆中的传感器同步性数据分析的系统
WO2022011638A1 (zh) 一种数据传输方法以及装置
CN113016153A (zh) 验证在自动驾驶车辆中使用的传感器的定时
WO2018066352A1 (ja) 画像生成システム、プログラム及び方法並びにシミュレーションシステム、プログラム及び方法
US11245909B2 (en) Timestamp and metadata processing for video compression in autonomous driving vehicles
CN112640332B (zh) 一种通信方法及装置
WO2023123416A1 (zh) 同步的方法、装置以及车辆
US11338823B2 (en) Multiple sensor data storage with compressed video stream in autonomous driving vehicles
Niaz et al. Smart city IoT application for road infrastructure safety and monitoring by using digital twin
US20230341554A1 (en) Methods and apparatus with hardware logic for pre-processing lidar data
CN115348657B (zh) 用于车辆时间同步的系统、方法及车辆
CN115556769A (zh) 障碍物状态量确定方法及装置、电子设备和介质
CN114415489A (zh) 一种车载传感器时间同步方法、装置、设备和介质
CN114609923A (zh) 仿真测试方法、装置及系统
CN116311943B (zh) 交叉路口的平均延误时间的估算方法及装置
US20240104905A1 (en) System and method for building multi-view machine learning datasets from fleet data
WO2024055252A1 (zh) 一种数据融合方法、装置及智能驾驶设备
WO2023010236A1 (zh) 一种显示方法、装置和系统
CN115937823A (zh) 用于检测障碍物的方法、装置、电子设备和介质

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 202180029658.3

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21969762

Country of ref document: EP

Kind code of ref document: A1