WO2024055252A1 - 一种数据融合方法、装置及智能驾驶设备 - Google Patents

一种数据融合方法、装置及智能驾驶设备 Download PDF

Info

Publication number
WO2024055252A1
WO2024055252A1 PCT/CN2022/119125 CN2022119125W WO2024055252A1 WO 2024055252 A1 WO2024055252 A1 WO 2024055252A1 CN 2022119125 W CN2022119125 W CN 2022119125W WO 2024055252 A1 WO2024055252 A1 WO 2024055252A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
group
data
laser beam
intelligent driving
Prior art date
Application number
PCT/CN2022/119125
Other languages
English (en)
French (fr)
Inventor
郭剑艇
鞠增伟
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2022/119125 priority Critical patent/WO2024055252A1/zh
Publication of WO2024055252A1 publication Critical patent/WO2024055252A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W28/00—Network traffic management; Network resource management
    • H04W28/02—Traffic management, e.g. flow control or congestion control
    • H04W28/06—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information

Definitions

  • This application relates to intelligent driving technology, which is applied to the field of fusion perception of intelligent driving, and in particular, to a data fusion method, device and intelligent driving equipment.
  • ADAS Advanced driver assistance system
  • sensors play a very important role in assisted driving and automatic driving of smart cars and are likened to the "eyes" of the car.
  • Sensors include visual sensors such as vehicle-mounted cameras and radar-based sensors such as vehicle-mounted lidar and vehicle-mounted ultrasonic radar.
  • the lidar has a fixed scanning period, and the camera is triggered at a fixed frequency.
  • fusion often occurs due to the mismatch between lidar and camera timestamps. The effect is poor, making it impossible to accurately identify problems in the vehicle surrounding environment.
  • This application discloses a data fusion method, device and intelligent driving equipment, which can improve the fusion effect of point cloud data scanned by a laser emitting device and picture data captured by a camera, thereby making the vehicle surrounding environment recognition more accurate and more conducive to Planning and execution of autonomous driving missions.
  • this application provides a data fusion method, which method includes: acquiring point cloud data, the point cloud data being generated in the first scanning direction and the second laser beam group according to the first laser beam group and the second laser beam group respectively. It is obtained by detecting the scanning direction, wherein the first scanning direction and the second scanning direction are different; the camera exposures in the M camera groups are sequentially triggered according to the triggering delays of the M camera groups to obtain M groups of pictures.
  • the trigger delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1;
  • the picture data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain the data corresponding to the camera group for perception processing.
  • the above M camera groups can be determined based on the installation positions of multiple cameras on the intelligent driving device. Taking the driving direction of the smart driving device as the benchmark, this application can fuse the point cloud data with the picture data of the camera group corresponding to the front field of view on the smart driving device to improve the point cloud data and picture data of the front field of view. Similarly, it can also improve the fusion effect of point cloud data and image data in the middle field of view and rear field of view, thereby making the vehicle surrounding environment recognition more accurate and more conducive to the planning and implementation of autonomous driving tasks.
  • the obtaining point cloud data includes: obtaining first data, which is detected by the first laser beam group in the first scanning direction; obtaining second data , the second data is detected by the second laser beam group in the second scanning direction; the first data and the second data are spliced to obtain the point cloud data.
  • the first data and the second data are spliced, fused and enhanced.
  • the point cloud data in this application has a higher point density and is more accurate in identifying the vehicle surrounding environment. high.
  • the first laser beam group and the second laser beam group come from the same laser emitting device.
  • the first laser beam group comes from a first laser emitting device
  • the second laser beam group comes from a second laser emitting device
  • two laser beam groups can be used for coverage scanning, thereby making the point density of the point cloud data higher.
  • the method further includes: determining the position of each of the M camera groups; and determining the position of each of the M camera groups according to the scanning period and the position of each of the M camera groups. Determine the trigger delay for each camera group.
  • the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is the first delay, and the trigger delay of the second camera group is The trigger delay is the second delay; the triggering delay of the M camera groups sequentially triggers the camera exposures in the M camera groups to obtain M groups of picture data, including: starting the first timer to set the initial time ; When the indicated time of the first timer is separated by the first time delay from the initial time, trigger the camera exposure in the first camera group to obtain the picture data corresponding to the first camera group; when The indicated time of the first timer is separated from the initial time by the second delay time, triggering camera exposure in the second camera group to obtain picture data corresponding to the second camera group.
  • the M camera groups include a front group, a side group, and a rear group;
  • the front group includes a first camera and a second camera, and the first camera and all The second camera is relatively disposed on the front of the intelligent driving device;
  • the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively disposed on both sides of the intelligent driving device.
  • the rear grouping includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.
  • the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.
  • image data corresponding to at least two camera groups among the M camera groups are fused to obtain data for perception processing.
  • this application provides an intelligent driving device, including: a computing platform and M camera groups, the computing platform being used to: obtain point cloud data, the point cloud data is obtained according to the first laser beam group and the The two laser beam groups are detected in the first scanning direction and the second scanning direction respectively, wherein the first scanning direction and the second scanning direction are different; and are triggered sequentially according to the triggering delay of the M camera groups.
  • the cameras in the M camera groups are exposed to obtain M groups of picture data; where, the cameras in the same camera group are exposed at the same time, and the triggering delay is related to the first laser beam group and the second laser beam group. is related to the scanning period, M is a positive integer greater than or equal to 1; the image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain the corresponding image data of the camera group for perception processing The data.
  • the computing platform when acquiring the point cloud data, is configured to: acquire first data, the first data being the position of the first laser beam group in the first scanning direction. detected; obtaining second data, which is detected by the second laser beam group in the second scanning direction; splicing the first data and the second data to obtain the point Cloud data.
  • the first laser beam group and the second laser beam group come from the same laser emitting device.
  • the intelligent driving equipment further includes: a first laser emitting device and a second laser emitting device; the first laser beam group comes from the first laser emitting device, and the second laser emitting device The laser beam group comes from the second laser emitting device.
  • the first laser emitting device and the second laser emitting device are lidar devices.
  • the computing platform is further configured to: determine the position of each of the M camera groups; The position determines the trigger delay of each camera group.
  • the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is the first delay, and the trigger delay of the second camera group is The trigger delay is the second delay, and the computing platform is also used to: start a first timer to set an initial time; and space the first delay between the indicated time of the first timer and the initial time.
  • the camera exposure in the first camera group is triggered to obtain the picture data corresponding to the first camera group.
  • the indicated time of the first timer is separated by the second time delay from the initial time
  • camera exposure in the second camera group is triggered to obtain picture data corresponding to the second camera group.
  • the M camera groups include a front group, a side group and a rear group; the front group includes a first camera and a second camera, and the first camera and all The second camera is relatively disposed at the front of the intelligent driving device; the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively disposed at the front of the intelligent driving device. Both sides; the rear grouping includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are arranged oppositely at the rear of the intelligent driving device.
  • the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.
  • this application provides a data fusion device.
  • the data fusion device is a vehicle-mounted computing platform or a cloud server or a chip system.
  • the data fusion device includes a processor and a memory.
  • the memory and the processor pass The circuits are interconnected, and the processor is used to obtain the computer program stored in the memory.
  • the computer program is called by the processor, it is used to execute the data fusion method in the above first aspect and each implementation manner.
  • the present application provides a computer-readable storage medium.
  • Computer instructions are stored in the computer-readable storage medium.
  • the first aspect or the first aspect is implemented when the computer instructions are run on a processor. The method described in any possible implementation manner of the aspect.
  • the present application provides a computer program product.
  • the computer program product includes computer program code.
  • the computer program code When the computer program code is run on a computer, it can implement the above first aspect or any one of the first aspects. Possible implementations of the methods described.
  • the present application provides a chip that includes a circuit, and the circuit is used to implement the method described in the above first aspect or any possible implementation manner of the first aspect.
  • Figure 1 is a schematic structural diagram of an intelligent driving device provided by an embodiment of the present application.
  • Figure 2 is a schematic diagram of a camera grouping provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of yet another camera grouping provided by an embodiment of the present application.
  • Figure 4 is a schematic structural diagram of another intelligent driving device provided by an embodiment of the present application.
  • Figure 5 is a schematic diagram of a driving control device of an intelligent driving device provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of the position distribution of cameras in an intelligent driving device provided by an embodiment of the present application.
  • Figure 7 is a schematic flow chart of a data fusion method provided by an embodiment of the present application.
  • Figure 8a is a top view of a first laser beam group and a second laser beam group provided by an embodiment of the present application;
  • Figure 8b is a side view of a first laser beam group and a second laser beam group provided by an embodiment of the present application;
  • Figure 9a is a top view of yet another first laser beam group and a second laser beam group provided by an embodiment of the present application.
  • Figure 9b is a side view of yet another first laser beam group and a second laser beam group provided by the embodiment of the present application.
  • Figure 10 is a schematic diagram of the surrounding environment provided by an embodiment of the present application.
  • Figure 11 is a schematic diagram of point cloud data provided by an embodiment of the present application.
  • Figure 12 is a schematic diagram of image data corresponding to a camera group provided by an embodiment of the present application.
  • Figure 13 is a schematic diagram of picture data corresponding to another camera group provided by an embodiment of the present application.
  • Figure 14 is a schematic diagram of the front, middle and rear fields of view provided by the embodiment of the present application.
  • Figure 15 is a schematic structural diagram of a data fusion device provided by an embodiment of the present application.
  • Figure 16 is a schematic diagram of an intelligent driving device provided by an embodiment of the present application.
  • Figure 17 is a schematic structural diagram of another data fusion device provided by an embodiment of the present application.
  • At least one mentioned in the embodiments of this application means one or more, and “multiple” means two or more. “At least one of the following” or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items).
  • at least one item (item) of a, b, or c can mean: a, b, c, "a and b", “a and c", “b and c”, or "a and b and c", where a, b, c can be single or multiple.
  • “And/or” describes the relationship between related objects, indicating that there can be three relationships.
  • a and/or B can mean: A alone exists, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural.
  • the character "/" generally indicates that the related objects are in an "or” relationship.
  • an embodiment means that a particular feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application.
  • the appearances of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art understand, both explicitly and implicitly, that the embodiments described herein may be combined with other embodiments.
  • Figure 1 is a schematic structural diagram of an intelligent driving device 100 provided by an embodiment of the present application.
  • the intelligent driving device 100 includes a computing platform 101 and M camera groups 102, where M is a positive integer greater than or equal to 1. .
  • M can also be a positive integer greater than or equal to 2.
  • the computing platform 101 may be installed in the intelligent driving device 100 or may not be installed in the intelligent driving device 100.
  • the computing platform 101 may be installed in a cloud server, which is not limited in the embodiments of this application.
  • the computing platform 101 can be understood as a device with computing capabilities.
  • the computing platform 101 is used to obtain point cloud data and M groups of picture data, and fuse the picture data and point cloud data corresponding to each camera group in the M groups of picture data to obtain the corresponding camera group for perception processing.
  • Data please see the following examples for specific relevant descriptions.
  • Each of the M camera groups includes at least one camera.
  • the intelligent driving device 100 includes 6 cameras, namely camera 1, camera 2, camera 3, camera 4, camera 5 and camera 6; each camera group includes 2 cameras; among them, camera 1 and camera 2 belong to group 1, camera 1 and Camera 2 is relatively arranged in front of the intelligent driving device 100; camera 3 and camera 4 belong to group 2, and camera 3 and camera 4 are relatively arranged on both sides of the intelligent driving device 100; camera 5 and camera 6 belong to group 3, and cameras 5 and 4
  • the camera 6 is relatively arranged at the rear of the intelligent driving device 100 .
  • the intelligent driving device 100 includes 6 cameras, namely camera 1, camera 2, camera 3, camera 4, camera 5, camera 6 and camera 7; each camera group includes at least 1 camera; wherein, camera 1 and Camera 2 belongs to group 1, and camera 1 and camera 2 are relatively arranged at the front of the intelligent driving device 100; camera 3 and camera 4 belong to group 2, and camera 3 and camera 4 are relatively arranged at both sides of the intelligent driving device 100; cameras 5 and Camera 6 belongs to group 3, and camera 5 and camera 6 are relatively arranged at the rear of the intelligent driving device 100; camera 7 belongs to group 4, and camera 7 faces the driving direction of the intelligent driving device 100.
  • the above Figures 2 and 3 serve as an example to illustrate the position of the camera in the smart driving device when the number of cameras in the smart driving device is an even number or an odd number. The camera position can also be different according to the car model of the smart driving device.
  • the embodiments of this application are not limiting.
  • the intelligent driving device 100 may also include one or more laser emitting devices 103.
  • the laser emitting device 103 is used to generate the first laser line beam group and the second laser line beam group, that is, it can be understood that the first laser line beam group and the second laser line beam group are from The same laser emitting device;
  • the intelligent driving device 100 includes two laser emitting devices 103, for example, a first laser emitting device and a second laser emitting device, the first laser emitting device is used to generate the first laser line beam group, and the second laser emitting device
  • the laser emitting device is used to generate the second laser beam group, that is, it can be understood that the first laser beam group and the second laser beam group come from different laser emitting devices, wherein the first laser beam group detects in the first scanning direction to obtain the second laser beam group.
  • One data, the second laser beam group detects in the second scanning direction to obtain the second data, and the first data and the second data are spliced to obtain the point cloud data.
  • the structure of the intelligent driving device in Figure 1 is only an exemplary implementation in the embodiment of the present application, and the intelligent driving device in the embodiment of the present application may also include more components as needed.
  • the intelligent driving device 100 may include various subsystems, such as a travel system 202 , a sensor system 204 , a control system 206 , one or more peripheral devices 208 as well as a power supply 210 , a computer system 212 and a user interface 216 .
  • the intelligent driving device 100 may include more or fewer subsystems, and each subsystem may include multiple elements.
  • each subsystem and component of the intelligent driving device 100 can be interconnected through wires or wirelessly.
  • the travel system 202 may include components that provide powered motion for the intelligent driving device 100 .
  • propulsion system 202 may include an engine 218 , an energy source 219 , a transmission 220 and wheels/tires 221 .
  • the engine 218 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine.
  • Engine 218 converts energy source 219 into mechanical energy.
  • Examples of energy sources 219 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity.
  • the energy source 219 can also provide energy for other systems of the intelligent driving device 100 .
  • Transmission 220 may transmit mechanical power from engine 218 to wheels 221 .
  • Transmission 220 may include a gearbox, differential, and driveshaft.
  • the transmission device 220 may also include other components, such as a clutch.
  • the drive shaft may include one or more axles that may be coupled to one or more wheels 221 .
  • the sensor system 204 may include several sensors that sense information about the environment surrounding the intelligent driving device 100 .
  • the sensor system 204 may include a positioning system 222 (the positioning system may be a GPS system, a Beidou system, or other positioning systems), an inertial measurement unit (IMU) 224, a radar 226, a laser rangefinder 228, and Camera 230.
  • the sensor system 204 may also include sensors that monitor internal systems of the intelligent driving device 100 (eg, in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding properties (position, shape, orientation, speed, etc.). This detection and identification are key functions for the safe operation of the autonomous intelligent driving device 100 .
  • the positioning system 222 may be used to estimate the geographical location of the intelligent driving device 100 .
  • the IMU 224 is used to sense the position and orientation changes of the intelligent driving device 100 based on inertial acceleration.
  • IMU 224 may be a combination of an accelerometer and a gyroscope.
  • the radar 226 may utilize radio signals to sense objects in the surrounding environment of the intelligent driving device 100 .
  • radar 226 may be used to sense the speed and/or heading of the object.
  • radar 226 may be laser emitting device 103 in FIG. 1 .
  • the intelligent driving device 100 includes a lidar device, and the lidar device includes two laser beam groups, namely a first laser beam group and a second laser beam group, wherein the first laser beam group includes at least one Laser beam, the second laser beam group includes at least one laser beam; the first laser beam group detects in the first scanning direction to obtain the first data; the second laser beam group detects in the second scanning direction to obtain the second data; the first scan The direction is different from the second scanning direction; that is, it can be understood that the rotation direction is different.
  • the first scanning direction can be a clockwise direction
  • the second scanning direction can be a counterclockwise direction.
  • the first data and the second data can be understood as point cloud information.
  • the intelligent driving equipment 100 includes two lidar devices, namely lidar device 1 and lidar device 2.
  • Lidar device 1 includes a laser beam group, which is the first laser beam group;
  • lidar device 2 includes a laser beam group, which is the second laser beam group; the first laser beam group detects the first data in the first scanning direction; the second laser beam group detects the second data in the second scanning direction; the first scan The direction is different from the second scanning direction.
  • the laser rangefinder 228 may use laser light to sense objects in the environment where the intelligent driving device 100 is located.
  • laser rangefinder 228 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.
  • the camera 230 may be used to capture multiple images of the surrounding environment of the intelligent driving device 100 .
  • Camera 230 may be a still camera or a video camera.
  • the number of cameras 230 may be one or more.
  • the cameras 230 can be divided into M camera groups 102 in Figure 1 .
  • the distribution of the cameras 230 in the intelligent driving device 100 can be as shown in Figure 2 or Figure 3 .
  • Control system 206 controls the operation of the intelligent driving device 100 and its components.
  • Control system 206 may include various elements, including steering system 232 , throttle 234 , braking unit 236 , sensor fusion algorithm 238 , computer vision system 240 , route control system 242 , and obstacle avoidance system 244 .
  • the steering system 232 is operable to adjust the forward direction of the intelligent driving device 100 .
  • it may be a steering wheel system.
  • the throttle 234 is used to control the operating speed of the engine 218 and thereby the speed of the intelligent driving device 100 .
  • the braking unit 236 is used to control the intelligent driving device 100 to decelerate. Braking unit 236 may use friction to slow wheel 221 . In other embodiments, the braking unit 236 may convert the kinetic energy of the wheels 221 into electrical current. The braking unit 236 may also take other forms to slow down the rotation speed of the wheels 221 to control the speed of the intelligent driving device 100 .
  • the computer vision system 240 may operate to process and analyze images captured by the camera 230 to identify objects and/or features in the environment surrounding the intelligent driving device 100 , and/or process and analyze data information captured by the radar 226 . Such objects and/or features may include traffic signals, road boundaries, and obstacles. Computer vision system 240 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, computer vision system 240 may be used to map an environment, track objects, estimate the speed of objects, and the like. Optionally, the computer vision system 240 may be the computing platform 101 in Figure 1 .
  • the route control system 242 is used to determine the driving route of the intelligent driving device 100 .
  • route control system 242 may combine data from sensor fusion algorithm 238 , positioning system 222 , and one or more predetermined maps to determine a driving route for intelligent driving device 100 .
  • the obstacle avoidance system 244 is used to identify, evaluate, and avoid or otherwise negotiate potential obstacles in the environment of the intelligent driving device 100 .
  • control system 206 may additionally or alternatively include components in addition to those shown and described. Alternatively, some of the components shown above may be reduced.
  • the intelligent driving device 100 interacts with external sensors, other vehicles, other computer systems, or users through peripheral devices 208 .
  • Peripheral devices 208 may include a wireless communication system 246 , an onboard computer 248 , a microphone 250 and/or a speaker 252 .
  • peripheral device 208 provides a means for a user of intelligent driving device 100 to interact with user interface 216 .
  • the on-board computer 248 may provide information to the user of the intelligent driving device 100 .
  • the user interface 216 may also operate the onboard computer 248 to receive user input.
  • the on-board computer 248 can be operated via a touch screen.
  • peripheral device 208 may provide a means for intelligent driving device 100 to communicate with other devices located within the vehicle.
  • microphone 250 may receive audio (eg, voice commands or other audio input) from a user of smart driving device 100 .
  • speaker 252 may output audio to a user of intelligent driving device 100 .
  • Wireless communication system 246 may wirelessly communicate with one or more devices directly or via a communication network.
  • wireless communication system 246 may use 3G cellular communications, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communications, such as LTE, or 5G cellular communications.
  • the wireless communication system 246 can communicate with a wireless local area network (WLAN) using Wi-Fi.
  • WLAN wireless local area network
  • wireless communication system 246 may utilize infrared links, Bluetooth, or ZigBee to communicate directly with the device.
  • Wireless communications system 246 may include one or more dedicated short range communications (DSRC) devices, which may include public and/or private data communications between vehicles and/or roadside stations.
  • DSRC dedicated short range communications
  • the power supply 210 may provide power to various components of the intelligent driving device 100 .
  • power source 210 may be a rechargeable lithium-ion or lead-acid battery.
  • One or more battery packs of such batteries may be configured as a power source to provide power to various components of the intelligent driving device 100 .
  • power supply 210 and energy source 219 may be implemented together.
  • Computer system 212 may include at least one processor 213 that executes instructions 215 stored in a non-transitory computer-readable medium such as memory 214.
  • the computer system 212 may also be multiple computing devices that control individual components or subsystems of the intelligent driving device 100 in a distributed manner.
  • Processor 213 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be such as an ASIC or other hardware-based processor.
  • the processor may be remote from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle and others are performed by a remote processor.
  • memory 214 may contain instructions 215 (eg, program logic) that may be executed by processor 213 to perform various functions of intelligent driving device 100 , including those described above.
  • Memory 214 may also contain additional instructions, including sending data to, receiving data from, interacting with, and/or controlling one or more of travel system 202, sensor system 204, control system 206, and peripherals 208. instructions.
  • the memory 214 can also store data, such as point cloud data and M sets of picture data corresponding to the M camera groups, and other information. This information may be used by the intelligent driving device 100 and the computer system 212 during operation of the intelligent driving device 100 in autonomous, semi-autonomous and/or manual modes.
  • the user interface 216 is used to provide information to or receive information from the user of the intelligent driving device 100 .
  • user interface 216 may include one or more input/output devices within a collection of peripheral devices 208 , such as wireless communications system 246 , on-board computer 248 , microphone 250 , and speaker 252 .
  • Computer system 212 may control functions of intelligent driving device 100 based on input received from various subsystems (eg, travel system 202 , sensor system 204 , and control system 206 ) and from user interface 216 .
  • computer system 212 may utilize input from control system 206 to control steering system 232 to avoid obstacles detected by sensor system 204 and obstacle avoidance system 244 .
  • computer system 212 is operable to provide control of many aspects of intelligent driving device 100 and its subsystems.
  • one or more of the above-mentioned components may be installed separately from or associated with the intelligent driving device 100 .
  • the memory 214 may exist partially or completely separately from the intelligent driving device 100 .
  • the components described above may be communicatively coupled together in wired and/or wireless manners.
  • the above-mentioned intelligent driving device 100 can be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, an entertainment vehicle, an amusement park vehicle, construction equipment, a tram, a golf cart, a train, a trolley, a smart Household equipment, etc. are not particularly limited in the embodiments of this application.
  • the structure of the intelligent driving equipment in Figure 4 is only an exemplary implementation in the embodiment of the present application.
  • the intelligent driving equipment in the embodiment of the present application includes but is not limited to the above structure.
  • Figure 5 is a schematic diagram of a driving control device of an intelligent driving device provided by an embodiment of the present application.
  • the controller 213 is coupled to the system bus 505.
  • Processor 213 may be one or more processors, each of which may include one or more processor cores.
  • the memory 214 can store relevant data information, and the memory 214 is coupled to the system bus 505 .
  • Display adapter (display adapter) 507, the display adapter 507 can drive the display 509, and the display 509 is coupled to the system bus 505.
  • System bus 505 is coupled to input/output (I/O) bus 513 through bus bridge 501 .
  • I/O input/output
  • I/O interface 515 is coupled to I/O bus 513.
  • the I/O interface 515 communicates with a variety of I/O devices, such as input devices 517 (such as keyboard, mouse, touch screen, etc.), multimedia tray (media tray) 521 (such as CD-ROM, multimedia interface, etc.).
  • Transceiver 523 can send and/or receive radio communication signals
  • camera 555 can capture scene and dynamic digital video images
  • external USB interface 525 external USB interface 525.
  • the interface connected to the I/O interface 515 may be a USB interface.
  • the processor 213 may be any traditional processor, including a reduced instruction set computer (RISC), a complex instruction set computer (CISC), or a combination of the above.
  • the processor may be an application specific integrated circuit ASIC.
  • the processor 213 may be a neural network processor or a combination of a neural network processor and the above-mentioned traditional processor.
  • the computer system 212 may be located remotely from the intelligent driving device and may communicate wirelessly with the intelligent driving device. In other aspects, some of the processes described herein are performed on a processor provided within the intelligent driving device, and others are performed by a remote processor.
  • Network interface 529 is a hardware network interface, such as a network card.
  • Network 527 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN).
  • the network 527 can also be a wireless network, such as a Wi-Fi network, a cellular network, etc.
  • the transceiver 523 (which can send and/or receive radio communication signals) can be used through various wireless communication methods such as 2G, 3G, 4G, 5G, etc., and can also be DSRC. technology, or long-term evolution vehicle-to-everything (LTE-V2X) technology, etc., its main function is to receive information data sent by external devices and convert the information generated by the intelligent driving device when driving on the target road section. The information data is sent back to the external device for storage and analysis.
  • LTE-V2X long-term evolution vehicle-to-everything
  • the hard disk drive interface 531 is coupled to the system bus 505.
  • the hard disk drive interface 531 and the hard disk drive 533 are connected.
  • System memory 535 is coupled to system bus 505 .
  • the data running in system memory 535 may include the operating system 537 and application programs 543 of the computer system 212 .
  • System memory 535 is coupled to system bus 505 .
  • the system memory 535 in this application can be used to store the driving information of vehicles passing the target road section in the memory in a certain format.
  • the operating system (OS) 537 includes a shell 539 and a kernel 541.
  • Shell 539 is an interface between the user and the kernel of the operating system.
  • Shell 539 is the outermost layer of the operating system.
  • Shell 539 manages the interaction between the user and the operating system: waiting for user input; interpreting user input to the operating system; and processing various operating system output results.
  • Kernel 541 consists of those parts of the operating system that manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel usually runs processes and provides inter-process communication, providing CPU time slice management, interrupts, memory management, IO management, etc.
  • Application programs 543 include self-driving related programs 547, such as programs that manage the interaction between self-driving cars and obstacles on the road, programs that control the route or speed of self-driving cars, and programs that control the interaction between self-driving cars and other self-driving cars on the road.
  • Application 543 also exists on the system where the deploying server 549 is deployed.
  • the computer system 212 may download the automatic driving related program 547 from the deployment server 549 .
  • Sensor 553 is associated with computer system 212.
  • Sensor 553 is used to detect the environment surrounding computer system 212.
  • the sensor 553 can detect animals, cars, obstacles and crosswalks, etc.
  • the sensor can also detect the environment around the above-mentioned animals, cars, obstacles and crosswalks, such as: the environment around animals, for example, the environment around animals. other animals, weather conditions, brightness of the surrounding environment, etc.
  • the sensors may be cameras, infrared sensors, chemical detectors, microphones, etc.
  • the driving control device of the intelligent driving equipment in Figure 5 is only an exemplary implementation in the embodiment of the present application.
  • the driving control device applied to the intelligent driving equipment in the embodiment of the present application includes but is not limited to The above structure.
  • the lidar device has a fixed scanning period, and the camera is triggered at a fixed frequency, as shown in Figure 6.
  • Figure 6 is a schematic diagram of the position distribution of the camera in an intelligent driving device.
  • the intelligent The driving equipment includes a lidar device and 6 cameras, which are T1, T2, T3, T4, T5, and T6.
  • the lidar device is located in the middle of the intelligent driving equipment, and only one laser beam group is emitted in the lidar device.
  • the laser beam group includes at least one laser beam.
  • the scanning starting position of the lidar device can be camera T1
  • the position can be rotated clockwise from the position of camera T1 and scanned one week in sequence. The time to scan to each camera is different.
  • the time to scan to the position of camera T1 is different from the time to scan to the position of camera T6.
  • the difference is very large, and correspondingly, the time difference between the corresponding point cloud data obtained is also very large.
  • the usual approach is to place the starting scanning angle with the largest difference in a direction with lower accuracy requirements such as backward.
  • the scanning starting position of the lidar device can be the position of camera T3, which can be determined by camera T3.
  • the position is rotated clockwise and scanned one circle in sequence, but this is only an evasive measure; in this way, the point cloud can be compensated, but for the scene of the oncoming car, since the speed of the other party is not accurately estimated, the accuracy of the point cloud compensation Poor.
  • the six cameras can be triggered at the same time, that is, six image data corresponding to the six cameras can be obtained at the same time, thereby triggering the corresponding model inference, but this will cause resource consumption. Concentrated occupancy and peak load competition lead to performance problems.
  • different cameras can be triggered in a time-sharing manner, that is, when the lidar device scans to the corresponding position of the camera, the camera exposure will be triggered to obtain the corresponding picture data. In this way, a single-view point can be achieved. The best match between the cloud and the image, and load balancing is done to a certain extent. However, due to the difference in the time when each camera is triggered, there is a time difference between the obtained picture data.
  • this application Based on the structure of the intelligent driving equipment provided in Figures 1 and 4 and the vehicle driving control device applied to the intelligent driving equipment provided in Figure 5, this application provides a data fusion method.
  • Figure 7 is a schematic flow chart of a data fusion method provided by an embodiment of the present application. This method can be applied to the intelligent driving device described in Figure 1 above, where the computing platform can be used to support and execute the figure.
  • the method flow shown in 7 is step S701-step S703.
  • This method can be applied in intelligent driving scenarios, which include intelligent assisted driving, automatic driving, driverless driving, etc., which are not limited by the embodiments of this application.
  • the method may include the following steps S701 to S703.
  • the point cloud data is obtained by detecting the first laser beam group and the second laser beam group in the first scanning direction and the second scanning direction respectively. Specifically, it may include: acquiring the first data and the second data, the first data is detected by the first laser beam group in the first scanning direction, and the second data is detected by the second laser beam group in the second scanning direction. Then, the first data and the second data are spliced to obtain point cloud data. .
  • the first laser beam group and the second laser beam group come from the same laser emitting device, and the laser emitting device may be called a bidirectional lidar device.
  • the first laser line beam group comes from the first laser line beam emitting device
  • the second laser line beam group comes from the second laser line beam emitting device, that is, it can be understood as the first laser line beam group and the second laser line beam group.
  • the first laser beam emitting device and the second laser emitting device may be lidar devices.
  • the first laser beam group includes at least one laser beam
  • the second laser beam group includes at least one laser beam.
  • the top view of the first laser line beam group and the second laser line beam group is as shown in Figure 8a, and the side view is as shown in Figure 8b. Show.
  • the top view of the first laser line beam group and the second laser line beam group is as shown in Figure 9a, and the side view is as shown in Figure 9a As shown in 9b.
  • first scanning direction and the second scanning direction can be understood as different rotation directions.
  • first scanning direction is clockwise and the second scanning direction is counterclockwise; for another example, the first scanning direction is counterclockwise.
  • second scanning direction is clockwise.
  • point cloud data may refer to partial environmental point cloud data, not complete environmental point cloud data.
  • the first data is data obtained by the first laser line beam group scanning a certain sector range in the first scanning direction
  • the second data is data obtained by the second laser line beam group scanning a certain sector range in the second scanning direction.
  • the scanning period of the first laser beam group and the second laser beam group are the same, which can be understood as the same scanning speed.
  • the starting scanning positions of the first laser beam group and the second laser beam group are adjustable. Splicing the first data and the second data can be understood as rotating and translating the first data and the second data to a unified coordinate system so that they can form part of the environmental point cloud data, that is, it can be understood as combining the first data and the second data.
  • Figure 10 is a schematic diagram of the surrounding environment provided by an embodiment of the present application; (a) in Figure 11 is the first laser beam group detecting the first result in the first scanning direction. A schematic diagram of data, in which the driving direction of the intelligent driving device is used as the reference, and the first scanning direction is counterclockwise; (b) in Figure 11 is the second data detected by the second laser beam group in the second scanning direction. A schematic diagram, in which the driving direction of the intelligent driving device is used as a reference, and the second scanning direction is clockwise; (c) in Figure 11 is a schematic diagram of splicing the first data and the second data to obtain point cloud data.
  • S702 Trigger the camera exposures in the M camera groups in sequence according to the trigger delays of the M camera groups to obtain M groups of picture data.
  • cameras in the same camera group are exposed at the same time. For example, if a camera group includes camera 1 and camera 2, then camera 1 and camera 2 are exposed at the same time.
  • the image data of each group in M groups of image data and the number of cameras in each group are the same.
  • a camera group includes camera 1 and camera 2.
  • there are 2 image data in the camera group that is, the The two image data are a set of image data, that is, the image data corresponding to one camera group.
  • M is a positive integer greater than or equal to 1.
  • M can also be a positive integer greater than or equal to 2.
  • the triggering delay is related to the scanning period of the first laser beam group and the second laser beam group, and the scanning period can be understood as the scanning speed.
  • the trigger delay is obtained by: determining the position of each camera group in the M camera groups; determining the trigger delay of each camera group based on the scanning period and the position of each camera group in the M camera groups.
  • the position of each camera in the M camera groups may refer to the spatial location of the camera distribution.
  • the cameras may be distributed on the front, sides or rear of the intelligent driving device.
  • the triggering delay of each camera group is determined according to the scanning period and the position of each camera group in the M camera groups, as shown in formula (1), specifically as follows:
  • ⁇ i,j is the trigger delay of the i-th camera (i ⁇ (0,M]) when scanning the j-th group (j ⁇ (0,2]) laser beam
  • ⁇ i,j is the i-th
  • the offset angle of the camera rotated in its scanning direction relative to the starting scanning angle of the j-th group of laser beams, is the scanning angular velocity of the jth group of laser beams.
  • the M camera groups include front groups, side groups and rear groups.
  • the front grouping includes a first camera and a second camera, and the first camera and the second camera are arranged oppositely at the front of the intelligent driving device;
  • the side grouping includes a third camera and a fourth camera, and the third camera and the fourth camera are arranged oppositely.
  • the rear group includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the smart driving device.
  • M 3, the three camera groups are group 1, group 2 and group 3 respectively.
  • the intelligent driving device includes 6 cameras, namely camera 1, camera 2, camera 3, camera 4 and camera 5. and camera 6; each camera group includes 2 cameras.
  • each camera group includes 2 cameras.
  • the M camera groups may also include a forward group including a seventh camera facing the driving direction of the intelligent driving device.
  • the intelligent driving device 100 includes 6 cameras, which are camera 1, camera 2, camera 3 and camera 3 respectively. 4. Camera 5, camera 6 and camera 7; each camera group includes at least 1 camera.
  • each camera group includes at least 1 camera.
  • the M camera groups include a first camera group and a second camera group.
  • the trigger delay of the first camera group is the first delay
  • the trigger delay of the second camera group is the second delay.
  • the camera exposures in the M camera groups are sequentially triggered to obtain M groups of picture data, including: starting the first timer to set the initial time; when the indicated time of the first timer The first time delay is separated from the initial time, and the camera exposure in the first camera group is triggered to obtain the picture data corresponding to the first camera group; when the indicated time of the first timer is separated by the second time delay from the initial time, the second camera exposure is triggered.
  • the cameras in the two camera groups are exposed to obtain image data corresponding to the second camera group.
  • the first delay may be 0 or other values.
  • the first delay is 0, it can be understood that when the first timer is started, the camera exposure in the first camera group is triggered to obtain the picture data corresponding to the first camera group.
  • the first laser beam group and the second laser beam group can be detected in the first scanning direction and the second scanning direction respectively at the initial moment.
  • the first laser beam group and the second laser beam group can also be detected in the first scanning direction and the second scanning direction respectively before or after the initial time, which is not limited by the embodiment of the present application.
  • the second delay may be greater than the first delay.
  • the picture data corresponding to the first camera group may refer to the picture data captured by each camera in the first camera group. For example, if the first camera group includes camera 1 and camera 2, then the picture data corresponding to the first camera group It can refer to the image data exposed by camera 1 and the image data exposed by camera 2. The same applies to the image data corresponding to the second camera group.
  • the number of first timers may be one or more. When the number of first timers is multiple, the number of first timers is the same as the number of cameras in the camera group. For example, the number of cameras in the first camera group The number of is 2, namely camera 1 and camera 2, then the number of first timers is 2, namely timer 1 and timer 2, where timer 1 and timer 2 can be synchronized, for example, The first delay is 0. Start timer 1 and timer 2 to trigger the exposure of camera 1 and camera 2 in the first camera group to obtain the picture data corresponding to the first camera group. That is, timer 1 triggers the exposure of camera 1 to obtain camera 1. According to the corresponding picture data, timer 2 triggers camera 2 to expose and obtains the corresponding picture data of camera 2.
  • the first delay is 0 milliseconds (ms)
  • the second delay is 30ms
  • the first camera group includes camera 1 and camera 2
  • the second camera group includes camera 3 and camera 4
  • the first timing is started.
  • the initial time of the timer is set to t0.
  • the indication time of the first timer is t0
  • the first delay between the indication time t0 and the initial time t0 is 0ms
  • the exposure of camera 1 and camera 2 is triggered to obtain the picture captured by camera 1. data and image data captured by camera 2.
  • Figure 12 is the image data captured by the camera 1, that is, the image data in the front field of view of the driving direction of the intelligent driving device
  • Figure 12 is the picture data captured by the camera 2, that is, the picture data in the front field of view of the driving direction of the intelligent driving device; when the indicated time of the first timer is t1, the indicated time t1 is separated from the initial time t0 by When the second delay is 30ms, trigger the exposure of camera 3 and camera 4 to obtain the image data captured by camera 3 and the image data captured by camera 4.
  • FIG. 13 For the environment in Figure 10, as shown in Figure 13, ( in Figure 13 a) is the image data captured by camera 3, that is, the image data in the middle field of view of the driving direction of the intelligent driving device. (b) in Figure 13 is the image data captured by camera 4, that is, the driving direction of the intelligent driving device. Image data in the mid-field of view.
  • fusing the picture data corresponding to each camera group in the M camera groups and the point cloud data to obtain the data corresponding to the camera group for perception processing may refer to merging the pictures corresponding to each camera group in the M camera groups.
  • Data and point cloud data are used as input data, and the input data is input into a fusion perception algorithm, such as a multi-modal fusion algorithm, to obtain data corresponding to the camera grouping for perception processing.
  • (c) in Figure 11 is a schematic diagram of point cloud data obtained by splicing the first data and the second data based on the environment of Figure 10; as shown in Figure 12, (a) in Figure 12 is a schematic diagram of the first camera grouping
  • the image data captured by camera 1, (b) in Figure 12 is the image data captured by camera 2 in the first camera group; accordingly, the point cloud data in (c) in Figure 11 and the point cloud data in Figure 12 can be
  • the image data captured by camera 1 in the first camera group of (a) and the image data captured by camera 2 in the first camera group of (b) in Figure 12 are fused to obtain the data for perception processing corresponding to the camera group.
  • Figure 14 is a schematic diagram of the front, middle and rear fields of view of the intelligent driving device in the environment shown in Figure 10.
  • the point cloud data and picture data in the front field of view are obtained.
  • the process of fusion results is similar.
  • the fusion result of point cloud data and picture data in the middle field of view of the driving direction of the intelligent driving device can be obtained, as well as the fusion result of point cloud data and picture data in the rear field of view of the driving direction of the intelligent driving device. result.
  • the picture data corresponding to each camera group in the M camera groups can be fused, that is, it can be understood that the picture data corresponding to at least two camera groups in the M camera groups can be As input data, this input data is input into a camera perception algorithm, such as an image 2D target detection algorithm, to obtain corresponding results.
  • a camera perception algorithm such as an image 2D target detection algorithm
  • the image data of camera 1 and the image data of camera 2 in camera group 1, and the image data of camera 3 and camera 4 in camera group 2 can be input to the camera perception algorithm to obtain corresponding results, for example, this
  • the results can be used for target obstacle detection, etc.
  • the point cloud data can be fused with the picture data of the camera group corresponding to the front field of view on the intelligent driving device to improve the points of the front field of view.
  • the fusion effect of cloud data and picture data can also improve the fusion effect of point cloud data and picture data in the middle field of view and rear field of view, thereby making the vehicle surrounding environment recognition more accurate and more conducive to the implementation of autonomous driving task planning. .
  • Figure 15 is a schematic structural diagram of a data fusion device 1500 provided by an embodiment of the present application.
  • the data fusion device 1500 is a vehicle-mounted computing platform or a cloud server.
  • the data fusion device 1500 includes: a communication unit 1501 and a processing unit. 1502.
  • the device 1500 can be implemented by hardware, software, or a combination of software and hardware.
  • the processing unit 1502 is used to obtain point cloud data, which is obtained by detecting the first laser beam group and the second laser beam group in the first scanning direction and the second scanning direction respectively, wherein, The first scanning direction and the second scanning direction are different;
  • the processing unit 1502 is configured to sequentially trigger camera exposures in the M camera groups according to the trigger delays of the M camera groups to obtain M groups of picture data; where, the cameras in the same camera group are exposed at the same time,
  • the triggering delay is related to the scanning period of the first laser beam group and the second laser beam group, and M is a positive integer greater than or equal to 1;
  • the processing unit 1502 is configured to fuse the image data corresponding to each camera group in the M camera groups and the point cloud data to obtain data corresponding to the camera group for perception processing.
  • the processing unit 1502 is configured to obtain first data, which is detected by the first laser beam group in the first scanning direction; the processing unit 1502, for acquiring second data, which is detected by the second laser beam group in the second scanning direction; the processing unit 1502, for splicing the first data and the Second data to obtain the point cloud data.
  • the first laser beam group and the second laser beam group come from the same laser emitting device.
  • the first laser beam group comes from a first laser emitting device
  • the second laser beam group comes from a second laser emitting device
  • the processing unit 1502 is configured to determine the position of each of the M camera groups; according to the scanning period and each of the M camera groups The position determines the trigger delay of each camera group.
  • the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is the first delay, and the trigger delay of the second camera group is The trigger delay is the second delay; the processing unit 1502 is used to start the first timer to set the initial time; the processing unit 1502 is used to set the indicated time of the first timer and the initial time. After the first time delay, trigger the camera exposure in the first camera group to obtain the picture data corresponding to the first camera group; the processing unit 1502 is configured to set the first timer When the indicated time is separated by the second time delay from the initial time, the camera exposure in the second camera group is triggered to obtain the picture data corresponding to the second camera group.
  • the M camera groups include a front group, a side group, and a rear group;
  • the front group includes a first camera and a second camera, and the first camera and all The second camera is relatively disposed on the front of the intelligent driving device;
  • the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively disposed on both sides of the intelligent driving device.
  • the rear grouping includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.
  • the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.
  • Figure 16 is a schematic diagram of an intelligent driving device 1600 provided by an embodiment of the present application.
  • the intelligent driving device 1600 includes a computing platform 1601 and M camera groups 1602.
  • the computing platform 1601 is used for:
  • Obtain point cloud data which are obtained by detecting the first laser beam group and the second laser beam group in the first scanning direction and the second scanning direction respectively, wherein the first scanning direction and the The second scanning direction is different;
  • the camera exposures in the M camera groups are sequentially triggered according to the trigger delays of the M camera groups to obtain M groups of picture data; where, the cameras in the same camera group are exposed at the same time, and the trigger delay is the same as
  • the scanning periods of the first laser beam group and the second laser beam group are related, and M is a positive integer greater than or equal to 1;
  • the image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain the data corresponding to the camera group for perception processing.
  • the computing platform 1601 when acquiring the point cloud data, is configured to: acquire first data, the first data being the first laser beam group in the first scan. direction detection; obtain second data, which is detected by the second laser beam group in the second scanning direction; splice the first data and the second data to obtain the Point cloud data.
  • the first laser beam group and the second laser beam group come from the same laser emitting device.
  • the first laser beam group comes from a first laser emitting device
  • the second laser beam group comes from a second laser emitting device
  • the first laser emitting device and the second laser emitting device are lidar devices.
  • the computing platform 1601 is further configured to: determine the position of each of the M camera groups; The position of the group determines the trigger delay for each camera group.
  • the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is the first delay, and the trigger delay of the second camera group is The trigger delay is the second delay.
  • the computing platform 1601 is also used to: start the first timer to set the initial time; and space the first time delay between the indicated time of the first timer and the initial time. In the case of , trigger the camera exposure in the first camera group to obtain the picture data corresponding to the first camera group. When the indicated time of the first timer is separated by the second time delay from the initial time, camera exposure in the second camera group is triggered to obtain picture data corresponding to the second camera group.
  • the M camera groups include a front group, a side group, and a rear group;
  • the front group includes a first camera and a second camera, and the first camera and all The second camera is relatively disposed on the front of the intelligent driving device;
  • the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively disposed on both sides of the intelligent driving device.
  • the rear grouping includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.
  • the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.
  • Figure 17 is a schematic diagram of a data fusion device 1700 provided by an embodiment of the present application.
  • the data fusion device 1700 can be a vehicle-mounted computing platform or a cloud server.
  • the data fusion device 1700 includes at least one processor 1701 and a communication device.
  • the interface 1703 optionally also includes a memory 1702.
  • the processor 1701, the memory 1702 and the communication interface 1703 are connected to each other through a bus 1704.
  • the data fusion device 1700 can be a vehicle-mounted computing platform or a cloud server.
  • Memory 1702 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or Portable read-only memory (compact disc read-only memory, CD-ROM), the memory 1702 is used for related computer programs and data.
  • Communication interface 1703 is used to receive and send data.
  • the processor 1701 may be one or more central processing units (CPUs).
  • CPUs central processing units
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the processor 1701 in the data fusion device 1700 is used to read the computer program code stored in the memory 1702 and perform the following operations:
  • Obtain point cloud data which are obtained by detecting the first laser beam group and the second laser beam group in the first scanning direction and the second scanning direction respectively, wherein the first scanning direction and the The second scanning direction is different;
  • the camera exposures in the M camera groups are sequentially triggered according to the trigger delays of the M camera groups to obtain M groups of picture data; wherein, the cameras in the same camera group are exposed at the same time, and the trigger delay is the same as the trigger delay.
  • the scanning periods of the first laser beam group and the second laser beam group are related, and M is a positive integer greater than or equal to 1;
  • the image data corresponding to each camera group in the M camera groups and the point cloud data are fused to obtain the data corresponding to the camera group for perception processing.
  • the processor 1701 is configured to obtain first data, which is detected by the first laser beam group in the first scanning direction; the processor 1701, used to obtain second data, which is detected by the second laser beam group in the second scanning direction; the processor 1701, used to splice the first data and the Second data to obtain the point cloud data.
  • the first laser beam group and the second laser beam group come from the same laser emitting device.
  • the first laser beam group comes from a first laser emitting device
  • the second laser beam group comes from a second laser emitting device
  • the processor 1701 is configured to determine the position of each of the M camera groups; according to the scanning period and each of the M camera groups The position determines the trigger delay of each camera group.
  • the M camera groups include a first camera group and a second camera group, the trigger delay of the first camera group is the first delay, and the trigger delay of the second camera group is The trigger delay is the second delay; the processor 1701 is used to start the first timer to set the initial time; the processor 1701 is used to adjust the indicated time of the first timer and the initial time. After the first time delay, trigger the camera exposure in the first camera group to obtain the picture data corresponding to the first camera group; the processor 1701 is configured to set the first timer When the indicated time is separated by the second time delay from the initial time, the camera exposure in the second camera group is triggered to obtain the picture data corresponding to the second camera group.
  • the M camera groups include a front group, a side group, and a rear group;
  • the front group includes a first camera and a second camera, and the first camera and all The second camera is relatively disposed on the front of the intelligent driving device;
  • the side group includes a third camera and a fourth camera, and the third camera and the fourth camera are relatively disposed on both sides of the intelligent driving device.
  • the rear grouping includes a fifth camera and a sixth camera, and the fifth camera and the sixth camera are relatively arranged at the rear of the intelligent driving device.
  • the M camera groups further include a forward group, the forward group includes a seventh camera, and the seventh camera faces the driving direction of the intelligent driving device.
  • processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processor, digital signal processor (DSP), or application-specific integrated circuit (application-specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.
  • a general-purpose processor can be a microprocessor or any conventional processor.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment, or other programmable device.
  • the computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.
  • the computer program or instructions may be transmitted from a website, computer, A server or data center transmits via wired or wireless means to another website site, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media.
  • the available media may be magnetic media, such as floppy disks, hard disks, and tapes; optical media, such as digital video optical disks; or semiconductor media, such as solid-state hard drives.
  • the computer-readable storage medium may be volatile or nonvolatile storage media, or may include both volatile and nonvolatile types of storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Traffic Control Systems (AREA)

Abstract

本申请实施例提供一种数据融合方法、装置及智能驾驶设备，该方法包括：获取点云数据，点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，第一扫描方向和第二扫描方向不同；根据M个相机分组的触发时延依次触发M个相机分组内的相机曝光以得到M组图片数据；将M个相机分组中每个相机分组对应的图片数据和点云数据进行融合以得到相机分组对应的用于感知处理的数据，采用本申请实施例能够提升激光发射装置扫描得到的点云数据和相机捕捉到的图片数据的融合效果，从而使得车周环境识别更准确，更有利于自动驾驶任务的规划和实施。

Description

一种数据融合方法、装置及智能驾驶设备 技术领域
本申请涉及智能驾驶技术，应用于智能驾驶的融合感知领域，尤其涉及一种数据融合方法、装置及智能驾驶设备。
背景技术
随着社会的发展，智能汽车正在逐步进入人们的日常生活中。高级驾驶辅助系统(advanced driverassistance system，ADAS)在智能汽车中发挥着十分重要的作用，它是利用安装在车上的各式各样的传感器，在车辆行驶过程中感应周围的环境，收集数据，进行静止、移动物体的辨识、侦测与追踪，并结合导航仪地图数据，进行系统的运算与分析，从而预先让驾驶者察觉到可能发生的危险，有效增加汽车驾驶的舒适性和安全性。总之，传感器在智能汽车的辅助驾驶和自动驾驶中发挥着十分重要的作用，被比作为汽车的“眼睛”。传感器包括车载摄像头等视觉系传感器和车载激光雷达和车载超声波雷达等雷达系传感器。
相对于单一的传感器感知，基于多传感器的融合感知越来越成为智能驾驶的主流，数据的多来源让最终的感知结果变得更加稳定可靠，更能利用到每个传感器的优势而避免缺陷。
在目前的多传感器的融合方案中，激光雷达有固定的扫描周期，而相机是定频触发的，在激光雷达与相机融合的场景下，常出现由于激光雷达与相机时间戳不匹配，导致融合效果差，从而无法准确的识别车周环境的问题。
发明内容
本申请公开了一种数据融合方法、装置及智能驾驶设备，能够提升激光发射装置扫描得到的点云数据和相机捕捉到的图片数据的融合效果，从而使得车周环境识别更准确，更有利于自动驾驶任务的规划和实施。
第一方面，本申请提供了一种数据融合方法，该方法包括：获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
上述M个相机分组可以根据多个相机在智能驾驶设备上的安装位置确定。以智能驾驶设备的行驶方向为基准，通过本申请，可以将该点云数据和智能驾驶设备上前视场对应的相机分组的图片数据进行融合，以提升前视场的点云数据和图片数据的融合效果，类似的，也可以提升中视场以及后视场的点云数据和图片数据的融合效果，从而使得车周环境识别更准确，更有利于自动驾驶任务的规划和实施。
在一种可能的实现方式中，所述获取点云数据包括：获取第一数据，所述第一数据为所 述第一激光线束组在所述第一扫描方向探测得到的；获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；拼接所述第一数据和所述第二数据以获取所述点云数据。
在上述方法中，将第一数据和第二数据拼接融合增强，相比单一方向扫描得到的点云数据，本申请中的点云数据的点密度更高，对车周环境的识别准确度更高。
在又一种可能的实现方式中，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
在又一种可能的实现方式中，所述第一激光线束组来自第一激光发射装置，所述第二激光线束组来自第二激光发射装置。
在上述方法中，通过两个激光线束组，能够覆盖式扫描，从而使得点云数据的点密度更高。
在又一种可能的实现方式中，所述方法还包括：确定所述M个相机分组中每个相机分组的位置；根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
在又一种可能的实现方式中，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延；所述根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据，包括：启动第一定时器以设置初始时刻；当所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延时，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据；当所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延时，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
在上述方法中，通过根据M个触发时延依次触发M个相机分组内的相机曝光以得到M组图片数据的方式，相比同时曝光智能驾驶设备中的所有相机导致会造成资源的集中占用，负载出现峰值竞争导致性能问题的方式，本申请的依次触发曝光能够在一定程度上进行负载均衡，从而避免同时曝光产生的问题。
在又一种可能的实现方式中，所述M个相机分组包括前部分组、侧部分组和后部分组；所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在智能驾驶设备的前部；所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
在又一种可能的实现方式中，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
在又一种可能的实现方式中，将所述M个相机分组中至少两个相机分组对应的图片数据进行融合以得到用于感知处理的数据。
第二方面，本申请提供了一种智能驾驶设备，包括：计算平台和M个相机分组，所述计算平台用于：获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；根据所述M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；将所述M个相机 分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
在一种可能的实现方式中，当获取所述点云数据时，所述计算平台用于：获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；拼接所述第一数据和所述第二数据以获取所述点云数据。
在又一种可能的实现方式中，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
在又一种可能的实现方式中，所述智能驾驶设备还包括：第一激光发射装置和第二激光发射装置；所述第一激光线束组来自所述第一激光发射装置，所述第二激光线束组来自所述第二激光发射装置。
在又一种可能的实现方式中，所述第一激光发射装置和所述第二激光发射装置为激光雷达装置。
在又一种可能的实现方式中，所述计算平台还用于：确定所述M个相机分组中每个相机分组的位置；根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
在又一种可能的实现方式中，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延，所述计算平台还用于：启动第一定时器以设置初始时刻；在所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延的情况下，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据。在所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延的情况下，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
在又一种可能的实现方式中，所述M个相机分组包括前部分组、侧部分组和后部分组；所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在所述智能驾驶设备的前部；所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
在又一种可能的实现方式中，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
关于第二方面或可能的实现方式所带来的技术效果，可参考对于第一方面或相应的实施方式的技术效果的介绍。
第三方面，本申请提供了一种数据融合装置，所述数据融合装置为车载计算平台或云端服务器或芯片系统，所述数据融合装置包括处理器和存储器，所述存储器和所述处理器通过线路互联，所述处理器用于获取所述存储器中存储的计算机程序，当所述计算机程序被所述处理器调用时，用于执行上述第一方面以及各实现方式中的数据融合方法。
第四方面，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机指令，当所述计算机指令在处理器上运行时，以实现上述第一方面或第一方面中任一可能的实现方式所述的方法。
第五方面，本申请提供了一种计算机程序产品，所述计算机程序产品中包括计算机程序代码，当所述计算机程序代码在计算机上运行时，以实现上述第一方面或第一方面中任一可 能的实现方式所述的方法。
第六方面，本申请提供了一种芯片，所述芯片包括电路，所述电路用于实现上述第一方面或第一方面中任一可能的实现方式所述的方法。
附图说明
图1是本申请实施例提供的一种智能驾驶设备的结构示意图；
图2是本申请实施例提供的一种相机分组的示意图；
图3是本申请实施例提供的又一种相机分组的示意图；
图4是本申请实施例提供的另一种智能驾驶设备的结构示意图；
图5是本申请实施例提供的一种智能驾驶设备的行驶控制装置示意图；
图6是本申请实施例提供的一种智能驾驶设备中相机的位置分布示意图；
图7是本申请实施例提供的一种数据融合方法的流程示意图；
图8a是本申请实施例提供的一种第一激光线束组和第二激光线束组的俯视图；
图8b是本申请实施例提供的一种第一激光线束组和第二激光线束组的侧视图；
图9a是本申请实施例提供的又一种第一激光线束组和第二激光线束组的俯视图；
图9b是本申请实施例提供的又一种第一激光线束组和第二激光线束组的侧视图；
图10是本申请实施例提供的一种周围环境示意图；
图11是本申请实施例提供的一种点云数据的示意图；
图12是本申请实施例提供的一种相机分组对应的图片数据的示意图；
图13是本申请实施例提供的又一种相机分组对应的图片数据的示意图；
图14是本申请实施例提供的一种前、中以及后视场的示意图；
图15是本申请实施例提供的一种数据融合装置的结构示意图；
图16是本申请实施例提供的一种智能驾驶设备的示意图；
图17是本申请实施例提供的又一种数据融合装置的结构示意图。
具体实施方式
下面结合本申请实施例中的附图对本申请实施例进行描述。
本申请中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其他实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
本申请中实施例提到的“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a、b、或c中的至少一项(个)，可以表示：a、b、c、“a和b”、“a和c”、“b和c”、或“a和b和c”，其中a、b、c可以是单个，也可以是多个。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A、同时存在A和B、单独存在B这三种情况，其中A、B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。
本申请中实施例提到的“第一”、“第二”、“第三”和“第四”等是用于区别不同对象，而不是用于描述特定顺序。此外，术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的 包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本文中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。
请参见图1，图1是本申请实施例提供的一种智能驾驶设备100的结构示意图，该智能驾驶设备100包括计算平台101和M个相机分组102，其中，M为大于等于1的正整数。M也可以为大于等于2的正整数。其中，该计算平台101可以搭载在智能驾驶设备100中，也可以不搭载在该智能驾驶设备100中，例如，该计算平台101可以搭载在云端服务器中，本申请实施例不做限定。该计算平台101可以理解为具有计算能力的装置。其中，该计算平台101用于获取点云数据以及M组图片数据，并将M组图片数据中每个相机分组对应的图片数据和点云数据进行融合以得到相机分组对应的用于感知处理的数据，具体相关描述见以下实施例。其中，M个相机分组中的每个相机分组中包括至少一个相机。在一种示例中，请参见图2，图2是本申请实施例提供的一种相机分组的示意图，M＝3，3个相机分组分别为分组1、分组2和分组3，该智能驾驶设备100包括6个相机，分别为相机1、相机2、相机3、相机4、相机5和相机6；每个相机分组中包括2个相机；其中，相机1和相机2属于分组1，相机1和相机2相对设置在智能驾驶设备100的前部；相机3和相机4属于分组2，相机3和相机4相对设置在智能驾驶设备100的两侧；相机5和相机6属于分组3，相机5和相机6相对设置在智能驾驶设备100的后部。在又一种示例中，请参见图3，图3是本申请实施例提供的又一种相机分组的示意图，M＝4，4个相机分组分别为分组1、分组2、分组3和分组4，该智能驾驶设备100包括6个相机，分别为相机1、相机2、相机3、相机4、相机5、相机6和相机7；每个相机分组中包括至少1个相机；其中，相机1和相机2属于分组1，相机1和相机2相对设置在智能驾驶设备100的前部；相机3和相机4属于分组2，相机3和相机4相对设置在智能驾驶设备100的两侧；相机5和相机6属于分组3，相机5和相机6相对设置在智能驾驶设备100的后部；相机7属于分组4，相机7朝向智能驾驶设备100的行驶方向。上述图2和图3作为一种示例说明当智能驾驶设备中相机的数量为偶数个或奇数个时相机在智能驾驶设备的位置，相机位置还可以根据智能驾驶设备的车型情况等有所不同，本申请实施例不做限定。
其中，该智能驾驶设备100中还可以包括一个或多个激光发射装置103。当智能驾驶设备100包括一个激光发射装置103时，该激光发射装置103用于生成第一激光线束组和第二激光线束组，也即可以理解为第一激光线束组和第二激光线束组来自同一激光发射装置；当该智能驾驶设备100包括2个激光发射装置103，例如，第一激光发射装置和第二激光发射装置时，第一激光发射装置用于生成第一激光线束组，第二激光发射装置用于生成第二激光线束组，也即可以理解为第一激光线束组和第二激光线束组来自不同的激光发射装置，其中第一激光线束组在第一扫描方向进行探测得到第一数据，第二激光线束组在第二扫描方向进行探测得到第二数据，拼接第一数据和第二数据得到点云数据。具体如下述实施例中的相关描述。
可以理解的是，图1中的智能驾驶设备的结构只是本申请实施例中的一种示例性的实施 方式，本申请实施例中的智能驾驶设备还可以根据需要包括更多的组件。
请参见图4，图4是在图1所示基础上对智能驾驶设备100的结构的进一步说明。智能驾驶设备100可包括各种子系统，例如行进系统202、传感器系统204、控制系统206、一个或多个外围设备208以及电源210、计算机系统212和用户接口216。可选地，智能驾驶设备100可包括更多或更少的子系统，并且每个子系统可包括多个元件。另外，智能驾驶设备100的每个子系统和元件可以通过有线或者无线互连。
行进系统202可包括为智能驾驶设备100提供动力运动的组件。在一个实施例中，行进系统202可包括引擎218、能量源219、传动装置220和车轮/轮胎221。引擎218可以是内燃引擎、电动机、空气压缩引擎或其他类型的引擎组合，例如汽油发动机和电动机组成的混动引擎，内燃引擎和空气压缩引擎组成的混动引擎。引擎218将能量源219转换成机械能量。
能量源219的示例包括汽油、柴油、其他基于石油的燃料、丙烷、其他基于压缩气体的燃料、乙醇、太阳能电池板、电池和其他电力来源。能量源219也可以为智能驾驶设备100的其他系统提供能量。
传动装置220可以将来自引擎218的机械动力传送到车轮221。传动装置220可包括变速箱、差速器和驱动轴。在一个实施例中，传动装置220还可以包括其他器件，比如离合器。其中，驱动轴可包括可耦合到一个或多个车轮221的一个或多个轴。
传感器系统204可包括感测关于智能驾驶设备100周边的环境的信息的若干个传感器。例如，传感器系统204可包括定位系统222(定位系统可以是GPS系统，也可以是北斗系统或者其他定位系统)、惯性测量单元(inertial measurement unit，IMU)224、雷达226、激光测距仪228以及相机230。传感器系统204还可包括被监视智能驾驶设备100的内部系统的传感器(例如，车内空气质量监测器、燃油量表、机油温度表等)。来自这些传感器中的一个或多个的传感器数据可用于检测对象及其相应特性(位置、形状、方向、速度等)。这种检测和识别是自主智能驾驶设备100的安全操作的关键功能。
定位系统222可用于估计智能驾驶设备100的地理位置。IMU 224用于基于惯性加速度来感测智能驾驶设备100的位置和朝向变化。在一个实施例中，IMU 224可以是加速度计和陀螺仪的组合。
雷达226可利用无线电信号来感测智能驾驶设备100的周边环境内的物体。在一些实施例中，除了感测物体以外，雷达226还可用于感测物体的速度和/或前进方向。在一个实例中，雷达226可以是图1中的激光发射装置103。例如，智能驾驶设备100中包括一个激光雷达装置，该激光雷达装置中包括两组激光线束组，分别为第一激光线束组和第二激光线束组，其中，第一激光线束组中包括至少一个激光线束，第二激光线束组中包括至少一个激光线束；第一激光线束组在第一扫描方向探测得到第一数据；第二激光线束组在第二扫描方向探测得到第二数据；第一扫描方向和第二扫描方向不同；也即可以理解为旋转方向不同，可选的，第一扫描方向可以为顺时针方向，第二扫描方向可以为逆时针方向。第一数据和第二数据可以理解为点云信息。又例如，智能驾驶设备100中包括两个激光雷达装置，分别为激光雷达装置1和激光雷达装置2，激光雷达装置1中包括一组激光线束组，为第一激光线束组；激光雷达装置2中包括一组激光线束组，为第二激光线束组；第一激光线束组在第一扫描方向探测得到第一数据；第二激光线束组在第二扫描方向探测得到第二数据；第一扫描方向和第二扫描方向不同。
激光测距仪228可利用激光来感测智能驾驶设备100所位于的环境中的物体。在一些实 施例中，激光测距仪228可包括一个或多个激光源、激光扫描器以及一个或多个检测器，以及其他系统组件。
相机230可用于捕捉智能驾驶设备100的周边环境的多个图像。相机230可以是静态相机或视频相机。相机230的数量可以为一个或多个。相机230可以分为图1中的M个相机分组102，例如，智能驾驶设备100中的相机230的分布可以如图2或图3所示。
控制系统206为控制智能驾驶设备100及其组件的操作。控制系统206可包括各种元件，其中包括转向系统232、油门234、制动单元236、传感器融合算法238、计算机视觉系统240、路线控制系统242以及障碍物避免系统244。
转向系统232可操作来调整智能驾驶设备100的前进方向。例如在一个实施例中可以为方向盘系统。
油门234用于控制引擎218的操作速度并进而控制智能驾驶设备100的速度。
制动单元236用于控制智能驾驶设备100减速。制动单元236可使用摩擦力来减慢车轮221。在其他实施例中，制动单元236可将车轮221的动能转换为电流。制动单元236也可采取其他形式来减慢车轮221转速从而控制智能驾驶设备100的速度。
计算机视觉系统240可以操作来处理和分析由相机230捕捉的图像以便识别智能驾驶设备100周边环境中的物体和/或特征，和/或处理和分析由雷达226捕捉的数据信息。所述物体和/或特征可包括交通信号、道路边界和障碍物。计算机视觉系统240可使用物体识别算法、运动中恢复结构(structure from motion，SFM)算法、视频跟踪和其他计算机视觉技术。在一些实施例中，计算机视觉系统240可以用于为环境绘制地图、跟踪物体、估计物体的速度等等。可选的，该计算机视觉系统240可以为图1中的计算平台101。
路线控制系统242用于确定智能驾驶设备100的行驶路线。在一些实施例中，路线控制系统242可结合来自传感器融合算法238、定位系统222和一个或多个预定地图的数据以为智能驾驶设备100确定行驶路线。
障碍物避免系统244用于识别、评估和避免或者以其他方式越过智能驾驶设备100的环境中的潜在障碍物。
当然，在一个实例中，控制系统206可以增加或替换地包括除了所示出和描述的那些以外的组件。或者也可以减少一部分上述示出的组件。
智能驾驶设备100通过外围设备208与外部传感器、其他车辆、其他计算机系统或用户之间进行交互。外围设备208可包括无线通信系统246、车载电脑248、麦克风250和/或扬声器252。
在一些实施例中，外围设备208提供智能驾驶设备100的用户与用户接口216交互的手段。例如，车载电脑248可向智能驾驶设备100的用户提供信息。用户接口216还可操作车载电脑248来接收用户的输入。车载电脑248可以通过触摸屏进行操作。在其他情况中，外围设备208可提供用于智能驾驶设备100与位于车内的其它设备通信的手段。例如，麦克风250可从智能驾驶设备100的用户接收音频(例如，语音命令或其他音频输入)。类似地，扬声器252可向智能驾驶设备100的用户输出音频。
无线通信系统246可以直接地或者经由通信网络来与一个或多个设备无线通信。例如，无线通信系统246可使用3G蜂窝通信，例如CDMA、EVD0、GSM/GPRS，或者4G蜂窝通信，例如LTE，或者5G蜂窝通信。无线通信系统246可利用Wi-Fi与无线局域网(wireless local area network，WLAN)通信。在一些实施例中，无线通信系统246可利用红外链路、蓝牙或ZigBee与设备直接通信。无线通信系统246可包括一个或多个专用短程通信(dedicated  short range communications，DSRC)设备，这些设备可包括车辆和/或路边台站之间的公共和/或私有数据通信。
电源210可向智能驾驶设备100的各种组件提供电力。在一个实施例中，电源210可以为可再充电锂离子或铅酸电池。这种电池的一个或多个电池组可被配置为电源为智能驾驶设备100的各种组件提供电力。在一些实施例中，电源210和能量源219可一起实现。
智能驾驶设备100的部分或所有功能受计算机系统212控制。计算机系统212可包括至少一个处理器213，处理器213执行存储在例如存储器214这样的非暂态计算机可读介质中的指令215。计算机系统212还可以是采用分布式方式控制智能驾驶设备100的个体组件或子系统的多个计算设备。
处理器213可以是任何常规的处理器，诸如商业可获得的CPU。替选地，该处理器可以是诸如ASIC或其它基于硬件的处理器。
在此处所描述的各个方面中，处理器可以远离该车辆并且与该车辆进行无线通信。在其它方面中，此处所描述的过程中的一些在布置于车辆内的处理器上执行而其它则由远程处理器执行。
在一些实施例中，存储器214可包含指令215(例如，程序逻辑)，指令215可被处理器213执行来执行智能驾驶设备100的各种功能，包括以上描述的那些功能。存储器214也可包含额外的指令，包括向行进系统202、传感器系统204、控制系统206和外围设备208中的一个或多个发送数据、从其接收数据、与其交互和/或发送对其进行控制的指令。
除了指令215以外，存储器214还可存储数据，例如点云数据以及M个相机分组对应的M组图片数据，以及其他信息。这种信息可在智能驾驶设备100在自主、半自主和/或手动模式中操作期间被智能驾驶设备100和计算机系统212使用。
用户接口216，用于向智能驾驶设备100的用户提供信息或从其接收信息。可选地，用户接口216可包括在外围设备208的集合内的一个或多个输入/输出设备，例如无线通信系统246、车载电脑248、麦克风250和扬声器252。
计算机系统212可基于从各种子系统(例如，行进系统202、传感器系统204和控制系统206)以及从用户接口216接收的输入来控制智能驾驶设备100的功能。例如，计算机系统212可利用来自控制系统206的输入以便控制转向系统232来避免由传感器系统204和障碍物避免系统244检测到的障碍物。在一些实施例中，计算机系统212可操作来对智能驾驶设备100及其子系统的许多方面提供控制。
可选地，上述这些组件中的一个或多个可与智能驾驶设备100分开安装或关联。例如，存储器214可以部分或完全地与智能驾驶设备100分开存在。上述组件可以按有线和/或无线方式来通信地耦合在一起。
可选地，上述组件只是一个示例，实际应用中，上述各个模块中的组件有可能根据实际需要增添或者删除，图4不应理解为对本申请实施例的限制。
上述智能驾驶设备100可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、手推车、智能家居设备等，本申请实施例不做特别的限定。
可以理解的是，图4中的智能驾驶设备的结构只是本申请实施例中的一种示例性的实施方式，本申请实施例中的智能驾驶设备包括但不仅限于以上结构。
请参见图5，图5是本申请实施例提供的一种智能驾驶设备的行驶控制装置示意图，应 用于上述图4中，相当于图4所示的计算机系统212，可以包括处理器213，处理器213和系统总线505耦合。处理器213可以是一个或者多个处理器，其中每个处理器都可以包括一个或多个处理器核。存储器214可以存储相关数据信息，存储器214和系统总线505耦合。显示适配器(display adapter)507，显示适配器507可以驱动显示器509，显示器509和系统总线505耦合。系统总线505通过总线桥501和输入输出(I/O)总线513耦合。I/O接口515和I/O总线513耦合。I/O接口515和多种I/O设备进行通信，比如输入设备517(如：键盘，鼠标，触摸屏等)，多媒体盘(media tray)521,(例如，CD-ROM，多媒体接口等)。收发器523(可以发送和/或接受无线电通信信号)，摄像头555(可以捕捉景田和动态数字视频图像)和外部USB接口525。其中，可选地，和I/O接口515相连接的接口可以是USB接口。
其中，处理器213可以是任何传统处理器，包括精简指令集计算机(reduced instruction set computer，RISC)、复杂指令集计算机(complex instruction set computer，CISC)或上述的组合。可选地，处理器可以是专用集成电路ASIC。可选地，处理器213可以是神经网络处理器或者是神经网络处理器和上述传统处理器的组合。
可选地，在本文所述的各种实施例中，计算机系统212可位于远离智能驾驶设备的地方，并且可与智能驾驶设备进行无线通信。在其它方面，本文所述的一些过程在设置在智能驾驶设备内的处理器上执行，其它由远程处理器执行。
计算机系统212可以通过网络接口529和软件部署服务器549通信。网络接口529是硬件网络接口，比如，网卡。网络527可以是外部网络，比如因特网，也可以是内部网络，比如以太网或者虚拟专用网络(virtual private network，VPN)。可选地，网络527还可以是无线网络，比如Wi-Fi网络，蜂窝网络等。
收发器523(可以发送和/或接受无线电通信信号)，可以通过不限于第二代移动通信网络(2th generation mobile networks，2G)、3G、4G、5G等各种无线通信方式，也可以是DSRC技术，或者长期演进-车联网技术(long-term evolutionvehicle-to-everything，LTE-V2X)等，其主要功能是接收外部设备发送的信息数据，并将该智能驾驶设备在目标路段行驶时产生的信息数据发送回给外部设备进行存储分析。
硬盘驱动接口531和系统总线505耦合。硬盘驱动接口531和硬盘驱动器533相连接。系统内存535和系统总线505耦合。运行在系统内存535中的数据可以包括计算机系统212的操作系统537和应用程序543。
系统内存535和系统总线505耦合。例如，本申请中系统内存535可以用于将通行目标路段车辆的行驶信息按照一定格式存储在存储器中。
操作系统(operating system，OS)537包括外壳(shell)539和内核(kernel)541。外壳539是介于使用者和操作系统之内核间的一个接口。外壳539是操作系统最外面的一层。外壳539管理使用者与操作系统之间的交互：等待使用者的输入；向操作系统解释使用者的输入；并且处理各种各样的操作系统的输出结果。
内核541由操作系统中用于管理存储器、文件、外设和系统资源的那些部分组成。直接与硬件交互，操作系统内核通常运行进程，并提供进程间的通信，提供CPU时间片管理、中断、内存管理、IO管理等等。
应用程序543包括自动驾驶相关程序547，比如，管理自动驾驶的汽车和路上障碍物交互的程序，控制自动驾驶汽车路线或者速度的程序，控制自动驾驶汽车和路上其他自动驾驶汽车交互的程序。应用程序543也存在于部署服务器(deploying server)549的系统上。在一个实施例中，在需要执行自动驾驶相关程序547时，计算机系统212可以从部署服务器549 下载自动驾驶相关程序547。
传感器553和计算机系统212关联。传感器553用于探测计算机系统212周围的环境。举例来说，传感器553可以探测动物，汽车，障碍物和人行横道等，进一步传感器还可以探测上述动物，汽车，障碍物和人行横道等物体周围的环境，比如：动物周围的环境，例如，动物周围出现的其他动物，天气条件，周围环境的光亮度等。可选地，如果计算机系统212位于自动驾驶的汽车上，传感器可以是摄像头，红外线感应器，化学检测器，麦克风等。
可以理解的是，图5中的智能驾驶设备的行驶控制装置只是本申请实施例中的一种示例性的实施方式，本申请实施例中的应用于智能驾驶设备的行驶控制装置包括但不仅限于以上结构。
在目前的多传感器的融合方案中，激光雷达装置有固定的扫描周期，而相机是定频触发的，如图6所示，图6是一种智能驾驶设备中相机的位置分布示意图，该智能驾驶设备中包括一个激光雷达装置和6个相机，该6个相机分别为T1，T2，T3，T4，T5，T6。其中激光雷达装置位于智能驾驶设备的中部，且该激光雷达装置中只发射一组激光束组，该一组激光线束组包括至少一个激光束，该激光雷达装置的扫描起始位置可以为相机T1所在的位置，可以由相机T1所在的位置顺时针旋转，依次扫描一周，扫描到各个相机的时间是不同的，例如，扫描到相机T1所在的位置的时间与扫描到相机T6所在的位置的时间相差很大，相应的，得到的对应的点云数据的时间差也很大。通常的做法是将这个差值最大的起始扫描角放到后向等对精度要求较低的方向，例如，该激光雷达装置的扫描起始位置可以为相机T3所在的位置，可以由相机T3所在的位置顺时针旋转，依次扫描一周，但这只是规避措施；通过这样的方式可对点云进行补偿，但对于对向来车的场景，由于估计不准对方的速度，点云补偿的准确度较差。而针对相机的触发，在一种方案中，可以为同时触发该6个相机，也即在同一时刻可以得到6个相机对应的6个图片数据，从而触发相应的模型推理，但是会造成资源的集中占用，负载出现峰值竞争导致性能问题。在又一种方案中，可以分时触发不同的相机，也即激光雷达装置扫描到相机相应的位置时，才会触发相机曝光从而得到相应的图片数据，通过这样的方式可以达到单视角的点云和图像的最佳匹配，且在一定程度上做了负载均衡，但是由于每个相机被触发的时间差异，从而导致得到的图片数据之间存在时间差，对于高速运动场景下，无法进行有效的拼接和融合。因此，在激光雷达与相机融合的场景下，常出现由于激光雷达与相机时间戳不匹配，导致融合效果差，从而无法准确的识别车周环境的问题。为了解决上述问题，本申请实施例提出以下解决方案。
基于图1和图4提供的智能驾驶设备的结构以及图5提供的应用于智能驾驶设备的车辆行驶控制装置，本申请提供了一种数据融合方法。
请参见图7，图7是本申请实施例提供的一种数据融合方法的流程示意图，该方法可应用于上述图1中所述的智能驾驶设备中，其中计算平台可以用于支持并执行图7中所示的方法流程步骤S701-步骤S703。该方法可以应用于智能驾驶场景中，该智能驾驶包括智能辅助驾驶、自动驾驶以及无人驾驶等等，本申请实施例不做限定。该方法可以包括以下步骤S701-步骤S703。
S701、获取点云数据。
其中，点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，具体可以包括：获取第一数据和第二数据，第一数据为第一激光线 束组在第一扫描方向探测得到的，第二数据为第二激光线束组在第二扫描方向探测得到的，然后，拼接该第一数据和第二数据以获取得到点云数据。
在一个实施方式中，第一激光线束组和第二激光线束组来自同一激光发射装置，该激光发射装置可以称为双向激光雷达装置。在另一个实施方式中，第一激光线束组来自第一激光线束发射装置，第二激光线束组来自第二激光线束发射装置，也即可以理解为，第一激光线束组和第二激光线束组来自不同的激光发射装置，该第一激光线束发射装置和第二激光发射装置可以为激光雷达装置。第一激光线束组中包括至少一个激光线束，第二激光线束组包括至少一个激光线束。在一种示例中，当第一激光线束组和第二激光线束组来自同一激光发射装置时，第一激光线束组和第二激光线束组的俯视图如图8a所示，侧视图如图8b所示。在又一种示例中，当第一激光线束组和第二激光线束组来自不同的激光发射装置时，第一激光线束组和第二激光线束组的俯视图如图9a所示，侧视图如图9b所示。
其中，第一扫描方向和第二扫描方向不同可以理解为旋转方向不同，例如，第一扫描方向为顺时针方向，第二扫描方向为逆时针方向；又例如，第一扫描方向为逆时针方向，第二扫描方向为顺时针方向。
其中，点云数据可以是指部分的环境点云数据，不是完整的环境点云数据。第一数据为第一激光线束组在第一扫描方向扫描一定的扇区范围得到的数据，第二数据为第二激光线束组在第二扫描方向扫描一定的扇区范围得到的数据。可选的，第一激光线束组和第二激光线束组的扫描周期相同，也即可以理解为扫描速度相同。可选的，第一激光线束组和第二激光线束组的起始扫描位置是可以调整的。拼接第一数据和第二数据可以理解为将第一数据和第二数据旋转平移到统一的坐标系下，使它们能够组成部分的环境点云数据，也即可以理解为是将第一数据和第二数据的重叠的部分相互配准的过程。拼接可以理解为点云拼接、整合等等，本申请实施例不做限定。在一种示例中，如图10所示，图10是本申请实施例提供的一种周围环境示意图；其中，图11中的(a)是第一激光线束组在第一扫描方向探测得到第一数据的示意图，其中，以智能驾驶设备的行驶方向为基准，第一扫描方向为逆时针方向；图11中的(b)是第二激光线束组在第二扫描方向探测得到第二数据的示意图，其中，以智能驾驶设备的行驶方向为基准，第二扫描方向为顺时针方向；图11中的(c)是拼接第一数据和第二数据得到点云数据的示意图。
S702、根据M个相机分组的触发时延依次触发M个相机分组内的相机曝光以得到M组图片数据。
其中，同一个相机分组中的相机是同时曝光的。例如，一个相机分组中包括相机1和相机2，那么相机1和相机2是同时曝光的。M组图片数据中每组的图片数据和每组中的相机数量是相同的，例如，一个相机分组中包括相机1和相机2，相应的，该相机分组中有2个图片数据，也即该2个图片数据为一组图片数据，也即一个相机分组对应的图片数据。其中，M为大于等于1的正整数。M也可以为大于等于2的正整数。
其中，触发时延与第一激光线束组和第二激光线束组的扫描周期相关，扫描周期可以理解为扫描速度。触发时延是通过如下方式得到的：确定M个相机分组中每个相机分组的位置；根据扫描周期和M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。M个相机分组中每个相机的位置可以是指相机分布的空间位置，例如，相机可以分布在智能驾驶设备的前部、两侧或后部。其中，根据扫描周期和M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延，如公式(1)所示，具体如下：
Figure PCTCN2022119125-appb-000001
其中，Δ i,j为第i个相机(i∈(0,M])在第j组(j∈(0,2])激光线束扫描时的触发时延，θ i,j为第i个相机相对于在第j组激光线束的起始扫描角，按其扫描方向旋转的偏移角度，
Figure PCTCN2022119125-appb-000002
为第j组激光线束的扫描角速度。
可选的，M个相机分组包括前部分组、侧部分组和后部分组。前部分组包括第一相机和第二相机，第一相机和第二相机相对设置在智能驾驶设备的前部；侧部分组包括第三相机和第四相机，第三相机和第四相机相对设置在智能驾驶设备的两侧；后部分组包括第五相机和第六相机，第五相机和第六相机相对设置在智能驾驶设备的后部。在一种示例中，M＝3，3个相机分组分别为分组1、分组2和分组3，该智能驾驶设备包括6个相机，分别为相机1、相机2、相机3、相机4、相机5和相机6；每个相机分组中包括2个相机，具体相关描述可以参考上述图2。可选的，M个相机分组还可以包括前向分组，该前向分组包括第七相机，该第七相机朝向所述智能驾驶设备的行驶方向。在一种示例中，M＝4，4个相机分组分别为分组1、分组2、分组3和分组4，该智能驾驶设备100包括6个相机，分别为相机1、相机2、相机3、相机4、相机5、相机6和相机7；每个相机分组中包括至少1个相机，具体可以参考上述图3。
其中，M个相机分组包括第一相机分组和第二相机分组，第一相机分组的触发时延为第一时延，第二相机分组的触发时延为第二时延。根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据，包括：启动第一定时器以设置初始时刻；当所述第一定时器的指示时刻与初始时刻间隔第一时延时，触发第一相机分组内的相机曝光以得到第一相机分组对应的图片数据；当第一定时器的指示时刻与初始时刻间隔第二时延时，触发第二相机分组内的相机曝光以得到第二相机分组对应的图片数据。
其中，第一时延可以为0，也可以为其他。当第一时延为0时，可以理解为启动第一定时器时触发第一相机分组内的相机曝光以得到第一相机分组对应的图片数据。可选的，第一激光线束组和第二激光线束组可以在该初始时刻分别在第一扫描方向和第二扫描方向进行探测。当然也可以第一激光线束组和第二激光线束组也可以在该初始时刻之前或之后分别在第一扫描方向和第二扫描方向进行探测，本申请实施例不做限定。可选的，第二时延可以大于第一时延。
其中，第一相机分组对应的图片数据可以是指第一相机分组中每个相机捕捉到的图片数据，例如，第一相机分组中包括相机1和相机2，那么第一相机分组对应的图片数据可以是指相机1曝光得到的图片数据以及相机2曝光得到的图片数据。第二相机分组对应的图片数据也是同理。
其中，第一定时器的数量可以为一个或多个，当第一定时器的数量为多个时，第一定时器的数量和相机分组中相机的数量相同，例如，第一相机分组中相机的数量为2个，分别为相机1和相机2，则第一定时器的数量为2个，分别为定时器1和定时器2，其中定时器1和定时器2可以是同步的，例如，第一时延为0，启动定时器1和定时器2触发第一相机分组中相机1和相机2曝光以得到第一相机分组对应的图片数据，也即定时器1触发相机1曝光得到相机1对应的图片数据，定时器2触发相机2曝光得到相机2对应的图片数据。
在一种示例中，第一时延为0毫秒(ms)，第二时延为30ms，第一相机分组包括相机1和相机2，第二相机分组包括相机3和相机4，启动第一定时器设置初始时刻为t0，当第一定 时器的指示时刻为t0，该指示时刻t0与初始时刻t0间隔第一时延为0ms时，触发相机1和相机2曝光以得到相机1捕捉到的图片数据以及相机2捕捉到的图片数据。针对图10中的环境，如图12所示，图12中的(a)是相机1捕捉到的图片数据，也即智能驾驶设备的行驶方向的前视场中的图片数据，图12中的(b)是相机2捕捉到的图片数据，也即智能驾驶设备的行驶方向的前视场中的图片数据；当第一定时器的指示时刻为t1，该指示时刻t1与初始时刻t0间隔第二时延为30ms时，触发相机3和相机4曝光以得到相机3捕捉到的图片数据以及相机4捕捉到的图片数据,针对图10中的环境，如图13所示，图13中的(a)是相机3捕捉到的图片数据，也即智能驾驶设备的行驶方向的中视场中的图片数据，图13中的(b)是相机4捕捉到的图片数据，也即智能驾驶设备的行驶方向的中视场中的图片数据。
S703、将M个相机分组中每个相机分组对应的图片数据和点云数据进行融合以得到相机分组对应的用于感知处理的数据。
其中，将M个相机分组中每个相机分组对应的图片数据和点云数据进行融合以得到相机分组对应的用于感知处理的数据可以是指将M个相机分组中每个相机分组对应的图片数据以及点云数据作为输入数据，将该输入数据输入到融合感知算法中，例如多模态融合算法，从而得到相机分组对应的用于感知处理的数据。
例如，图11中的(c)是基于图10的环境拼接第一数据和第二数据得到的点云数据的示意图；如图12所示，图12中的(a)是第一相机分组中相机1捕捉到的图片数据，图12中的(b)是第一相机分组中相机2捕捉到的图片数据；相应的，可以将图11中的(c)的点云数据和图12中的(a)的第一相机分组中相机1捕捉到的图片数据和图12中(b)的第一相机分组中相机2捕捉到的图片数据进行融合从而得到相机分组对应的用于感知处理的数据，也即得到智能驾驶设备的行驶方向的前视场中的点云数据和图片数据的融合结果。进一步地，如图14所示，图14是图10所示的环境中智能驾驶设备的前、中以及后视场的示意图，相应的，与得到前视场中的点云数据和图片数据的融合结果的过程类似，可以得到智能驾驶设备的行驶方向的中视场中的点云数据和图片数据的融合结果，以及智能驾驶设备的行驶方向的后视场中的点云数据和图片数据的融合结果。
在一种可能的实现方式中，M个相机分组中每个相机分组之间对应的图片数据可以进行融合，也即可以理解为，M个相机分组中的至少两个相机分组对应的图片数据可以作为输入数据，将该输入数据输入到相机感知算法中，例如图像2D目标检测算法，从而得到相应的结果。例如，M＝3，该3个相机分组分别为相机分组1、相机分组2以及相机分组3，其中，每个相机分组中包括2个相机，例如，相机分组1中包括相机1和相机2，相机分组2中包括相机3和相机4，相机分组3中包括相机5和相机6。例如，可以将相机分组1中相机1的图片数据和相机2的图片数据，以及相机分组2中相机3的图片数据和相机4的图片数据输入到相机感知算法从而得到相应的结果，例如，该结果可以用于目标障碍物的检测等。
在上述方法中，以智能驾驶设备的行驶方向为基准，通过本申请，可以将该点云数据和智能驾驶设备上前视场对应的相机分组的图片数据进行融合，以提升前视场的点云数据和图片数据的融合效果，类似的，也可以提升中视场以及后视场的点云数据和图片数据的融合效果，从而使得车周环境识别更准确，更有利于自动驾驶任务规划的实施。
上述详细阐述了本申请实施例的方法，下面提供了本申请实施例的装置。
请参见图15，图15是本申请实施例提供的一种数据融合装置1500的结构示意图，该数 据融合装置1500为车载计算平台或云端服务器，该数据融合装置1500包括：通信单元1501和处理单元1502，该装置1500可以通过硬件、软件或者软硬件结合的方式来实现。
所述处理单元1502，用于获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；
所述处理单元1502，用于根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；
所述处理单元1502，用于将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
在一种可能的实现方式中，所述处理单元1502，用于获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；所述处理单元1502，用于获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；所述处理单元1502，用于拼接所述第一数据和所述第二数据以获取所述点云数据。
在又一种可能的实现方式中，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
在又一种可能的实现方式中，所述第一激光线束组来自第一激光发射装置，所述第二激光线束组来自第二激光发射装置。
在又一种可能的实现方式中，所述处理单元1502，用于确定所述M个相机分组中每个相机分组的位置；根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
在又一种可能的实现方式中，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延；所述处理单元1502，用于启动第一定时器以设置初始时刻；所述处理单元1502，用于在所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延的情况下，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据；所述处理单元1502，用于在所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延的情况下，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
在又一种可能的实现方式中，所述M个相机分组包括前部分组、侧部分组和后部分组；所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在智能驾驶设备的前部；所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
在又一种可能的实现方式中，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
各个模块的实现及有益效果还可以对应参照图7所示的方法实施例的相应描述。
请参见图16，图16是本申请实施例提供的一种智能驾驶设备1600的示意图，该智能驾驶设备1600包括计算平台1601和M个相机分组1602，所述计算平台1601用于：
获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；
根据所述M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；
将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
在一种可能的实现方式中，当获取所述点云数据时，所述计算平台1601用于：获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；拼接所述第一数据和所述第二数据以获取所述点云数据。
在又一种可能的实现方式中，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
在又一种可能的实现方式中，所述第一激光线束组来自第一激光发射装置，所述第二激光线束组来自第二激光发射装置。
在又一种可能的实现方式中，所述第一激光发射装置和所述第二激光发射装置为激光雷达装置。
在又一种可能的实现方式中，所述计算平台1601还用于：确定所述M个相机分组中每个相机分组的位置；根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
在又一种可能的实现方式中，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延，所述计算平台1601还用于：启动第一定时器以设置初始时刻；在所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延的情况下，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据。在所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延的情况下，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
在又一种可能的实现方式中，所述M个相机分组包括前部分组、侧部分组和后部分组；所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在智能驾驶设备的前部；所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
在又一种可能的实现方式中，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
各模块的实现及有益效果还可以对应参照图7所示的方法实施例的相应描述。
请参见图17，图17是本申请实施例提供的一种数据融合装置1700的示意图，该数据融合装置1700可以为车载计算平台或云端服务器，该数据融合装置1700包括至少一个处理器1701和通信接口1703，可选的，还包括存储器1702，所述处理器1701、存储器1702和通信接口1703通过总线1704相互连接。可选的，该数据融合装置1700可以为车载计算平台或云端服务器。
存储器1702包括但不限于是随机存储记忆体(random access memory，RAM)、只读存储器(read-only memory，ROM)、可擦除可编程只读存储器(erasable programmable read only memory，EPROM)、或便携式只读存储器(compact disc read-only memory，CD-ROM)，该 存储器1702用于相关计算机程序及数据。通信接口1703用于接收和发送数据。
处理器1701可以是一个或多个中央处理器(central processing unit，CPU)，在处理器1701是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
该数据融合装置1700中的处理器1701用于读取所述存储器1702中存储的计算机程序代码，执行以下操作：
获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；
根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；
将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
在一种可能的实现方式中，所述处理器1701，用于获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；所述处理器1701，用于获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；所述处理器1701，用于拼接所述第一数据和所述第二数据以获取所述点云数据。
在又一种可能的实现方式中，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
在又一种可能的实现方式中，所述第一激光线束组来自第一激光发射装置，所述第二激光线束组来自第二激光发射装置。
在又一种可能的实现方式中，所述处理器1701，用于确定所述M个相机分组中每个相机分组的位置；根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
在又一种可能的实现方式中，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延；所述处理器1701，用于启动第一定时器以设置初始时刻；所述处理器1701，用于在所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延的情况下，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据；所述处理器1701，用于在所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延的情况下，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
在又一种可能的实现方式中，所述M个相机分组包括前部分组、侧部分组和后部分组；所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在智能驾驶设备的前部；所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
在又一种可能的实现方式中，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
各个操作的实现及有益效果还可以对应参照图7所示的方法实施例的相应描述。
可以理解的是，本申请的实施例中的处理器可以是中央处理单元(central processing unit，CPU)，还可以是其它通用处理器、数字信号处理器(digital signal processor，DSP)、专用集 成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)或者其它可编程逻辑器件、晶体管逻辑器件，硬件部件或者其任意组合。通用处理器可以是微处理器，也可以是任何常规的处理器。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘；还可以是半导体介质，例如，固态硬盘。该计算机可读存储介质可以是易失性或非易失性存储介质，或可包括易失性和非易失性两种类型的存储介质。
在本申请的各个实施例中，如果没有特殊说明以及逻辑冲突，不同的实施例之间的术语和/或描述具有一致性、且可以相互引用，不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。

Claims (19)

  1. 一种数据融合方法，其特征在于，应用于智能驾驶场景中，所述方法包括：
    获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；
    根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；
    将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
  2. 根据权利要求1所述的方法，其特征在于，所述获取点云数据包括：
    获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；
    获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；
    拼接所述第一数据和所述第二数据以获取所述点云数据。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
  4. 根据权利要求1或2所述的方法，其特征在于，所述第一激光线束组来自第一激光发射装置，所述第二激光线束组来自第二激光发射装置。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述方法还包括：
    确定所述M个相机分组中每个相机分组的位置；
    根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延；
    所述根据M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据，包括：
    启动第一定时器以设置初始时刻；
    当所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延时，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据；
    当所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延时，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
  7. 根据权利要求1-6中任一项所述的方法，其特征在于，所述M个相机分组包括前部分组、侧部分组和后部分组；
    所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在智能驾驶设备的前部；
    所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；
    所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
  8. 根据权利要求7所述的方法，其特征在于，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
  9. 一种智能驾驶设备，其特征在于，包括计算平台和M个相机分组，所述计算平台用于：
    获取点云数据，所述点云数据是根据第一激光线束组和第二激光线束组分别在第一扫描方向和第二扫描方向进行探测得到的，其中，所述第一扫描方向和所述第二扫描方向不同；
    根据所述M个相机分组的触发时延依次触发所述M个相机分组内的相机曝光以得到M组图片数据；其中，同一个相机分组中的相机是同时曝光的，所述触发时延与所述第一激光线束组和第二激光线束组的扫描周期相关，M为大于等于1的正整数；
    将所述M个相机分组中每个相机分组对应的图片数据和所述点云数据进行融合以得到所述相机分组对应的用于感知处理的数据。
  10. 根据权利要求9所述的智能驾驶设备，其特征在于，当获取所述点云数据时，所述计算平台用于：
    获取第一数据，所述第一数据为所述第一激光线束组在所述第一扫描方向探测得到的；
    获取第二数据，所述第二数据为所述第二激光线束组在所述第二扫描方向探测得到的；
    拼接所述第一数据和所述第二数据以获取所述点云数据。
  11. 根据权利要求9或10所述的智能驾驶设备，其特征在于，所述第一激光线束组和所述第二激光线束组来自同一激光发射装置。
  12. 根据权利要求9或10所述的智能驾驶设备，其特征在于，所述智能驾驶设备还包括：第一激光发射装置和第二激光发射装置；
    所述第一激光线束组来自所述第一激光发射装置，所述第二激光线束组来自所述第二激光发射装置。
  13. 根据权利要求12所述的智能驾驶设备，其特征在于，所述第一激光发射装置和所述第二激光发射装置为激光雷达装置。
  14. 根据权利要求9-13中任一项所述的智能驾驶设备，其特征在于，所述计算平台还用于：
    确定所述M个相机分组中每个相机分组的位置；
    根据所述扫描周期和所述M个相机分组中每个相机分组的位置确定所述每个相机分组的触发时延。
  15. 根据权利要求9-14中任一项所述的智能驾驶设备，其特征在于，所述M个相机分组包括第一相机分组和第二相机分组，所述第一相机分组的触发时延为第一时延，所述第二相机分组的触发时延为第二时延，所述计算平台还用于：
    启动第一定时器以设置初始时刻；
    在所述第一定时器的指示时刻与所述初始时刻间隔所述第一时延的情况下，触发所述第一相机分组内的相机曝光以得到所述第一相机分组对应的图片数据；
    在所述第一定时器的指示时刻与所述初始时刻间隔所述第二时延的情况下，触发所述第二相机分组内的相机曝光以得到所述第二相机分组对应的图片数据。
  16. 根据权利要求9-15中任一项所述的智能驾驶设备，其特征在于，所述M个相机分组包括前部分组、侧部分组和后部分组；
    所述前部分组包括第一相机和第二相机，所述第一相机和所述第二相机相对设置在所述智能驾驶设备的前部；
    所述侧部分组包括第三相机和第四相机，所述第三相机和所述第四相机相对设置在所述智能驾驶设备的两侧；
    所述后部分组包括第五相机和第六相机，所述第五相机和所述第六相机相对设置在所述智能驾驶设备的后部。
  17. 根据权利要求16所述的智能驾驶设备，其特征在于，所述M个相机分组还包括前向分组，所述前向分组包括第七相机，所述第七相机朝向所述智能驾驶设备的行驶方向。
  18. 一种数据融合装置，其特征在于，包括处理器和存储器，所述存储器和所述处理器通过线路互联，所述处理器用于获取所述存储器中存储的计算机程序，当所述计算机程序被所述处理器执行时，所述数据融合装置用于实现权利要求1-8中任一项所述的方法。
  19. 根据权利要求18所述的数据融合装置，其特征在于，所述数据融合装置为车载计算平台或云端服务器。
PCT/CN2022/119125 2022-09-15 2022-09-15 一种数据融合方法、装置及智能驾驶设备 WO2024055252A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/119125 WO2024055252A1 (zh) 2022-09-15 2022-09-15 一种数据融合方法、装置及智能驾驶设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/119125 WO2024055252A1 (zh) 2022-09-15 2022-09-15 一种数据融合方法、装置及智能驾驶设备

Publications (1)

Publication Number Publication Date
WO2024055252A1 true WO2024055252A1 (zh) 2024-03-21

Family

ID=90273939

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/119125 WO2024055252A1 (zh) 2022-09-15 2022-09-15 一种数据融合方法、装置及智能驾驶设备

Country Status (1)

Country Link
WO (1) WO2024055252A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108957478A (zh) * 2018-07-23 2018-12-07 上海禾赛光电科技有限公司 多传感器同步采样系统及其控制方法、车辆
CN109858512A (zh) * 2018-12-10 2019-06-07 北京百度网讯科技有限公司 点云数据的处理方法、装置、设备、车辆及存储介质
CN112291024A (zh) * 2019-07-25 2021-01-29 北京地平线机器人技术研发有限公司 信息同步方法、信息同步装置及电子设备
CN113138393A (zh) * 2020-01-17 2021-07-20 阿里巴巴集团控股有限公司 环境感测系统、控制装置以及环境感测数据融合装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108957478A (zh) * 2018-07-23 2018-12-07 上海禾赛光电科技有限公司 多传感器同步采样系统及其控制方法、车辆
CN109858512A (zh) * 2018-12-10 2019-06-07 北京百度网讯科技有限公司 点云数据的处理方法、装置、设备、车辆及存储介质
CN112291024A (zh) * 2019-07-25 2021-01-29 北京地平线机器人技术研发有限公司 信息同步方法、信息同步装置及电子设备
CN113138393A (zh) * 2020-01-17 2021-07-20 阿里巴巴集团控股有限公司 环境感测系统、控制装置以及环境感测数据融合装置

Similar Documents

Publication Publication Date Title
WO2021027568A1 (zh) 障碍物避让方法及装置
WO2022027304A1 (zh) 一种自动驾驶车辆的测试方法及装置
WO2022001773A1 (zh) 轨迹预测方法及装置
US20220215639A1 (en) Data Presentation Method and Terminal Device
WO2021212379A1 (zh) 车道线检测方法及装置
US12001517B2 (en) Positioning method and apparatus
WO2021189210A1 (zh) 一种车辆换道方法及相关设备
EP4206731A1 (en) Target tracking method and device
US20240017719A1 (en) Mapping method and apparatus, vehicle, readable storage medium, and chip
US20230048680A1 (en) Method and apparatus for passing through barrier gate crossbar by vehicle
JP2024504990A (ja) 軌道予測方法及び装置、及び地図
CN112810603B (zh) 定位方法和相关产品
US20220309806A1 (en) Road structure detection method and apparatus
WO2022052881A1 (zh) 一种构建地图的方法及计算设备
WO2022062582A1 (zh) 控制摄像头模组补光时间的方法及装置
WO2021217646A1 (zh) 检测车辆可通行区域的方法及装置
CN115056784B (zh) 车辆控制方法、装置、车辆、存储介质及芯片
CN115221151B (zh) 车辆数据的传输方法、装置、车辆、存储介质及芯片
CN115203457B (zh) 图像检索方法、装置、车辆、存储介质及芯片
WO2024055252A1 (zh) 一种数据融合方法、装置及智能驾驶设备
WO2022068643A1 (zh) 多任务部署的方法及装置
WO2022022284A1 (zh) 目标物的感知方法及装置
WO2021159397A1 (zh) 车辆可行驶区域的检测方法以及检测装置
CN115334109A (zh) 用于交通信号识别的系统架构、传输方法，车辆，介质及芯片
CN115334111A (zh) 用于车道识别的系统架构、传输方法，车辆，介质及芯片

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22958444

Country of ref document: EP

Kind code of ref document: A1