CN211656247U

CN211656247U - Multi-sensor data synchronization system

Info

Publication number: CN211656247U
Application number: CN201821978384.4U
Authority: CN
Inventors: 邱小波
Original assignee: Huizhou Desay SV Intelligent Transport Technology Research Institute Co Ltd
Current assignee: Huizhou Desay SV Intelligent Transport Technology Research Institute Co Ltd
Priority date: 2018-11-29
Filing date: 2018-11-29
Publication date: 2020-10-09
Anticipated expiration: 2028-11-29

Abstract

The application provides a multi-sensor data synchronization system, which comprises a host and a plurality of sensors connected with the host; the host is used for acquiring data input by the plurality of sensors and the image acquisition device; a plurality of sensors including a first type of sensor, a second type of sensor, and a third type of sensor; the first type of sensor is connected with a vehicle body CAN bus, the second type of sensor is connected with a private CAN bus, the third type of sensor comprises a signal end and a data end, the signal end is connected with the vehicle body CAN bus and used for outputting frame signals, and the data end is connected with a host through a data line and outputs data to the host; the vehicle body CAN bus and the private CAN bus are respectively connected with the host through the same data converter so as to synchronize data input to the host. The embodiment of the application can greatly reduce the delay generated in the data transmission process, improve the instantaneity of the data and enable the collected data to realize the unification of time marks.

Description

Multi-sensor data synchronization system

Technical Field

The application relates to the field of data acquisition, in particular to a multi-sensor data synchronization system.

Background

In the field of automatic driving, vehicles are equipped with various sensors to sense and detect complex road environments, such as cameras, millimeter-wave radars, ultrasonic probes, and the like.

The deep learning method is adopted to research the automatic driving algorithm model, massive real road condition data are needed to train the algorithm model, and massive data are needed for product testing and simulation no matter the algorithm research and development and the algorithm improvement need massive data.

Under the existing vehicle conditions, data generated by each sensor is transmitted to a unified ECU (Electronic Control Unit) through a CAN bus protocol for processing, or is provided with each ECU for processing. However, the data transmission modes and interfaces of the sensors may be different, and the external host and the sensors cannot be directly connected and communicated. Meanwhile, each sensor is not necessarily provided with a clock system, even if a clock system of a single sensor is difficult to perform relatively accurate time comparison on the clock systems of different sensors, each item of data acquired by the host computer has no time mark, or because the data has large time delay, and the recorded time has errors with the actual time. The data of a plurality of sensors cannot be unified on time marks, so that the difficulty of subsequent data analysis performed by researchers is increased.

SUMMERY OF THE UTILITY MODEL

The application provides a multisensor data synchronization system, can make the data that the sensor gathered realize time synchronization.

The application provides a multisensor data synchronization system, includes:

the system comprises a host, an image acquisition device and a plurality of sensors, wherein the image acquisition device and the plurality of sensors are respectively connected with the host;

the host is used for acquiring data input by the plurality of sensors and the image acquisition device;

the plurality of sensors comprise a first type of sensor, a second type of sensor and a third type of sensor;

the first type of sensor is connected with a vehicle body CAN bus, the second type of sensor is connected with a private CAN bus, the third type of sensor comprises a signal end and a data end, the signal end is connected with the vehicle body CAN bus and used for outputting frame signals, and the data end is connected with the host through a data line and outputs data to the host; the vehicle body CAN bus and the private CAN bus are respectively connected with the host through the same data converter so as to synchronize data input to the host;

the average data volume generated by the first sensor, the second sensor and the third sensor in the working period is respectively a first data volume, a second data volume and a third data volume, and the first data volume, the second data volume and the third data volume are sequentially increased.

Optionally, the data converter is configured to converge data sent by the vehicle body CAN bus and the private CAN bus, and convert the converged data from data meeting a CAN protocol to data meeting an ethernet protocol; and converting the data of the host from the data meeting the Ethernet protocol into the data meeting the CAN protocol, and sending the data to the corresponding vehicle body CAN bus or the private CAN bus.

Optionally, the third type of sensor is an image acquisition device, and the image acquisition device includes a camera module and a signal processing module that can be connected to each other for communication;

the camera module is connected with the data end, and the signal processing module is connected with the vehicle body CAN bus through a signal end and used for generating and outputting the frame signal to the vehicle body CAN bus through the signal end.

Optionally, the data line is provided with an adapter, and the adapter is used for switching between the interface of the data end and the interface of the host.

Optionally, the adaptor is used for transferring between an HDMI interface and a USB interface;

the adapter HDMI interface is connected with the output end of the camera module, and the USB interface is connected with the host.

Optionally, the first type of sensor includes one or more of a vehicle speed sensor, a steering wheel angle sensor, a turn light sensor, an accelerator opening sensor, and a brake sensor.

Optionally, the second type of sensor includes a millimeter wave radar.

The application also discloses an image data synchronization method applied to the multi-sensor data synchronization system of any one of claims 1 to 7, the method comprising:

obtaining a frame image, and obtaining a frame identifier corresponding to the frame image according to the frame image;

generating a frame signal according to the frame identifier, wherein the frame signal carries the frame identifier;

outputting the frame signal to a vehicle body CAN bus so as to send the frame signal to a host through the vehicle body CAN bus and enable the frame signal to realize time synchronization with other data on the vehicle body CAN bus;

and sending the frame image to a host through a data line.

Optionally, the obtaining, according to the frame image, a frame identifier corresponding to the frame image includes:

obtaining a frame number corresponding to the frame image according to the frame image, wherein the frame number is related to the acquisition time sequence of the frame image;

and acquiring a corresponding frame identifier based on the frame sequence number.

Optionally, the sending the frame image to the host through the data line includes:

associating the frame image with the frame identifier to obtain an associated frame image;

and sending the associated frame image to a host through a data line.

Optionally, the associating the frame image with the frame identifier to obtain an associated frame image includes:

and storing the information of the frame identification into a preset area of the frame image.

Optionally, the storing the information of the frame identifier in a preset area of the frame image includes:

obtaining a target color value corresponding to the information of the frame identifier;

obtaining a target pixel in a preset area of the frame image;

and switching the color value of the target pixel into the target color value.

Optionally, the method further includes:

acquiring a time node of a data stream of the frame signal on the CAN bus of the vehicle body;

obtaining a target frame image associated with a frame identifier carried by the frame signal;

and associating the target frame image with the time node corresponding to the frame signal.

As CAN be seen from the above, in the multi-sensor data synchronization system and the image data synchronization method in the embodiments of the present application, the host and the multiple sensors are connected in a manner that data is transmitted through multiple data transmission channels, so that delay of data generated during transmission CAN be greatly reduced, instantaneity of data is improved, signals of the multiple sensors are respectively connected to the multiple CAN buses, and data of the multiple CAN buses are aggregated by the data converter and then transmitted to the host, so that the collected data CAN be unified in time stamp.

Drawings

Fig. 1 is a schematic structural diagram of a multi-sensor data synchronization system according to an embodiment of the present disclosure.

Fig. 2 is another schematic structural diagram of a multi-sensor data synchronization system according to an embodiment of the present disclosure.

Detailed Description

The following detailed description of the preferred embodiments of the present application, taken in conjunction with the accompanying drawings, will make the advantages and features of the present application more readily appreciated by those skilled in the art, and thus will more clearly define the scope of the invention.

Referring to fig. 1, a structure of a multi-sensor data synchronization system according to an embodiment of the present application is shown.

The multi-sensor data synchronization system can be used for an on-board electronic device, which can acquire environmental parameters of an automobile through a plurality of sensors 20.

The multi-sensor data synchronization system includes a host 10 and a plurality of sensors 20.

Specifically, the host 10 is configured to collect data input by a plurality of sensors 20. The sensor 20 may include a pressure sensor, an acceleration sensor, an ultrasonic probe, an image sensor, a millimeter wave radar, etc., and may also be other sensors for acquiring environmental parameters, which are not exhaustive here. The host 10 may be a pc (personal computer), or may be other data acquisition devices, such as a server, a vehicle-mounted computer, and the like.

The plurality of sensors 20 include a first type sensor 21, a second type sensor 22, and a third type sensor 23, average data amounts generated by the first type sensor 21, the second type sensor 22, and the third type sensor 23 during an operation period are a first data amount, a second data amount, and a third data amount, respectively, and the first data amount, the second data amount, and the third data amount are sequentially increased. Wherein, the signal ends of the first type sensor 21 and the third type sensor 23 are connected with a vehicle body CAN bus 31; the second type of sensor 22 is connected to a private CAN bus 32; the body CAN bus 31 and the private CAN bus 32 are connected to the host computer 10 through the same data converter 40, respectively, so that data input to the host computer are synchronized.

In some embodiments, the first type of sensor 21, which may be the sensor 20 generating a small amount of data after each data collection operation, may include one or more of a vehicle speed sensor 20, a steering wheel angle sensor 20, a turn signal sensor 20, an accelerator opening sensor 20, and a brake sensor 20. After the plurality of first type sensors 21 are connected to the vehicle body CAN bus 31, the data of the first type sensors 21 CAN be directly transmitted to the host computer 10 through the vehicle body CAN bus 31 due to the small data volume, and data congestion is avoided.

In some embodiments, the second type of sensor 22 may be the sensor 20 that generates a larger amount of data each time a data collection operation is completed. In this embodiment, the second type sensor 22 may include a millimeter wave radar, which may avoid data congestion of the body CAN bus 31 through the private CAN bus 32 due to a large amount of generated data.

Of course, the maximum transmission rate of the conventional CAN bus 30 is 1Mbit/s, and other sensors 20 with transmission rates satisfying the transmission conditions of the CAN bus 30 CAN be used besides the millimeter wave radar. The private CAN bus 32 may have one or more, specific numbers and types of sensors 20 attached thereto.

The third type sensor 23 includes a signal terminal and a data terminal, the data terminal is connected to the host 10 through a data line and outputs data to the host 10, and the signal terminal is used for outputting a frame signal. The third sensor 23 may be an image acquisition device, which includes a camera and a signal processing module connected to the camera for communication and used for acquiring data. The camera can be a CMOS camera or a CCD camera, and the specific type and model of the camera can be adjusted according to practical application.

It will be appreciated that in addition to the image capture means, the third type of sensor may be other sensors with larger data volumes, particularly sensors that generate data traffic outside the bandwidth limits of a conventional CAN bus, such as sensing devices like array radar.

If the third type of sensor is an image capturing device, the frame signal is a signal corresponding to a frame of image generated when the image capturing device captures the frame of image. That is, one frame image corresponds to one frame signal, and the number of frames of the image corresponds to the number of frame signals.

In some embodiments, the data line may be a dedicated serial/parallel data line, such as an HDMI data line, a USB data line, a DVI data line, or an AV data line, and the like, and the specific type is not limited in this application. The image acquisition device is connected with the host computer 10 through a data line, so that images or images with large data volume acquired by the image acquisition device CAN be directly transmitted with the host computer 10 without occupying limited CAN bus 30 resources, and the transmission efficiency is improved.

Specifically, after the frame image acquired by the image acquiring apparatus is transmitted to the host 10, the signal terminal transmits a corresponding frame signal, and the time stamp transmitted by the frame signal on the CAN bus 30 may be used to determine the transmission time of the frame signal. Then, the sending time of the frame signal is associated with the frame image, so that the host 10 CAN know the specific sending time of the frame image through the time stamp of the frame signal, thereby unifying the time stamp of the frame image and other data sent through the CAN bus 30.

In addition, in order to obtain a specific time stamp of each data, the host computer 10 may connect the vehicle body CAN bus 31 and the other private CAN buses 32 through the data converter 40, so that the vehicle body CAN bus 31 and the other private CAN buses 32 summarize data into the same line through the data converter 40, and the CAN protocol is used to realize that the input data all have corresponding time stamps, so that the data on the plurality of different CAN buses 30 are unified in time stamp.

In some embodiments, the data converter 40 may include a plurality of CAN ports, with different CAN buses 30 connected to the CAN ports of the data converter 40 through their own CAN ports. The data converter 40 CAN convert the data meeting the CAN protocol into a data form readable by the host 10 through an internal data conversion function, and transmit the data to the host 10, so that the host 10 CAN conveniently receive and store the data.

As CAN be seen from the above, in the multi-sensor data synchronization system in the embodiment of the present application, the host and the plurality of sensors are connected in a manner that data is transmitted through the plurality of data transmission channels, so that delay of data generated in the transmission process CAN be greatly reduced, and instantaneity of data is improved, signals of the plurality of sensors are respectively accessed to the plurality of CAN buses, and data transmission is performed with the host after data of the plurality of CAN buses are aggregated by the data converter, so that time stamp unification of the acquired data CAN be realized.

Referring to fig. 2, another structure of the multi-sensor data synchronization system according to the embodiment of the present application is shown.

As shown in fig. 2, in one embodiment, the multi-sensor data synchronization system includes a host 10, an image acquisition device 23, and an Electronic Control Unit (ECU).

The host 10 is a PC 10, and the PC 10 may include a processor, a memory, and an interface connected to the processor and the memory, such as a USB interface, a network card interface connected to an Ethernet (Ethernet), and the like. The PC 10 is used to acquire sensor data and image data of a multi-sensor data synchronization system.

The image capturing device 23 includes a camera module 231 and a signal processing module 232, and the image capturing device 23 further includes an SOC for data acquisition and transmission. The camera module 231 includes a camera and a related peripheral circuit, and is configured to perform sensitization on external ambient light to obtain a frame image. The SOC is used to process the frame image and transmit the frame image to the PC 10 through the data line.

Here, since the PC 10 and the image capturing device 23 may not be directly connected due to the interface problem, an adapter for switching between the interface of the data terminal of the image capturing device 23 and the interface of the PC 10 may be provided on the data line. Specifically, in this embodiment, the adaptor is an HDMI-to-USB device 50, which is used for switching between an HDMI interface and a USB interface, the HDMI interface of the adaptor is connected to the output terminal of the camera module 231, that is, the output terminal of the SOC, and the USB interface is connected to the PC. Of course, the adapter can determine the type of interface to be switched according to practical situations, such as switching between HDMI interface, USB interface, DVI interface, or AV interface.

The signal processing module of the image acquisition device 23 is connected to the vehicle body CAN bus 31, and transmits a corresponding frame signal to the vehicle body CAN bus 31, and the transmission time of the frame signal CAN be determined by using the time stamp transmitted by the frame signal on the CAN bus 30. Then, the sending time of the frame signal is associated with the frame image, so that the host 10 CAN know the specific sending time of the frame image through the time stamp of the frame signal, thereby unifying the time stamp of the frame image and other data sent through the CAN bus 30.

The ECU may include one or more combinations of a vehicle speed sensor 20, a steering wheel angle sensor 20, a turn signal sensor 20, an accelerator opening sensor 20 and a brake sensor 20, and the sensors 20 in the ECU may be connected to a vehicle body CAN bus 31 and transmit sensor data by using the vehicle body CAN bus 31.

It is understood that, in addition to the sensor 20 in the ECU, other sensors 20 provided in the system, such as the acceleration sensor 20, the pressure sensor 20, and the like, may be connected to the vehicle body CAN bus 31, and these sensors are not illustrated here.

The sensor 20 may further include a millimeter wave radar 22, and the millimeter wave radar 22 is accessed to the private CAN bus 32 because the generated data amount is large, so as to independently transmit the data of the millimeter wave radar 22 through the private CAN bus 32, avoid collision between the data of the millimeter wave radar 22 and data of other sensors, and effectively reduce data delay. In addition to millimeter-wave radar 22, other sensors 20 that may generate large amounts of data may utilize a private CAN bus 32 for data transmission.

The data of the car body CAN bus 31 and the data of the private CAN bus 32 both need to be connected with the PC 10, and in order to provide better compatibility and ensure the transmission speed of the data, the car body CAN bus 31 and the private CAN bus 32 CAN be connected with the PC 10 through the data converter 40, that is, the CAN-to-Ethernet device 40, so that the data of the CAN buses are summarized by the data converter 40 and then transmitted to the PC 10 through the Ethernet. Specifically, the data converter 40 is configured to converge data sent by the body CAN bus 31 and the private CAN bus 32, and convert the converged data from data meeting the CAN protocol to data meeting the ethernet protocol; or converts the data of the host computer 10 from data satisfying the ethernet protocol to data satisfying the CAN protocol, and transmits the data to the corresponding body CAN bus 31 or the private CAN bus 32.

Data of each CAN bus CAN realize data collection through data converter 40, and the unified CAN protocol that utilizes carries out the time mark to it, has solved the problem of the time mark is nonuniform because of adopting many CAN bus transmission data to this data converter 40 CAN carry out the network deployment through ethernet, need not to worry the not enough problem of CAN port, and expansibility is strong.

The Ethernet CAN have a data transmission rate of 100Mbit/s or more, CAN ensure that the delay of the data transmission process of a plurality of CAN buses is maintained at a lower level, and the Ethernet is communicated with the PC 10 through protocols such as TCP, UDP and the like and CAN be connected with the PC 10 through a network cable connector, so that the PC 10 CAN be in data communication with the CAN buses without a specific interface, and the compatibility is higher.

Therefore, the multi-sensor data synchronization system is more reliable in data transmission, low in delay, compatible with different hosts, capable of summarizing and unifying various data by using the data adapter, and capable of solving the problem of non-uniformity of time marks caused by non-uniformity of data sources acquired by the hosts in the prior art.

The embodiments of the present application have been described in detail with reference to the drawings, but the present application is not limited to the above embodiments, and various changes can be made without departing from the spirit of the present application within the knowledge of those skilled in the art.

Claims

1. A multi-sensor data synchronization system, comprising:

2. The multi-sensor data synchronization system according to claim 1, wherein the data converter is configured to merge data sent by the body CAN bus and the private CAN bus, and convert the merged data from data satisfying a CAN protocol to data satisfying an ethernet protocol; and converting the data of the host from the data meeting the Ethernet protocol into the data meeting the CAN protocol, and sending the data to the corresponding vehicle body CAN bus or the private CAN bus.

3. The multi-sensor data synchronization system of claim 1, wherein the third type of sensor is an image acquisition device, the image acquisition device comprising a camera module and a signal processing module that are connectable to communicate with each other;

4. The multi-sensor data synchronization system of claim 3, wherein the data line is provided with a switch for switching between the interface of the data terminal and the interface of the host.

5. The multi-sensor data synchronization system of claim 4, wherein the adapter is for interfacing between an HDMI interface and a USB interface;

6. The multi-sensor data synchronization system of claim 1, wherein the first type of sensor comprises one or more of a vehicle speed sensor, a steering wheel angle sensor, a turn signal sensor, a throttle opening sensor, and a brake sensor.

7. The multi-sensor data synchronization system of claim 1, wherein the second type of sensor comprises a millimeter wave radar.