WO2019184709A1 - 多传感器融合的数据处理方法、装置与多传感器融合方法 - Google Patents

多传感器融合的数据处理方法、装置与多传感器融合方法 Download PDF

Info

Publication number
WO2019184709A1
WO2019184709A1 PCT/CN2019/078001 CN2019078001W WO2019184709A1 WO 2019184709 A1 WO2019184709 A1 WO 2019184709A1 CN 2019078001 W CN2019078001 W CN 2019078001W WO 2019184709 A1 WO2019184709 A1 WO 2019184709A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
matrix
pixel
detection
sensor
Prior art date
Application number
PCT/CN2019/078001
Other languages
English (en)
French (fr)
Inventor
蒋宏
Original Assignee
上海智瞳通科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海智瞳通科技有限公司 filed Critical 上海智瞳通科技有限公司
Priority to EP19776961.5A priority Critical patent/EP3779867A4/en
Priority to US17/040,191 priority patent/US11675068B2/en
Publication of WO2019184709A1 publication Critical patent/WO2019184709A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/865—Combination of radar systems with lidar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2465—Query processing support for facilitating data mining operations in structured databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/251—Fusion techniques of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G06T5/60—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/862—Combination of radar systems with sonar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/411—Identification of targets based on measurements of radar reflectivity
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/417—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section involving the use of neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Definitions

  • the present invention relates to the field of data processing of electronic devices, and in particular, to a multi-sensor fusion data processing method, apparatus and multi-sensor fusion method.
  • the sensors currently used for environmental sensing and target detection may include: image acquisition sensors such as cameras, microwave radars, infrared sensors, ultrasonic radars, and laser radars, etc., which are widely used in vehicle driving assistance systems (ADAS) and Autopilot systems, robots, automated guided vehicles (AGVs), smart homes, smart security, and a variety of devices and systems that require environmental awareness and target detection capabilities.
  • image acquisition sensors such as cameras, microwave radars, infrared sensors, ultrasonic radars, and laser radars, etc.
  • ADAS vehicle driving assistance systems
  • AGVs automated guided vehicles
  • smart homes smart security
  • a variety of devices and systems that require environmental awareness and target detection capabilities.
  • the image acquisition sensor can sense the texture (shape, outline, light illumination, etc.) and color of the target, and record the image information for a moment.
  • the camera can also record video information, string the recorded events with a timeline to form a video stream, which can be used for event playback and time-related event analysis.
  • the infrared sensor is a kind of image acquisition sensor that captures the infrared radiation information of the target and saves it in the format of pictures and videos.
  • Microwave radar (or collectively referred to as radar) can capture the relative distance of the target, the relative motion speed, the radar cross section RCS data of the target, and the heat map, the relative distance of the target, the relative motion speed, the radar cross section RCS data dimension of the target.
  • the laser radar mainly outputs the point cloud data of the target by detecting the spatial position (relative distance, spatial angular position coordinate information) of the target.
  • Various sensors have their own information-aware dimensions, such as our commonly used camera, which captures the image information of the target, and vividly records the texture and color information of the environment and the target at the moment, but we may not be able to get from a single image. In accurately extracting the distance and speed information of the target, it is difficult for us to predict from a traditional photo what will happen next time.
  • the data detected by each sensor are independent of each other, and lack of deep fusion. Further, when the detected data is used for feature extraction and data mining, the ability of environment sensing and target detection is weak.
  • the invention provides a multi-sensor fusion data processing method, device and multi-sensor fusion method to solve the problem that the sensor detected data lacks deep fusion.
  • a data processing method for multi-sensor fusion comprising:
  • Forming a multi-dimensional matrix structure specifically, for example, forming a multi-dimensional matrix structure of an array of "multi-dimensional measurement parameters" matrix (also referred to as a multi-dimensional pixel matrix); wherein:
  • the multi-dimensional matrix structure includes a plurality of matrix layers distributed in a longitudinal direction, the plurality of matrix layers including at least one pixel matrix layer and at least one sensing matrix layer, and each pixel matrix layer correspondingly is used to represent a pixel data matrix.
  • Each of the sensing matrix layers is configured to represent a set of sounding data elements, wherein the sounding data elements in the sounding data set vertically correspond to pixel elements in the pixel matrix layer; the values of the sounding data elements are Determined based on the value of the probe data.
  • a data processing apparatus for multi-sensor fusion including:
  • An acquiring module configured to acquire image data of the target object and at least one set of detection data groups; the image data is detected by an image acquisition sensor, the detection data set is detected by other sensors; and the image data is used for utilizing At least one pixel data matrix is used to represent the target image collected by the image acquisition sensor; different detection data sets are detection data of different detection dimensions;
  • the multi-dimensional matrix structure includes a plurality of matrix layers distributed in a longitudinal direction, the plurality of matrix layers including at least one pixel matrix layer and at least one sensing matrix layer, and each pixel matrix layer correspondingly is used to represent a pixel data matrix.
  • Each of the sensing matrix layers is configured to represent a set of sounding data elements, wherein the sounding data elements in the sounding data group correspond to pixel elements in the pixel matrix layer; the values of the sounding data elements are all based on The probe data assignment is determined.
  • a multi-sensor fusion method comprising:
  • the information contained in each pixel is longitudinally expanded, and in addition to the brightness and color information originally contained therein, a plurality of vertical dimensions are added for each pixel, and the pixel can be input in the increased vertical dimension.
  • the camera detects probe information of a plurality of corresponding dimensions detected by the target object of the spatial map by the other sensors, and the probe information includes at least one of the following: a relative distance, a relative motion speed, a target radar cross section RCS data, and a target heat.
  • the multi-dimensional detection information is assembled in a layered manner onto the target object description originally based on the image pixels, and a multi-dimensional pixel is generated, which is mathematically represented as a matrix array of a unified structure.
  • a multi-dimensional pixel matrix is obtained.
  • a processing apparatus including a memory and a processor is provided;
  • the memory is configured to store sensing data, intermediate running data, system output data, and executable instructions of the processor
  • the processor is configured to perform the method of the first aspect and its alternatives, or the method of the third aspect and its alternatives, by executing the executable instructions.
  • a sensing device including a memory, a processor and a sensor is provided;
  • the memory is configured to store sensing data, intermediate running data, system output data, and executable instructions of the processor
  • the processor is configured to perform the method of the first aspect and its alternatives, or the method of the third aspect and its alternatives, by executing the executable instructions.
  • a storage medium having stored thereon a program, wherein the program is executed by a processor to implement the method of the first aspect and its alternatives, or the third aspect and The alternatives involve methods that store both perceptual data, intermediate operational data, and system output data.
  • the multi-sensor fusion data processing method, device and multi-sensor fusion method provided by the invention can calculate data of a plurality of different dimensions measured by different sensors on the basis of pixel elements, and adopt various data according to the spatial data sampling model of the imaging. Mapping, performing unified data alignment and merging, forming a multi-dimensional matrix structure of an array of "multi-dimensional measurement parameters" matrix on the data structure, and combining them in the form of a multi-dimensional matrix structure, thereby performing multi-layered data on the acquired data. Fusion and deep learning, which can achieve more effective data mining and feature extraction, resulting in more effective environment awareness and target detection capabilities.
  • FIG. 1 is a schematic flow chart of a data processing method for multi-sensor fusion in Embodiment 1 of the present invention
  • Embodiment 1 of the present invention is a schematic diagram showing the principle of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention
  • FIG. 3 is a schematic diagram showing another principle of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention.
  • FIG. 4 is a schematic diagram showing still another principle of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention.
  • FIG. 5 is another schematic flowchart of a data processing method for multi-sensor fusion in Embodiment 1 of the present invention.
  • FIG. 6 is a schematic flowchart of establishing a mapping relationship in Embodiment 1 of the present invention.
  • step S16 is a schematic flowchart of step S16 in Embodiment 1 of the present invention.
  • Figure 9 is a schematic view showing projection in an embodiment of the present invention.
  • Figure 10 is a schematic view of an image forming surface in an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of a data processing apparatus for multi-sensor fusion in Embodiment 1 of the present invention.
  • FIG. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
  • Figure 13 is a block diagram showing the structure of a sensing device in accordance with an embodiment of the present invention.
  • Embodiment 1 is a schematic flow chart of a data processing method for multi-sensor fusion in Embodiment 1 of the present invention.
  • the data processing method of multi-sensor fusion including:
  • S11 Acquire image data of the target object and at least one set of detection data sets.
  • the processing method may specifically include a process of collecting processing, may also be included in the process of storing processing, and may also include a process of intermediate processing before storage, that is, any one of data collection, intermediate processing, and storage.
  • the method involved in the present embodiment and its alternatives can be understood as implementing the processing method.
  • the methods involved in the present embodiment and its alternatives in the device for collecting data, the device for processing data, and the device for storing data can also be understood as the implementation of the processing method. That is, it does not depart from the scope of the description of the invention and its alternatives.
  • the image data can be understood as being detected by the image acquisition sensor.
  • the image acquisition sensor can be any device capable of image acquisition, for example, a camera, a mobile phone, a tablet computer, a computer, or the like with a built-in camera.
  • the installation parameters of the image acquisition sensor, image acquisition, storage format, etc. can be predetermined.
  • the image data is specifically configured to represent the target image acquired by the image acquisition sensor by using at least one pixel data matrix.
  • the pixel data matrix can vary depending on the image acquisition sensor.
  • the RGB image can be represented by three layers of data, specifically red, green, Blue three-layer data, each color data can be represented by a pixel data matrix; if the image is YUV, where Y can represent brightness Luma, that is, gray scale value, U and V can represent chrominance and color difference information, wherein Each layer of data can be characterized by an image data matrix.
  • the pixel data matrix having only the brightness Y may be used instead of the image data matrix of U and V, and the pixel data matrix having only at least one of R, G, and B may be further included. In an example, only one of the U, V pixel data matrices may be used.
  • the target image can be characterized by RGB or YUV three-layer data matrix, or the data can be encapsulated into one layer, that is, a layer of data matrix is used to represent the content of the three layers, and a unit of the multi-bit data unit can also be used.
  • the multi-bit data can be, for example, 24 bit, 32 bit data or even more bits of data.
  • only a monochrome camera can participate in the fusion.
  • the camera as an image acquisition sensor can be used only for infrared imaging scenes, and the camera only outputs a monochrome image; in this case, the above-mentioned multidimensional
  • the pixel structure can still be effective.
  • the RGB three-layer or YUV three-layer can be changed to a single-layer brightness Y data structure, that is, an image data matrix having only brightness Y, which can be utilized on the basis of single-layer brightness.
  • Each step involved in each of the alternative embodiments incorporates data from other sensors to produce a multi-dimensional matrix structure.
  • the monochrome camera can be a non-color camera, specifically an infrared camera, and further, can directly capture the gray value of the image.
  • the other target dimension information that the same target object can be captured by other sensors is assembled in a layered manner onto the pixel matrix layer of the image data captured by the image acquisition sensor. Further, the pixels of the image can be used later. As a basis for the combination of each probe data, the data is one-to-one corresponding to the pixels.
  • the camera as an image acquisition sensor can arrange the collected image data in three layers according to the RGB color in an unrestricted order to obtain a corresponding image data matrix
  • the camera resolution is X*Y (for example: 1920*1080, which can correspond to a 1080P resolution camera; in another example, if the original data input is in the YUV format, the corresponding image data matrix can also be obtained according to the YUV three-layer arrangement, and in another example, Converting the YUV format to the RGB format can help reduce the association of data between layers and facilitate subsequent independent feature extraction.
  • the detection data set can be understood as any data group detected by other sensors than the image acquisition sensor. Each data group can be understood as corresponding to one layer of the sensing matrix layer, and one sensor can generate one or more sets of detection data sets. .
  • Different detection data sets are detection data of different detection dimensions, and different detection dimensions can be understood as: detection data having different physical meanings, which can also be understood as: in the same detection dimension, regardless of its spatial position
  • the difference is in the value itself, not the difference in its physical meaning.
  • the physical units of the probe data in the same probe dimension are generally the same, ie, in the example, if the two probe data have different physical units, they typically belong to different probe dimensions. At the same time, this embodiment does not exclude exceptions.
  • one other sensor may obtain multiple probe data sets for multiple probe dimensions, or one probe data set for one probe dimension.
  • the other sensors are at least one of the following: microwave radar, ultrasonic radar, laser radar, and infrared sensor.
  • the infrared sensor can be used to detect the heat radiation temperature data, etc.
  • the microwave radar can detect the target distance, the relative speed, the radar scattering cross section RCS (Radar-Cross Section) data, etc., based on the distance between the objects and other mapping relationships, the specific implementation manner
  • the vector velocity, acceleration, orientation, etc. can be further calculated. Therefore, the probe data may include at least one of the following: distance data, speed data, acceleration data, orientation data, reflection characteristic data for microwaves, and heat radiation temperature data, etc., and may further be distance data for the target object in the detection domain. Variations, changes in velocity data, changes in acceleration data, changes in orientation data, RCS data, and characterization of changes in thermal radiation temperature data.
  • the distance data mentioned above, as well as the speed data, acceleration data, azimuth data and the like obtained based on the distance data, can also be detected by the radar.
  • the detection data of the other sensors radar scattering cross section RCS data; wherein the RCS, specifically Radar-Cross Section. Further, it can be understood as a representation of a change in RCS data of a target object in the detection domain.
  • the detection data referred to above whether it is the detection data of other sensors or the detection data of the image acquisition sensor such as RGB data, may be directly detected by the sensor, or may be directly detected by the sensor and then indirectly. Calculated.
  • the probe data may also be intermediate data of the sensor when directly generating the probe data, for example, optical flow data of a camera as an image acquisition sensor.
  • the optical flow expresses the change of the image. Since it contains the information of the target motion, it can be used by the observer to determine the motion of the target.
  • the optical flow data is a parameter for deriving the pixel relationship between successive image frames of the camera as an image acquisition sensor, and may be a two-dimensional vector (X, Y direction). Therefore, in this embodiment, the current pixel corresponding to the optical stream data of the previous frame can be added.
  • the image data matrix can add an optical flow data matrix, thereby further adding more data dimensions to the data organization of the system. For subsequent data integration processing.
  • the intermediate data referred to above may also be, for example, a calculated three-dimensional vector velocity, that is, vector data of the motion velocity, and the like.
  • a calculated three-dimensional vector velocity that is, vector data of the motion velocity, and the like.
  • it is necessary to accurately record the moving speed of the target object.
  • it can be characterized by vector speed instead of relative speed.
  • the system to calculate the target vector speed data.
  • the velocity data may correspond to the three-layer sensing matrix layer, respectively corresponding to X, Y, Z
  • the multi-dimensional matrix structure according to the present embodiment can also record the vector value of the moving speed of the target object.
  • the velocity data referred to above may be a vector component of the motion velocity or a velocity value.
  • any data that can be two-dimensionally distributed by the sensor can be understood as the above-mentioned probe data, that is, without departing from the scope described in this embodiment.
  • the image capturing sensor of the camera may be a sensor that can sense the texture (shape, contour, light illumination, etc.) and color of the target object, and record image information for a moment.
  • the camera can also record video information, string the recorded events with a timeline to form a video stream, which can be used for event playback and time-related event analysis.
  • the infrared sensor involved in the above is also understood as an image acquisition sensor. Therefore, the infrared sensor can be used as an image acquisition sensor as described above, or can be used as another sensor to capture a target. Infrared radiation information is saved in the form of pictures and videos.
  • the microwave radar involved above can capture the relative distance of the target, the relative motion velocity, the radar scattering cross section RCS data of the target, and quantify the heat map, the relative distance of the target object, the relative motion speed, and the radar cross section RCS data dimension of the target. Expression, or point cloud data output.
  • the above-mentioned laser radar mainly outputs the point cloud data of the target by detecting the spatial position (relative distance, spatial angular position coordinate information) of the target object.
  • the laser radar usually outputs a point cloud data structure: gray pixel point cloud data (X1, Y1, Z1, gray value 1), or color point cloud data (X1, Y1, Z1, r1, g1, b1), It is also possible to combine the data of the 4 layers or 6 layers to this target point and map the spatial sample projection model of the image captured by the optical camera to the corresponding position of the matrix layer.
  • gray pixel point cloud data X1, Y1, Z1, gray value 1
  • color point cloud data X1, Y1, Z1, r1, g1, b1
  • various sensors have their own information-aware dimensions, such as cameras commonly used in the field, which can capture the image information of the target, and vividly record the texture and color information of the environment and the target at the moment, but may not be able to It is difficult to accurately extract the distance and speed information of the target from the picture, and it is difficult to predict what will happen next time from a traditional photo. But the way video comes with huge amounts of data, and the resulting large transmission bandwidth and space for storing data.
  • Other sensor recording methods such as radar, ultrasonic, and lidar, can record information about the sensor's own perceived dimensions, such as the distance and speed of the target; the data information they record and the current recording method (data structure) for us.
  • a comprehensive description of the required target characteristics can be directly used for environmental perception and event prediction, and the dimensional and completeness of the data is insufficient.
  • the method involved in the embodiment can perform efficient multi-layer data fusion on various collected data, and fully utilize information combinations from different dimensions to support more effective information sampling and storage for target feature extraction and data. analysis.
  • the time stamping time of the structure can also be added to the structure, which can facilitate the collection and storage of multi-dimensional information data of the environment and the target.
  • a multi-dimensional matrix structure to include a plurality of matrix layers distributed longitudinally, the plurality of matrix layers including at least one pixel matrix layer and at least one sensing matrix layer.
  • Each of the sounding data elements in the sounding data group corresponds to a pixel element in the pixel matrix layer, which may be one-to-one correspondence, or may be one-to-many or many-to-one, visible, single detection data.
  • the matrix may correspond vertically to a single pixel element or vertically to a range of regions containing a plurality of pixel elements.
  • the method involved in the embodiment increases the perceived depth of the system, and establishes a multi-dimensional multi-dimensional depth-aware array structure with pixels as granularity, that is, a multi-dimensional matrix structure.
  • this embodiment can expand each pixel in the related art to each multi-dimensional data structure, that is, the detection data of different dimensions are combined in the form of a matrix array to form a multi-dimensional matrix. structure.
  • the multi-dimensional matrix structure can be used to represent the mapping result of each probe data and pixel, and the mapping result of the probe data and the pixel, such as a semantic description, a table, and the like, can also be characterized by other description manners.
  • FIG. 2 is a schematic diagram showing the principle of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention
  • FIG. 3 is another schematic diagram of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention
  • FIG. 4 is a multi-dimensional matrix structure formed in Embodiment 1 of the present invention
  • Another schematic diagram of the principle is a schematic diagram showing the principle of forming a multi-dimensional matrix structure in Embodiment 1 of the present invention.
  • each pixel matrix layer is correspondingly used to represent a pixel data matrix, wherein the pixel matrix layer may be a pixel data matrix itself, such as the RGB three-layer or YUV three-layer, etc., as described above, to FIG. 2 to
  • the embodiment shown in FIG. 4 is an example, which may be an RGB three-layer pixel matrix layer, which can be specifically understood as the leftmost three-layer data structure in FIG. 3 and FIG. 4, and can also be specifically understood as the topmost in FIG. The three-tier data structure.
  • the pixel matrix layer may also be other matrix layers generated according to the pixel data matrix, and may be, for example, other matrix layers generated by interpolating or converting the pixel data matrix.
  • FIG. 4 specifically use a combination of an image acquisition sensor + a radar + an infrared sensor for illustration, which is merely an example combination.
  • the alternative of the embodiment may further add other sensors, or remove some sensors. The principle of action is the same.
  • each of the sensing matrix layers is used to represent a set of sounding data sets, and the values of the sounding data elements are all determined according to the sounding data assignment values.
  • the data elements of each position of each matrix layer in the multi-dimensional matrix structure can also be characterized by the cross-sectional coordinates (x, y) of the multi-dimensional matrix structure. It can be seen that the vertical direction involved in the context can be understood as the distribution direction between the matrix layers, and the horizontal direction involved in the context can be understood as the distribution direction of each element in a single matrix.
  • the sensing matrix layer of the L layer can be understood as a sensing matrix layer that characterizes the distance data detected by the radar.
  • the sensing matrix layer of the S layer can be understood as characterizing the radar.
  • the sensing matrix layer of the detected velocity data wherein the sensing matrix layer of the R layer can be understood as a sensing matrix layer for characterizing the RCS data involved above, wherein the sensing matrix layer of the H layer can be understood as an infrared sensor Sensing matrix layer of detected thermal radiation temperature data.
  • the value of the probe data element is the corresponding probe data itself, and it can be understood that the value of the probe data element is determined by directly assigning the corresponding probe data.
  • the value of the probe data element may also be determined according to the corresponding probe data conversion. Specifically, it may be understood that the value of the probe data element may also be determined by using a value determined by the conversion.
  • the sensing matrix layer of the R layer can input the corresponding value of the RCS mapping to the radar receiving power P.
  • RCS(x, y) refers to the RCS value corresponding to the cross-sectional coordinate position of (x, y);
  • L(x, y) is the relative distance value detected by the radar to the target object.
  • copying can be performed using the converted P(x, y).
  • the value determined by the conversion may be determined by a single detection data conversion, or may be determined by using a plurality of different types of detection data conversion.
  • the value assigned to each data element in F1(L) can be understood as a function-converted value of the distance data L detected by the radar; among them, F2(S)
  • the value assigned to each data element can be understood as a function-converted value of the velocity data S detected by the radar; the value assigned to each data element in F3(R) can be understood as detected by the radar.
  • the value of the RCS data is converted by the function; the value assigned to each data element in F4 (H) can be understood as a function-converted value of the thermal radiation temperature data detected by the infrared sensor.
  • the assignments mentioned above are mainly for the assignment of the target object part in the detection domain, that is, the assignment itself can also be characterized as the detection result of the target object. It can also be seen that when applying the multi-dimensional matrix structure involved in the embodiment and its alternatives, it can be used for target detection and context awareness associated therewith.
  • FIG. 5 is a schematic flow chart of a data processing method for multi-sensor fusion according to another embodiment of the present invention.
  • the method may include:
  • S121 Determine, according to the established mapping relationship, a pixel element of the target object corresponding to each probe data of the target object.
  • the process of determining the pixel elements of the target object may be a process of obtaining target pixel information, which may be understood to be used to represent pixels corresponding to the probe data.
  • target pixel information which may be understood to be used to represent pixels corresponding to the probe data.
  • the cross-sectional coordinates such as (x, y) characterized above, may be utilized to characterize the target pixel information, and any predetermined identification information may be utilized to characterize the target pixel information.
  • the pixel matrix layer is a matrix of pixel data, and the pixel data matrix itself is used to represent the pixels therein, when the multi-dimensional matrix structure is formed in this embodiment, only the mapping relationship is used to determine the corresponding pixel element.
  • the mapping relationship can be understood as a mapping relationship between the probe data and different pixel elements used to characterize different locations of different detection dimensions in the detection domain of the other sensors.
  • the mapping relationship can be characterized by an arbitrary data structure, for example, in the form of a table, or in any sentence form.
  • the mapping relationship is specifically used to represent the correspondence between the probe data of different locations of different detection dimensions in the detection domain of the sensor and different single pixel elements.
  • FIG. 6 is a schematic flow chart of establishing a mapping relationship according to an embodiment of the present invention
  • FIG. 9 is a schematic diagram of projection in an embodiment of the present invention
  • FIG. 10 is a schematic diagram of an imaging surface according to an embodiment of the present invention.
  • the process of establishing the mapping relationship includes:
  • the entrance center 27 can be understood with reference to the conventional principle of an image acquisition sensor such as a camera, which can coincide with the detection apex 28, which can also be understood as the detection origin of the sensor.
  • the detection domain mapping of the multi-sensor combination is unified and converted to the three-dimensional detection space of the image acquisition sensor such as the camera through the spatial coordinate transformation, thereby facilitating the detection direction and detection of the image acquisition sensor.
  • Space is used to achieve projection, and then to establish the required mapping relationship.
  • Variations therein may include, for example, translation, rotation, and scaling of the coordinate system.
  • the detection coordinate system may be a Cartesian coordinate system, and thus may have mutually perpendicular X-axis, Y-axis, and Z-axis. Further, geometric space conversion may be used in a standard three-dimensional Euclidean geometric space.
  • the detection domains of each sensor (the physical space region of the data detection acquisition) are associated, and the translation, rotation and scaling of the respective coordinate systems are unified into the coordinates of the 3D detection space of the camera, that is, through the above step S13 Establish a unified detection space for the system and a common detection perspective.
  • the alternative embodiment of the present embodiment can use the spatial parameters of the sensor installation to set the central axis (X'/Y' of each detection domain (the respective detection space) of each sensor.
  • the /Z' axis is calculated, and the above step S13 is implemented based on this.
  • the optional embodiment of the present embodiment can also detect the central axis of each sensor's detection domain in combination with the geometric calibration process, in order to calibrate the central axes.
  • the target is placed at a position that determines the coordinates of the spatial position, and the position readings of the targets measured by the sensor are used to correlate the target position readings with their physical physical position in reality to establish the sensor detection.
  • the detection coordinate system may also be a spherical coordinate system. That is, we can also use the spherical coordinate system to correlate the detection domains of each sensor to establish a one-to-one correspondence.
  • ⁇ can be understood as the distance from the axis of the target distance coordinate system
  • ⁇ can be understood as the angle of the target relative to the z-axis
  • ⁇ can be understood as the angle of the target with respect to the x-axis.
  • the spherical coordinate origin can be, for example, the center of the entrance mentioned in the foregoing. Therefore, the mapping relationship can be understood as a spatial mapping relationship of the target object corresponding to the spherical coordinate origin in the radial direction.
  • the European solid geometric space coordinates and the spherical coordinates can be converted to each other, and the conversion relationship can be characterized as follows:
  • the detection domain 21 of the image acquisition sensor and the detection domain of other sensors can be expressed in FIG. 7 by using a positional relationship in a reference plane.
  • the detection domain 22 can be It is understood to be the detection domain of the infrared sensor, which can be embodied as an infrared thermal imager, the corresponding detection data of which can be, for example, thermal radiation temperature data; wherein the detection domain 23 can be understood as the detection domain of the radar.
  • the detection domain 21 of the image acquisition sensor can be projected on the imaging surface to form an imaging area 24, that is, a projection area of the image acquisition sensor, or can be characterized as an imaging area; the detection domain of other sensors is projected. Thereafter, a projection area can be projected on the imaging surface 24, for example, the detection field 22 of the infrared sensor can be projected to obtain the projection area 26, and the detection field 23 of the radar can be projected to obtain the projection area 25.
  • Each of the projection regions can be in the imaging region 24, and in other alternative embodiments, the present embodiment does not exclude that the projection region can be partially outside of the imaging region 24.
  • S15 Determine the mapping relationship according to a positional relationship between the projection area in the two-dimensional plane and an imaging area of the image acquisition sensor.
  • the mapping may be determined according to a projection relationship between a projection area of the sensor and a current detection domain, and a positional relationship between the projection area in the two-dimensional plane and an imaging area of the image acquisition sensor. relationship.
  • the detection domain can be unified into the same plane by the above-mentioned variation and projection, thereby facilitating the characterization and determination of the mapping relationship between the location points.
  • the mapping relationship may be determined according to the correspondence between the positions of the detection domain before and after the change, that is, according to the projection area of the sensor.
  • the mapping relationship between the current detection domain, the correspondence between the detection domain before and after the change, and the positional relationship between the projection area and the imaging region of the image acquisition sensor in the two-dimensional plane are determined.
  • the method may include: S122: assigning the probe data to a probe data element corresponding to a vertical direction of a pixel element corresponding thereto.
  • FIG. 7 is a schematic flowchart of steps S121 and S123 in Embodiment 1 of the present invention.
  • step S121 If the target pixel information is obtained in the foregoing step S121, after step S121, please refer to FIG. 7, which may include:
  • S123 Determine, according to the target pixel information, a corresponding target probe data element in a sensing matrix layer corresponding to the probe data.
  • the target probe data element can be characterized, for example, as a particular (x,y) coordinate in a particular sensor matrix layer.
  • the step S122 may specifically include: assigning a target detection data element corresponding to the target detection data element according to the detection data.
  • the detection result of the target object can be expressed by the result of the assignment, which can be further adapted to be recognized.
  • FIG. 8 is a schematic flow chart of step S16 in Embodiment 1 of the present invention.
  • the algorithm when the mapping relationship is established, the algorithm may be used to finely adjust the geometric spatial mapping relationship to reduce the geometric mapping positioning error.
  • step S16 may be implemented, including:
  • S161 performing positioning calculation on any one or more reference target objects detected in the detection domain of the other sensors to obtain a target positioning result; the target positioning result is used to represent that the reference target object is in the detection space. position;
  • S162 Mapping a location represented by the target positioning result to a pixel matrix layer, to obtain verification positioning information
  • S163 Align the verification positioning information with the original positioning information to obtain positioning error information, where the original positioning information is used to represent that the pixel element determined when forming the multi-dimensional matrix structure is the same for the same reference target object.
  • S164 Adjust the corresponding sensing matrix layer according to the positioning error information, so that a longitudinal correspondence relationship between the sensing matrix layer and the pixel matrix layer changes;
  • S165 Adjust an assignment of the probe data element in the sensing matrix layer according to the changed vertical correspondence.
  • the plurality of detected target objects in the detection domain are independently positioned and calculated in the respective sensor detection space regions, and the final positioning result is matched into the multi-dimensional matrix structure, and the matching result is matched with
  • the detection results of the same target object are compared, and the position error of the same target object is determined.
  • the error can be calculated by using geometric space conversion, and further, the corresponding sensing matrix layer and the pixel matrix can be scaled and translated.
  • the relative position of the layer that is, adjusting the vertical correspondence between the data elements in the sensing matrix layer and the data elements in the pixel matrix layer, can also be understood as adjusting the correspondence between the data elements and the pixels in the sensing matrix layer, thereby reducing the error and at the same time
  • the mapping relationship determined by the foregoing steps S13 to S15 is adjusted accordingly.
  • the above detection can be performed again in the sensing matrix layer by directly detecting the positioning of the target object and directly mapping the positioning detection result to the corresponding sensing matrix layer.
  • the target object should be consistent with the target object expressed by the assignment in step S12 by using the pre-established mapping relationship.
  • the above-mentioned errors can be characterized by comparing the differences between the two, thereby effectively detecting the target object. The results are adjusted and the established mappings can be adjusted.
  • each sensor should be as close to coaxial as possible on the spatial structure, and the closer it is, the better, which is beneficial to reduce the mapping error introduced by the geometric space conversion of different axes (the result is similar to the virtual image).
  • the resolution of the other sensor may be different at the initial input, that is, the resolution of the other sensor does not match the resolution of the image acquisition sensor, and is optional in this embodiment.
  • the resolution may be matched to ensure that the detection data elements in each sensing matrix layer can correspond to the image data elements in the pixel matrix layer.
  • the above resolutions are inconsistent. It can be understood that the probe data of other sensors are distributed in the same size range after being projected onto the two-dimensional plane of the imaging surface, and the distribution of the rows and columns and the image of the image acquisition sensor in the array of pixels in the imaging plane Inconsistent, for example, at least one of the number of rows and the number of columns may be inconsistent, and may also refer to processing of the detected data by conversion, calculation, etc., and after being projected onto the imaging plane, the distribution of the data in the same size range and the pixels in the imaging plane The ranks are inconsistent.
  • the macroblock Macroblock may be defined by a pixel plane, and the mapping relationship is specifically used to represent the detection data of different positions in the detection domain of the sensor and different macroblocks.
  • the macroblock contains at least two pixels.
  • the data can be matched by the mapping data of the one-to-one mapping relationship with the data detected by the low-resolution sensor after the screen is divided into a plurality of pre-defined macroblocks.
  • the specific definition parameters of the macroblock need to be described in the data organization, for example, in the data file header or description.
  • the method may also be implemented by interpolation, for example, the data elements in the sensing matrix layer further include a first interpolation data element, and/or: the pixel matrix layer
  • the data element in the file also includes a second interpolated data element.
  • the values of the interpolated data elements are understood to be interpolated.
  • the radar detection data is converted into dense image data with a tensor structure, which can be used as a detection data group, and then the geometric detection method is used to project the detection domain of the radar to
  • the two-dimensional plane of the imaging surface which can be used as the projection surface of the target object of the radar detection; the resolution can be equal to the pixel resolution of the image acquisition sensor matched in the system, and the point-to-point correspondence between the radar data and the camera data is established.
  • the target object detected by the radar is projected onto the projection surface to generate a radar sensing matrix structure of the radar detection data;
  • the data layer of the radar sensing matrix may include, for example, an L (target distance value) layer, S (relative speed value) Layer, R (radar cross-section value) layer; similarly, the order of these layers can be interactive or flexible combination.
  • L, S, and R can all be activated, or only one or two of them can be activated.
  • one or more sensing matrix layers can be formed; at present, the spatial resolution of the millimeter wave radar is relatively low, and the angular resolution of the target is not high.
  • the numerical projection has a relatively large possible coverage area on the two-dimensional mapping surface of the target.
  • the original pixel particle size similar to the radar is larger than the pixel size of the image acquisition sensor, and the resolution is low, for each layer of each multi-dimensional pixel.
  • the data elements in the sensing matrix layer are assigned corresponding values.
  • Each data element in the sense matrix layer is assigned one by one, so the matched sensor matrix layer can have interpolated data elements; since the radar data is sparse, the data structure in the radar (L layer, S layer, R layer, etc.) In the radar detection area (equivalent pixel position), the radar data will be assigned in one-to-one correspondence, but in areas where no target is detected, the original data of the radar corresponding to these areas can be assigned: “0”, or set to the default value representing the background according to the prior agreement, to ensure each data in the sensing matrix layer corresponding to the radar data The elements are all assigned.
  • the image it captures is also in pixels.
  • the image resolution can be enlarged by appropriate interpolation, and then the resolution of the camera is matched, and the image collected by the infrared thermal imager (generally black and white brightness pixel information) is assigned to the corresponding sensing matrix layer point by point.
  • the resolution of an infrared thermal imager is lower than that of a normal camera.
  • this embodiment does not exclude the case where the resolution of other sensors is higher than that of the image acquisition sensor.
  • the image resolution of the infrared thermal imager is higher than the resolution of the camera installed in the system, in which case the pixel matrix layer is interpolated.
  • the solution can also be processed by reducing the resolution of the thermal imager; in short, the basic principle is to make the resolution of the two sensors the same and then assign the values of the data elements of the corresponding matrix layer of the multi-dimensional matrix structure.
  • Data elements in the pixel matrix layer can also be interpolated for similar reasons.
  • nearest neighbor interpolation for example, bilinear interpolation, cubic convolution, and the like can be used.
  • the matrix layer in the multi-dimensional matrix structure can be selectively activated.
  • the activation therein can be understood as that the matrix layer can be used only when activated, and the data of the matrix layer is iterated by the update.
  • the matrix layer may be written in a pre-programmed program, and thus may be activated or not activated.
  • the matrix layer may also be added after it has not been written in advance, or it may be automatically generated according to a predefined rule. It can be seen that the presence or absence of the matrix layer and whether it is activated can be distinguished.
  • the other sensors are at least one of: microwave radar, ultrasonic radar, laser radar, infrared sensor, and terahertz image sensor;
  • the detection data of the other sensors includes at least one of the following: distance Data, velocity data, acceleration data, azimuth data, radar cross section RCS data, and thermal radiation temperature data;
  • the image data matrix includes at least one of the following: a luminance data matrix, an RGB three-layer data matrix, and a YUV three-layer data matrix.
  • an optical flow data matrix is at least one of: microwave radar, ultrasonic radar, laser radar, infrared sensor, and terahertz image sensor;
  • the detection data of the other sensors includes at least one of the following: distance Data, velocity data, acceleration data, azimuth data, radar cross section RCS data, and thermal radiation temperature data;
  • the image data matrix includes at least one of the following: a luminance data matrix, an RGB three-layer data matrix, and a YUV three-layer data matrix.
  • an optical flow data matrix is at least
  • the applied sensor, the matrix layer present, and the activated matrix layer can all vary.
  • a camera can be used as an image acquisition sensor, and a microwave radar and an infrared thermal imager are respectively used as other sensors, and the camera outputs a color image (RGB or YUV data), and the microwave radar outputs distance data and relative speed of the target object.
  • Data such as azimuth orientation data and RCS data of the target object
  • the infrared thermal imager outputs a thermal radiation temperature distribution image of the target, wherein each pixel can be understood as thermal radiation temperature data.
  • the combination of the examples can sense and detect targets from multiple dimensions, and can work effectively under a variety of conditions (day, night, fog, rain and other harsh environments).
  • a variety of sensors can be flexibly combined and combined, for example, all three can be used (camera + microwave radar + infrared thermal imager), or a combination of two or two: camera plus microwave radar, camera plus infrared thermal imager Or a combination of microwave radar and infrared thermal imager.
  • the system can dynamically adjust the dimensions of the sensor input parameters for target recognition according to the hardware configuration or the scene (during different conditions such as daytime and nighttime). That is, the activated matrix layer is adjusted and detected using a subset of multi-dimensional pixels. For example, in the night driving, we need to detect the target outside the illumination range of the vehicle.
  • the flexible combination can also have the following positive effects: in one scenario, one of the sensors of the system fails (or is damaged), and the method according to the embodiment and the system implementing the method can be maintained to operate effectively. , to enhance the robustness of the system operation. In some application areas, such as ADAS or autopilot applications, it is necessary to increase the robustness of the system.
  • the number of similar sensors can also be flexibly configured, for example, multiple cameras, multiple radars, and multiple thermal imagers can be used.
  • multiple cameras multiple radars
  • multiple thermal imagers can be used.
  • the detection parameters brought by the new sensor can also be attached to our "camera + mm
  • the multi-dimensional matrix structure of the wave radar + infrared thermal imager system has become a part of our multidimensional.
  • the system can have a higher target detection rate and better recognition ability and recognition quality.
  • the marked Class A area is a region jointly detected by three types of sensors
  • the marked Class B area and the Class C area are areas jointly detected by the radar and the camera
  • the labeled Class D area and the Class E area are indicated. It is the area that the camera and the infrared thermal imager jointly detect
  • the G type area is the area that only the camera detects.
  • the A-type area is a common exploration area of three kinds of sensors. This area can make full use of the accurate detection of multi-data dimensions brought by multi-sensor fusion. Therefore, sensor fusion in this area is more important.
  • the area where the camera and radar overlap detection, such as the B-type area and the C-type area, the camera and radar detection dimensions are highly complementary, and the fusion of these two sensors is also important.
  • Another important fusion area is the area where the camera overlaps with the infrared thermal imager.
  • the infrared thermal imager can make up for the lack of the camera in nighttime, foggy conditions, etc.; Information, the fusion of information between them, more need the process of resolution matching mentioned above, for example, the image resolution (pixel number) of the infrared sensor needs to be enlarged by image interpolation, or the resolution of the camera image is reduced, To achieve mutual matching, then combine the image of the infrared sensor (we mark it as H dimension) with the RGB (or YUV) of the camera to form the desired multidimensional matrix structure.
  • the optional embodiment of the embodiment may use various traditional feature values + classifiers to analyze and process each layer matrix layer to detect the target object; or use neural network to perform subsequent detection processing. Or a mixture of two methods. In either case, since the multi-dimensional detection information is unified into a data structure and mapped in units of pixels, such deep fusion data combination can effectively improve the detection quality of the target object.
  • the network because the input is a multi-dimensional matrix structure, the network will generate more layers of feature maps accordingly, and with richer multi-layer and multi-dimensional feature extraction, it is more efficient and high quality.
  • Multi-dimensional matrix matching algorithm is better.
  • Currently popular neural network algorithms such as R-CNN, Faster R-CNN, SSD, etc., can match the application applicable to the multi-dimensional matrix structure involved in this embodiment.
  • the matrix description manner of the multi-dimensional measurement parameters involved in the embodiment can facilitate the sample collection and training of the target object; the information of each layer matrix layer is relatively independent, and the structure of each layer can be increased or decreased, and subsequent processing It can dynamically activate one layer or multiple layers (the information of these layers participates in the determination and description of the target) or inactive (the information of these layers does not participate in the determination and description of the target), but generally does not hinder the multi-dimensional
  • the pixel structure is used to detect and mark the target.
  • the multi-sensor is used to identify the target from multiple detection dimensions.
  • the multi-dimensional matrix structure can organically combine the multi-dimensionally-perceived detection data, and the subsequent data processing (whether using the traditional eigenvalue plus classifier method or the nerve)
  • the method of the network, or a combination of the two, and the sampling of the training samples bring great convenience.
  • Such data assembly is also very effective for spatial navigation and positioning (such as SLAM).
  • the description of the target object using the structure not only has feature data (for classification and recognition), but also has three-dimensional position and space information (XY).
  • XY three-dimensional position and space information
  • the spatial angle of the axis and the distance from the detector to the target), the result of the target recognition can be directly used for spatial positioning.
  • the combination of information from different dimensions produces more effective data mining and feature extraction potential, which not only provides a more efficient event recording method (format), but also effectively improves system environment perception and
  • the target detection capability can also greatly save the bandwidth required for transmitting data and the space for storing data, and facilitate the transmission of effective and sufficient data streams to different systems, while reducing the amount of data that the system (processing unit) performs event prediction and analysis. With the requirements of real-time computing processing capabilities, the cost of the environment-aware system is effectively reduced.
  • the method can include the relative distance and velocity information of the target in the multi-dimensional matrix structure, the method can be used to describe the intention of the target and predict the scenario where the event will occur in the data corresponding to one frame of pixels.
  • ADAS vehicle driving assistance systems
  • AGVs unmanned vehicles
  • Equipment and systems can be applied to vehicle driving assistance systems
  • ADAS vehicle driving assistance systems
  • robots robots
  • unmanned vehicles AGVs
  • Equipment and systems can be applied to vehicle driving assistance systems
  • robots robots
  • unmanned vehicles AGVs
  • various environmental sensing and target detection capabilities Equipment and systems.
  • the method involved in the present application is applied to its learning, since machine learning requires a huge sample set, these sample sets also need to be tailored for self-sensor combination system, and the data organization method of the present invention can be very Targeted multi-sensor system for multi-dimensional and efficient data samples is well-suited for multi-sensor fusion sample sets (positive and negative sample sets and semantic sample sets), and saves storage space.
  • the machine learning system involves "cloud” + “end” or “edge computing"
  • the system needs to sample the data to be transmitted locally and in the cloud.
  • the method involved in this alternative can achieve more efficient transmission of multi-dimensional information while avoiding The necessary redundant data occupies the data transmission bandwidth; in addition, in some specific areas (such as security, monitoring, insurance forensics) applications, the system requires as much storage space as possible to store as much information as possible, the data sampling of the present invention
  • the storage method can store multi-dimensional target information on a frame of data matrix (which can include information such as target distance and running speed vector), which can greatly improve the preservation efficiency of forensic information.
  • Figure 11 is a block diagram showing the structure of a multi-sensor fusion data processing apparatus in Embodiment 1 of the present invention.
  • a multi-sensor fusion data processing apparatus 300 includes:
  • the acquiring module 301 is configured to acquire image data of the target object and at least one set of detection data groups; the image data is detected by an image acquisition sensor, and the detection data group is detected by other sensors; the image data is used for Characterizing the target image collected by the image acquisition sensor by using at least one image data matrix; the different detection data sets are detection data of different detection dimensions;
  • Forming a module 302 for forming a multi-dimensional matrix structure wherein:
  • the multi-dimensional matrix structure includes a plurality of matrix layers distributed in a longitudinal direction, the plurality of matrix layers including at least one pixel matrix layer and at least one sensing matrix layer, and each pixel matrix layer correspondingly is used to represent an image data matrix.
  • Each of the sensing matrix layers is configured to represent a set of sounding data sets, each of the sounding data elements in the sounding data set corresponds to one pixel element in the pixel matrix layer; the value of the sounding data element Both are determined based on the value of the probe data.
  • the forming module is specifically configured to:
  • mapping relationship Determining, according to the established mapping relationship, a pixel element of the target object corresponding to each probe data of the target object, where the mapping relationship is used to represent different locations of different detection dimensions of the other sensors in the detection domain.
  • the probe data is assigned to the probe data element corresponding to the vertical direction of the corresponding pixel element.
  • the process of establishing the mapping relationship includes:
  • the detection domain of each sensor is projected onto the two-dimensional plane where the imaging surface of the image acquisition sensor is located, and the projection area corresponding to each sensor is obtained;
  • mapping relationship determining the mapping relationship according to a projection relationship between a projection area of the sensor and a current detection domain, and a positional relationship between the projection area in the two-dimensional plane and an imaging area of the image acquisition sensor.
  • the detection coordinate system is a spatial rectangular coordinate system, or a spherical coordinate system.
  • mapping relationship is specifically used to represent the correspondence between the detection data of different locations in the detection domain of the sensor and different individual pixels, or:
  • the resolution of the other sensor does not match the resolution of the image acquisition sensor, and the mapping relationship is specifically used to represent the correspondence between the detection data of different positions in the detection domain of the sensor and different macroblocks.
  • the macroblock contains at least two pixels.
  • the resolution of the other sensor does not match the resolution of the image acquisition sensor
  • the data elements in the sensing matrix layer further comprise a first interpolated data element, and/or: the data elements in the pixel matrix layer further comprise a second interpolated data element.
  • the value of the probe data element is the corresponding probe data itself, or the value of the probe data element is determined according to the corresponding probe data conversion.
  • the matrix layer in the multi-dimensional matrix structure can be selectively activated.
  • the other sensor is at least one of: a microwave radar, an ultrasonic radar, a laser radar, an infrared sensor, and a terahertz imaging sensor; and the detection data of the other sensor includes at least one of the following: distance data, speed data. , acceleration data, azimuth data, radar cross section RCS data, and heat radiation temperature data;
  • the pixel data matrix includes at least one of the following: a luminance data matrix, a gray data matrix, an RGB three-layer data matrix, an R-layer data matrix, a G-layer data matrix, a B-layer data matrix, and a YUV three-layer data matrix Y-layer data matrix. , U layer data matrix, V layer data matrix, and optical flow data matrix.
  • the data processing apparatus of the multi-sensor fusion provided by the embodiment can combine data of a plurality of different dimensions measured by different sensors on the basis of pixel elements, and combine them in a multi-dimensional matrix structure, thereby facilitating the acquisition.
  • Data multi-faceted data fusion and deep learning can help achieve more efficient data mining and feature extraction, resulting in more effective environment awareness and target detection capabilities.
  • This embodiment provides a multi-sensor fusion method, including:
  • the information contained in each pixel is longitudinally expanded, and in addition to the brightness and color information originally contained therein, a plurality of vertical dimensions are added for each pixel, and the pixel can be input in the increased vertical dimension.
  • the camera detects probe information of a plurality of corresponding dimensions detected by the target object of the spatial map by the other sensors, and the probe information includes at least one of the following: a relative distance, a relative motion speed, a target radar cross section RCS data, and a target heat.
  • the multi-dimensional detection information is assembled in a layered manner onto the target object description originally based on the image pixels, and a multi-dimensional pixel is generated, which is mathematically represented as a matrix array of a unified structure.
  • a multi-dimensional pixel matrix is obtained.
  • the multi-dimensional pixel matrix can be understood as a unit of pixels, and thus is expressed as a multi-dimensional pixel matrix, which is actually assembled by a plurality of matrices, which can be correspondingly understood to be involved in Embodiment 1.
  • the multidimensional matrix structure that is, the multidimensional pixel matrix and the multidimensional matrix structure are the same meanings.
  • the present embodiment combines target detection data of multiple dimensions in the form of a matrix array (like a stereo matrix). Based on the two-dimensional image space imaged by the camera, this embodiment extends the information contained in each pixel. In addition to the brightness and color information originally included in the image, this embodiment adds multiple vertical dimensions to each pixel. Entering, in the increased vertical dimension, information about a plurality of corresponding dimensions of the target object unit mapped by the camera in the camera detection space (objective space), such as relative distance, relative motion speed, and radar cross section of the target.
  • the RCS data and the target's thermal radiation temperature distribution and other data, the multi-dimensional information is assembled in a layered manner onto the target object descriptor originally in the image pixel unit, and mathematically represented as a matrix array of unified structure.
  • the present embodiment refers to a matrix array description of "multi-dimensional measurement parameters" of such a target as a "multi-dimensional pixel” structure.
  • the distance between the other sensors, such as radar and infrared sensors, the relative velocity, the radar cross-section RCS data of the target, and the heat radiation temperature distribution of the target are added to the basic dimensions of the camera imaging to increase the perceived depth of the system.
  • An array of stereo multi-dimensional depth perception matrices with camera pixels as granularity is established, and each of the original pixels becomes each multi-dimensional pixel in this embodiment.
  • the detection data of different dimensions are combined in the form of a matrix array to form a matrix array of “multi-dimensional measurement parameters”, which is referred to as a multi-dimensional pixel matrix.
  • a matrix array of "multi-dimensional measurement parameters" (both multi-dimensional pixel matrix) is depicted in Figures 2 through 4.
  • This embodiment can also add more data dimensions (data encapsulation with more sensors) on the basis of the same combination.
  • the order of the matrix arrays in the vertical direction can be changed (of course, the change in order may mean that machine learning is to be retrained again).
  • a multi-dimensional matrix structure including a plurality of matrix dimensions that is, the multi-dimensional pixel matrix involved above, the multi-dimensional pixel matrix necessarily includes the sensing matrix layer corresponding to the detection data, that is, the above
  • the layered content in the vertical dimension the multi-dimensional pixel matrix necessarily also includes the pixel matrix layer corresponding to the pixel, that is, the pixel of the brightness and color information originally included in the above, wherein each pixel can be expressed as a pixel element, and the pixel The element and the probe data element are necessarily vertically corresponding.
  • a monochrome camera participates in the combination of multiple sensors, for example, the camera is only used for infrared imaging scenes, and the camera only outputs a monochrome image; in this case, the multi-dimensional pixel structure of the embodiment Still valid, only the RGB (or YUV) three-layer input data is changed to a single-layer Y (pixel brightness) data structure, and the same multi-dimensional pixel structure method is used to combine the input information of other sensors based on the brightness of the single-layer pixel.
  • the camera is a color camera or a monochrome camera
  • the data matrix output by the color camera is three layers of RGB or YUV, and the multi-dimensional pixel matrix is obtained by encapsulating the detection information of other sensors on the three-layer data matrix;
  • the camera is a monochrome camera
  • the camera only outputs a monochrome image obtained by combining the detection information of other sensors on the basis of the brightness of the single layer of pixels.
  • the cross-sectional coordinate pair of the multi-dimensional pixel is equal to the pixel coordinate of the camera avatar plane, because the multi-dimensional pixel is expanded based on the pixel information of the camera avatar plane, and each pixel is added with a plurality of longitudinal dimension information combinations; this embodiment takes The individual of the multi-dimensional pixel from which the pixel of the camera pixel plane coordinate position (x, y) is expanded (both "pixel (x, y)") is called “multidimensional pixel (x, y)".
  • the relative distance, relative speed, and target radar cross section RCS data value can be directly assigned to each multi-dimensional pixel.
  • these assignments can also be calculated by the corresponding formula and then the calculation result is assigned to the matrix layer corresponding to each multi-dimensional pixel.
  • the input of the R (RCS) layer of the multi-dimensional pixel is desired in this embodiment.
  • the RCS is mapped to the corresponding value of the radar received power P.
  • the multi-dimensional pixel adds data such as the distance brought by other sensors such as radar and infrared sensors, the relative velocity, the radar cross-section RCS data of the target, and the heat radiation temperature distribution image of the target in each pixel-based dimension, which can be
  • the direct assignment of the probe data can also be assigned after the function is converted by the function, as shown in the schematic diagram of FIG.
  • the present embodiment combines the detection target data from different sensors.
  • the first embodiment uses geometric space conversion to measure the detection domain of each sensor in a standard three-dimensional Euclidean geometric space (the space coordinate system is labeled as X/Y/Z axis).
  • the physical space areas are associated and establish a one-to-one correspondence.
  • the present embodiment calculates the central axis (X'/Y'/Z' axis) of each detection domain (the respective detection space) of each sensor by the spatial parameter of the sensor installation, and then Through the translation, rotation and scaling of the respective coordinate systems, they are unified into the 3D detection space (objective field of view) coordinates of the camera, so that they are aligned with the optical axis of the camera to determine the unified detection space of the system and the common
  • the detection angle is then based on the radar, infrared thermal imager and other sensor detection fields to establish a mapping relationship on the two-dimensional object surface corresponding to the imaging surface of the camera (two-dimensional object surface in the object space).
  • the targets detected by them are docked with the pixels imaged by the camera according to the mapping relationship, and the target detection data are assigned to corresponding positions in the multi-dimensional pixel matrix in a one-to-one correspondence.
  • mapping relationship is established in the detection domain and the object plane, it is inevitable to represent the mapping relationship between each point in the detection domain and the point of the object surface. Therefore, the use of the mapping relationship is necessarily It is used to determine that the probe data in the detection domain is mapped to that pixel, that is, it can be inferred that the step S121 needs to be implemented, and at the same time, since the assignment must be assigned to the sensing matrix layer, it can be undoubtedly It is inferred that step S122 needs to be implemented, and therefore, the contents of steps S121 and S122 shown in FIG. 5 can be inferred from the above contents without any doubt.
  • this embodiment can also detect the central axis (X'/Y'/Z' axis) of each sensor's detection domain (the respective detection space) in combination with the geometric calibration method. Then, by transforming, rotating and scaling the respective coordinate systems, they are unified into the same coordinate system, and the unified detection domain of the system (three-dimensional geometric space) is established, and the independent detection areas of the sensors are unified in the system. A one-to-one correspondence is established in the region.
  • the method of geometric calibration is: this embodiment places the target at a position where a plurality of spatial position coordinates are determined, and then uses the positional reading of the targets measured by the sensor to read the target position and their physical space in reality. The position establishes a mapping relationship to establish a correspondence between the coordinate reading of the sensor detection target space and the actual geometric spatial position).
  • the algorithm further fine-adjusts the geometric spatial mapping relationship to reduce the geometric mapping positioning error.
  • the principle is as follows: After the final target of the system is detected, the present embodiment performs independent positioning calculations on the plurality of detection targets in the detection domain in respective sensor detection space regions, and maps the final result to the corresponding of "multidimensional pixels".
  • the error of the positioning position of the same target generated by the previous geometric space conversion method (referred to as “geometric mapping positioning error” in this embodiment) is compared (calculated), Then scale and translate the relative position of the corresponding matrix array (layer) to the pixel layer of the camera, reduce the error, that is, reduce the "geometric mapping positioning error", and the value in the corresponding data matrix array (layer) of the "multidimensional pixel” Make adjustments according to the new pixel vertical correspondence.
  • the present embodiment can further reduce the mapping error introduced by the geometric space conversion of different axes, so that the "multi-dimensional pixel" matrix array is combined more accurately.
  • the error is determined by performing positioning calculation on the object, and then mapping the positioning result into the pixel matrix layer, and comparing it with the result of using the mapping relationship map to identify the same object, and then As can be seen, the content of the above embodiment can infer the solution of step S16 in Embodiment 1 without any doubt.
  • the spatial projection relationship of the detection domain of the multi-sensor combination is as shown in FIG. Since the combination of the multi-dimensional pixels of the present embodiment is "each pixel adds a plurality of longitudinal dimensions, the target object unit mapped in the camera detection space (object-space) input by the pixel in the increased longitudinal dimension is detected by other sensors.
  • a variety of information corresponding to the dimension so the detection space of other various sensors is uniformly mapped to the camera detection space and aligned with its optical axis, and the vertices of other various sensor detection domains coincide with the entrance center of the camera, refer to FIG.
  • FIG. 10 the alternative embodiment of the present embodiment can be illustrated by a combination of a camera + radar + infrared thermal imager, which is a typical combination. If other sensors are added, the principle of multi-dimensional pixel data mapping is the same. of.
  • each sensor should be as close to coaxial as possible on the spatial structure, and the closer it is, the better, which is beneficial to reduce the mapping error introduced by the geometric space conversion of different axes (the result is similar to the virtual image).
  • this embodiment will solve the problem by interpolation in the data assembly.
  • the subsequent embodiment can analyze and process the data of each layer by using various traditional feature values + classifiers to detect the target; or use the neural network method for subsequent detection. Processing; or a mixture of two methods.
  • the multi-dimensional target detection information is unified into a matrix array and mapped in units of pixels, such deep fusion data combination has a huge improvement on the detection quality of the target. help.
  • the network will generate more layers of feature maps. With richer multi-layer and multi-dimensional feature extraction, this embodiment can detect targets more efficiently and with high quality. Positioning. Multi-dimensional pixel matrix matching algorithm is very good.
  • Currently popular neural network algorithms, such as R-CNN, Faster R-CNN, SSD, etc. corresponding to the multi-dimensional pixel matrix input (multi-layer input) of this embodiment, corresponding changes are applicable.
  • the detection of the target is often in the form of machine learning, which involves the collection of the target sample and the training of the system.
  • the matrix array description mode (multi-dimensional pixel) of the multi-dimensional measurement parameter of the embodiment facilitates the collection and training of the target sample; the information of each layer of the multi-dimensional pixel is relatively independent, and the structure of each layer can be increased or decreased, and subsequent processing It can dynamically activate one layer or multiple layers (the information of these layers participates in the determination and description of the target) or inactive (the information of these layers does not participate in the determination and description of the target), but generally does not hinder the multi-dimensional
  • the pixel structure is used to detect and mark the target.
  • This embodiment proposes to activate all matrix array layers of multi-dimensional pixels when the target sample is collected, but can dynamically combine specific matrices according to specific scenarios (daytime, nighttime, overlapping state of sensor detection field of view, etc.) during training.
  • the activation of the array layer is used for training to match the corresponding scene.
  • this embodiment can use the "multi-dimensional pixel" matrix array to organically combine them, use the same processing algorithm framework to detect targets; their training methods (including data The collection of samples can also be combined and implemented at one time. This is another technical feature of the present embodiment, and is also a benefit of the method of describing a target object using a "multi-dimensional pixel" structure.
  • a multi-sensor fusion system is formed by using a camera, a microwave radar and an infrared thermal imager.
  • This is a commonly used multi-sensor combination.
  • the camera outputs a color image (RGB or YUV data).
  • the microwave radar outputs the distance, relative velocity, azimuth and target radar cross section RCS (Radar Cross-Section) data.
  • the thermal imager outputs a thermal radiation temperature distribution image of the target. This combination can sense and detect targets from multiple dimensions, and can work effectively under a variety of conditions (daytime, night, fog, rain and other harsh environments).
  • a variety of sensors can be flexibly combined and combined, so here a system can be used in all three (camera + microwave radar + infrared thermal imager), or a combination of two or two: camera plus microwave radar , camera plus infrared thermal imager, and even a combination of microwave radar and infrared thermal imager. Since the target detection data of multiple dimensions is combined in the form of a matrix array (like a stereo matrix), the system can dynamically adjust the sensor for target recognition according to the hardware configuration or the scene (during different conditions such as daytime and nighttime).
  • the dimension of the input parameter (adjusting the number of active layers of the multi-dimensional target detection data matrix array) is detected using a subset of the multi-dimensional pixels, for example, in the night driving, the target to detect the target outside the illumination range of the vehicle, this embodiment It is possible to activate only the input matrix of the radar and the input matrix of the infrared thermal imager. You can even dynamically add or remove a sensor by hardware dynamic configuration, and the system will still work.
  • the flexible combination of system hardware in addition to providing flexibility in the choice of system hardware configuration cost, can also bring benefits to users: in one scenario, one of the sensors of the system fails (or is damaged), and the system passes the software. The configuration adjustments remain operational and enhance the robustness of the system. In some application areas, such as ADAS or autopilot applications, it is necessary to increase the robustness of the system.
  • the outer solid line frame represents the imaging area of the camera, and the inner solid frame inside is the infrared thermal imager imaging area, and the dotted area is the radar detection area, which partially overlaps to form an interlaced detection area.
  • the Class A area is the area jointly detected by the three sensors
  • the Class B and Class C areas are the areas jointly detected by the radar and the camera
  • the Class D and Class E areas are the areas jointly detected by the camera and the infrared thermal imager.
  • the G-type area is an area that only the camera detects.
  • the most concerned about this embodiment is the area (Class A area) that the three sensors jointly explore, which can make the best use of the accurate detection of multiple data dimensions brought by multi-sensor fusion.
  • the second important area is the area where the camera and the millimeter wave radar overlap (B and C areas). The detection dimensions of the camera and the millimeter wave radar are highly complementary, and the fusion of the two sensors is also very meaningful.
  • the third important fusion area is the area where the camera overlaps with the infrared thermal imager.
  • the infrared thermal imager can make up for the lack of the camera in the night, foggy days, etc.; Image information, information fusion between them, more technical challenges come from the matching with the resolution, the image resolution (pixel number) of the infrared sensor needs to be enlarged by image interpolation, or the resolution of the camera image is reduced.
  • the image of the infrared sensor (in this embodiment labeled H dimension) is then attached to a matrix array of RGB (or YUV) data structures with cameras.
  • the camera collects the collected data in three layers according to the RGB color (sequence can be exchanged), assuming that the image size (both resolution) of each layer is X*Y (for example: 1920*1080 – camera corresponding to 1080P resolution), if the raw data input is YUV format, it can also be arranged in three layers according to YUV, but this embodiment suggests to convert to RGB data structure (YUV to RGB). ), because this can reduce the association of each layer of data, which is conducive to subsequent independent feature extraction.
  • the three-dimensional stereo data structure (size: X*Y*3) is taken as the “original data input layer of the camera”, and then the microwave radar acquisition is added according to the structure of the multi-dimensional pixel on the original data input layer of the camera.
  • the data is taken as the “original data input layer of the camera”, and then the microwave radar acquisition is added according to the structure of the multi-dimensional pixel on the original data input layer of the camera. The data.
  • the radar data is too sparse. If it is to be directly matched point by point with the camera image data according to the pixel relationship, it needs to be processed first to convert the radar data into a dense structure with a tensor structure. Class image data.
  • the following method is designed to input the data of the radar into the system “multi-dimensional pixel matrix” of the embodiment: 1) using the geometric projection method to project the detection space of the radar to the imaging plane of the camera.
  • the 2-dimensional object as a 2-dimensional mapping surface of the radar target (as shown in the schematic diagram of Figure 3); its 2-dimensional spatial resolution is equal to the pixel resolution of the matching camera in the system, and the radar data and camera are established.
  • the radar sensing matrix data on the matrix layer (depth) Layer) is composed of the following "radar raw data input”: L (target distance value) layer, S (relative speed value) layer, R (radar cross section value) layer; likewise, the order of these layers can be interactive, or Flexible combination (all of L, S, and R are activated), or only one of them (L or S or R) or two of them (L+S, S+R, etc.).
  • the spatial resolution of millimeter-wave radar is relatively low, and the angular resolution of the target is not high, so that its numerical projection has a relatively large "possible coverage area" on the 2-dimensional mapping surface of the target, similar to the original pixel particle size ratio of the radar.
  • the camera has a large pixel size and a low resolution.
  • the present embodiment needs to interpolate the "radar two-dimensional mapping surface" to improve the resolution and the camera.
  • Common interpolation methods such as nearest neighbor interpolation, bilinear interpolation, and cubic convolution, can be used.
  • the radar data Since the radar data is sparse, in the radar data structure (matrix data layer L, S, R, etc.), in the area where the radar has detected the target, the radar data will be assigned in one-to-one correspondence. However, in the region where the target is not detected, the present embodiment assigns the original data of the radar corresponding to these regions to “0”, or according to a preset default value representing the background to ensure each matrix in the radar data matrix. Units have assignments.
  • the images acquired by the infrared thermal imager are also in pixels.
  • the resolution of the image is matched by appropriate interpolation to match the resolution of the camera, and the image collected by the infrared thermal imager (generally black and white brightness pixel information) is assigned point by point to the corresponding data in the “multidimensional pixel” data structure.
  • the matrix which is referred to herein as the "H" matrix.
  • the resolution of the infrared thermal imager is lower than that of the ordinary camera.
  • This embodiment uses interpolation to amplify the resolution. Of course, the special case is not excluded.
  • the image resolution of the infrared thermal imager is higher than that of the system. Camera resolution, in this case, the embodiment can be processed by reducing the resolution of the thermal imager; in short, the basic principle is that the resolution of the two sensors is the same, and then the corresponding data layers of the multi-dimensional pixels are assigned the same resolution.
  • the interpolated data elements are necessarily included in the matrix, that is, it can undoubtedly obtain the content that can include the interpolated data elements in the matrix.
  • the present embodiment selects the most common camera + millimeter wave radar + infrared thermal imager; this combination is flexible and diverse, and all three sensors can be selected to form one system, or two of them can be selected ( The camera + other) constitutes a system, and the number of sensors is also flexible. It can be a system consisting of multiple cameras plus multiple radars plus multiple thermal imagers.
  • the camera outputs a color image (RGB or YUV data), the distance of the microwave radar output detection target, the relative velocity, the azimuth angle, and the target radar cross section RCS data
  • the infrared thermal imager outputs the target Temperature distribution image
  • this embodiment maps these physical detection dimensions to the detection target, so that the detection target can be detected, classified, and identified beyond the single sensor detection dimension, and the system can have higher target detection rate and better recognition.
  • Ability and recognition quality is: the camera outputs a color image (RGB or YUV data), the distance of the microwave radar output detection target, the relative velocity, the azimuth angle, and the target radar cross section RCS data
  • the infrared thermal imager outputs the target Temperature distribution image
  • this embodiment can also introduce other kinds of sensors, such as laser radar, etc., the detection parameters brought by the new sensor. It can also be added to the data structure combination of the "camera + millimeter wave radar + infrared thermal imager" system of the present embodiment, and becomes a part of the "multidimensional pixel" of the present embodiment.
  • the present embodiment uses multiple sensors to identify targets from multiple detection dimensions.
  • target detection data of multiple dimensions is combined in the form of a matrix array (like a stereo matrix), and the matrix of the data.
  • Each "multidimensional pixel" of the array is organically combined with multi-dimensionally sensed detection information, and in a unit (a "multidimensional pixel"), such a structure for subsequent data processing (regardless of the use of traditional eigenvalues plus classification)
  • the method of the device, or the method of the neural network, or a combination of the two, and the sampling of the training samples bring great convenience.
  • Such data assembly is also very effective for spatial navigation and positioning (such as SLAM), because in the "multi-dimensional pixel" description system of this embodiment, the direct description of the target object not only has feature data (for classification recognition), but also With three-dimensional position and space information (the spatial angle on the XY axis and the distance from the detector to the target), the result of the target recognition can be directly used for spatial positioning.
  • SLAM spatial navigation and positioning
  • This embodiment is applicable to vehicle driving assistance systems (ADAS) and automatic driving systems, robots, unmanned aerial vehicles (AGVs), and various devices and systems that require environmental sensing and target detection capabilities.
  • ADAS vehicle driving assistance systems
  • AVS unmanned aerial vehicles
  • FIG. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
  • the electronic device 40 may include a memory 42 and a processor 41.
  • the memory 42 is configured to store sensing data, intermediate running data, system output data, and executable instructions of the processor 41.
  • the processor 41 is configured to perform the methods involved in Embodiment 1 and Embodiment 2 and its alternatives by executing the executable instructions.
  • the memory 42 and the processor 41 can communicate via the bus 43.
  • Figure 13 is a block diagram showing the structure of a sensing device in accordance with an embodiment of the present invention.
  • a sensing device 50 includes a memory 52, a processor 51 and a sensor 54; the memory 52 is configured to store sensing data, intermediate running data, system output data, and executable instructions of the processor 51;
  • the processor 51 is configured to perform the methods involved in Embodiment 1 and Embodiment 2 and its alternatives by executing the executable instructions.
  • the memory 52, the processor 51 and the sensor 54 can communicate via the bus 53.
  • An embodiment of the present invention further provides a storage medium having stored thereon sensing data, intermediate running data, system output data and a program, and the program is executed by the processor to implement Embodiment 1 and Embodiment 2 and its alternatives. The method involved.

Abstract

一种多传感器融合的数据处理方法及数据处理装置、多传感器融合方法，能够将不同传感器测得的多个不同维度的数据以像素元素为基础，用多维矩阵结构的形式组合在一起，进而，可有利于对取得的数据做多层面的数据融合与深度学习，有利于实现更多样更有效的数据挖掘与特征提取，从而产生更有效的环境感知与目标检测的能力。

Description

多传感器融合的数据处理方法、装置与多传感器融合方法 技术领域
本发明涉及电子设备的数据处理领域，尤其涉及一种多传感器融合的数据处理方法、装置与多传感器融合方法。
背景技术
在目标识别与环境感知领域，需要相对完整的数据采集和保存用于环境感知与目标检测任务处理，如果采用机器学习则还需要大量的正负样本用于学习与训练；在目标识别过程中，会产生大量的中间数据供处理单元处理使用，而且可能在目标识别过程中有“云计算”(远程处理)的参与，需要高效的数据采集、保存技术。同时，有效的环境感知与目标检测需要多维度的探测数据支撑。
目前常用于环境感知与目标检测的传感器可包括：以摄像头为例的图像采集传感器、微波雷达、红外传感器、超声波雷达以及激光雷达等等，它们被广泛地用于车辆驾驶辅助系统(ADAS)和自动驾驶系统、机器人、无人搬运车(AGV)、智能家居、智能安防，以及各种需要有环境感知与目标检测能力的设备与系统中。
图像采集传感器(摄像头)可以感知目标的纹理(形状，轮廓，光照明暗等)与色彩，记录一瞬间的图像信息。摄像头还可以录制视频信息，把记录的事件用时间轴串起来形成视频流，可用于事件的回放与时间关联的事件分析。红外传感器(红外摄像头)是图像采集传感器的一种，能捕捉目标的红外辐射信息并以图片和视频的格式来保存。微波雷达(或者统称为雷达)可以捕捉目标的相对距离、相对运动速度、目标的雷达散射截面RCS数据，并以热图、目标物的相对距离、相对运动速度、目标的雷达散射截面RCS数据维度的定量表述(Radar Object Data Output)，或者点云数据输出。激光雷达则主要通过探测目标的空间位置(相对距离，空间角度位置坐标信息)输出目标的点云数据。各种传感器都有自己的信息感知维度，比如说我们常用的摄像头，它可以捕捉目标的图像信息，栩栩如生的记录拍摄那一刻的环境与目标的纹理与色彩信息，但是我们可能无法从单张图片中准确提取目标的距离、速度信息，我们也很难从一张传统的照片来预测事件下一刻将要会发生什么。我们用视频的方式(视频的本质是把照片用它们各自被拍摄瞬间的时间轴串联起来再按照时间轴回放的图片系列)来记录和分析事件，但是视频的方式带来巨量的数据，以及因此带来的大传输带宽和存储数据的空间的需求。其它传感器的记录方式，比如说雷达，超声波以及激光雷达等，能记录各自传感器自身感知维度的信息，比如，目标的距离、速度；他们记录的数据信息以及目前的记录方式(数据结构)对于我们需要的全面描述记录目标特征并可以直接用于环境感知与事件预测，数据的维度与完整度不足。
现有相关技术中，各传感器探测到的数据是相互独立的，缺乏深度融合，进而，在利用探测到的数据进行特征提取、数据挖掘等处理时，环境感知与目标检测的能力都较弱。
发明内容
本发明提供一种多传感器融合的数据处理方法、装置与多传感器融合方法，以解决传感器探测到的数据缺乏深度融合的问题。
根据本发明的第一方面，提供了一种多传感器融合的数据处理方法，包括：
获取目标对象的图像数据与至少一组探测数据组；所述图像数据是图像采集传感器探测到的，所述探测数据组是其他传感器探测到的；所述图像数据用于利用至少一个像素数据矩阵表征所述图像采集传感器采集到的目标图像；不同探测数据组为不同探测维度的探测数据；将图像数据与其它探测数据建立起映射关系,在数据结构上
形成多维矩阵结构，具体可例如形成“多维度测量参数”矩阵数组的多维矩阵结构(也可称其为多维像素矩阵)；其中：
所述多维矩阵结构包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层，每个像素矩阵层对应用于表征一个像素数据矩阵，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据组中的探测数据元素纵向对应于所述像素矩阵层中的的像素元素；所述探测数据元素的数值均是根据探测数据赋值确定的。
根据第二方面，提供了一种多传感器融合的数据处理装置，包括：
获取模块，用于获取目标对象的图像数据与至少一组探测数据组；所述图像数据是图像采集传感器探测到的，所述探测数据组是其他传感器探测到的；所述图像数据用于利用至少一个像素数据矩阵表征所述图像采集传感器采集到的目标图像；不同探测数据组为不同探测维度的探测数据；
形成模块，用于形成多维矩阵结构；其中：
所述多维矩阵结构包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层，每个像素矩阵层对应用于表征一个像素数据矩阵，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据组中的探测数据元素纵向对应于所述像素矩阵层中的像素元素；所述探测数据元素的数值均是根据探测数据赋值确定的。
根据本发明的第三方面，提供了一种多传感器融合方法，包括：
把来自多传感器的多个维度的探测数据用多维像素矩阵的形式组合在一起，建立以摄像头像素为颗粒度的立体多维深度感知矩阵数组；
在多维像素矩阵中，把每个像素包含的信息做了纵向扩展，除了其原本包含的亮度与颜色信息，还为每个像素增加了多个纵向维度，在增加的纵向维度上能够输入该像素在摄像头探测空间映射的目标对象被其它传感器探测到的多种对应维度的探测信息，所述探测信息包括以下至少之一：相对距离、相对运动速度、目标的雷达散射截面RCS数据以及目标的热辐射温度分布等 数据；其中，把多维度的探测信息以分层的方式装配到原本以图像像素为单元的目标对象描述之上，产生多维像素，其在数学上表现为统一结构的矩阵数组，以使得原来的每一个像素变成了一个多维像素，得到多维像素矩阵。
根据本发明的第四方面，提供了一种处理设备，包括存储器与处理器；
所述存储器，用于存储感知数据，中间运行数据，系统输出数据和所述处理器的可执行指令；
所述处理器配置为经由执行所述可执行指令来执行第一方面及其可选方案涉及的方法，或者第三方面及其可选方案涉及的方法。
根据本发明的第五方面，提供了一种传感设备，包括存储器、处理器与传感器；
所述存储器，用于存储感知数据，中间运行数据，系统输出数据和所述处理器的可执行指令；
所述处理器配置为经由执行所述可执行指令来执行第一方面及其可选方案涉及的方法，或者第三方面及其可选方案涉及的方法。
根据本发明的第六方面，提供了一种存储介质，其上存储有程序，其特征在于，该程序被处理器执行时实现第一方面及其可选方案涉及的方法，或者第三方面及其可选方案涉及的方法，同时可存储感知数据，中间运行数据以及系统输出数据。
本发明提供的多传感器融合的数据处理方法、装置与多传感器融合方法，能够将不同传感器测得的多个不同维度的数据以像素元素为基础，按照其成像的空间数据采样模型把多种数据映射,进行统一的数据对齐与归并,在数据结构上形成“多维度测量参数”矩阵数组的多维矩阵结构，用多维矩阵结构的形式组合在一起，进而，可对取得的数据做多层面的数据融合与深度学习，其可实现更多样更有效的数据挖掘与特征提取，从而产生更有效的环境感知与目标检测的能力。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍。
图1是本发明实施例1中多传感器融合的数据处理方法的一流程示意图；
图2是本发明实施例1中形成多维矩阵结构的一原理示意图；
图3是本发明实施例1中形成多维矩阵结构的另一原理示意图；
图4是本发明实施例1中形成多维矩阵结构的又一原理示意图；
图5是本发明实施例1中多传感器融合的数据处理方法的另一流程示意图；
图6是本发明实施例1中建立映射关系的流程示意图；
图7是本发明实施例1中步骤S121与步骤S123的流程示意图；
图8是本发明实施例1中步骤S16的流程示意图；
图9是本发明一实施例中投射的示意图；
图10是本发明一实施例中成像面的示意图；
图11是本发明实施例1中多传感器融合的数据处理装置的结构示意图；
图12是本发明一实施例中电子设备的结构示意图；
图13是本发明一实施例中传感设备的结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
实施例1
图1是本发明实施例1中多传感器融合的数据处理方法的一流程示意图。
请参考图1，多传感器融合的数据处理方法，包括：
S11：获取目标对象的图像数据与至少一组探测数据组。
其中的处理方法，可具体包括采集处理的过程，也可以包括在存储处理的过程，还可以包括采集后存储前的中间处理的过程，即：在数据的采集、中间处理与存储的任意之一过程中使用本实施例及其可选方案所涉及的方法，均可理解为实施了该处理方法。对应的，在采集数据的设备、处理数据的设备、存储数据的设备中使用本实施例及其可选方案所涉及的方法，也均可理解为对该处理方法的实施。即：其均不脱离本发明及其可选方案的描述范围。
图像数据，可以理解为是图像采集传感器探测到的，该图像采集传感器可以为任意能够实现图像采集的设备，例如可以为摄像头、还可以为手机、平板电脑、计算机等内置有摄像头的设备。该图像采集传感器的安装参数、图像的采集、存储格式等都可以预先确定。
图像数据，具体可用于利用至少一个像素数据矩阵表征所述图像采集传感器采集到的目标图像。该像素数据矩阵可根据图像采集传感器的不同而发生变化。
其中一种实施方式中，若该图像为RGB图像，其中的R可以表示Red红色，G可以表示Green，B可以表示Blue，则：该RGB图像可被三层数据表征，具体为红、绿、蓝三层数据，每个颜色的数据可通过一个像素数据矩阵表征；若该图像为YUV，其中的Y可以表示明亮度Luma，即灰阶值，U和V可以表示色度与色差信息，其中的每层数据可通过一个图像数据矩阵表征。
在另一可选实施方式中，也可仅具有明亮度Y的像素数据矩阵，而不具有U、V的图像数据矩阵，还可仅具有R、G、B中任意至少之一的像素数据矩阵，一种举例中，也可以只用U、V其中之一的像素数据矩阵。
具体实施过程中，目标图像可以利用RGB或者YUV三层数据矩阵来表征，也可以把数据封装到一层，即采用一层数据矩阵来表征三层的内容，还可用多位数据单元的一个单元来表述一个像素的RBG或者YUV组合值，该 多位数据可例如24bit、32bit的数据甚至更多位的数据。
另一实施过程中，也可只有单色摄像头参与到融合中，例如作为图像采集传感器的摄像头可以只用于红外成像场景，摄像头只输出单色图像；在这种情况下，以上所涉及的多维像素结构仍然可有效，具体可将RGB三层或者YUV三层更改为单层明亮度Y数据结构，即仅具有明亮度Y的图像数据矩阵，在单层明亮度的基础上，可利用后文各可选实施方式所涉及的各步骤融合其他传感器的数据，产生多维矩阵结构。该单色摄像头可为非彩色的摄像头，具体可举例为红外摄像头，进而，可直接捕捉图像的灰度值。
不管是何种情况，同一目标对象可以被其他传感器捕捉的其它探测维度信息以分层的方式装配到它被图像采集传感器捕捉的图像数据的像素矩阵层上，进而，后文中可将图像的像素做为每个探测数据的组合基础，数据是一一与像素对应的。
具体举例中，作为图像采集传感器的摄像头可将采集到的图像数据，按照不局限顺序的RGB颜色进行三层排列，得到对应的图像数据矩阵，具体可例如摄像头分辨率为X*Y(例如：1920*1080，其可对应于1080P分辨率的摄像头)；另一举例中，如果原始数据输入是YUV格式，也可以按照YUV三层排列，得到对应的图像数据矩阵，再一举例中，还可将YUV格式转换为RGB格式的，其可有利于减少各层数据的关联，有利于后续独立的特征提取。
探测数据组，可理解为除了图像采集传感器以外的其他传感器探测到的任意数据组，每个数据组可理解为与一层传感矩阵层对应，一个传感器可产生一组或多组探测数据组。不同探测数据组为不同探测维度的探测数据，其中的不同的探测维度，可理解为：具有不同物理含义的探测数据，其也可被理解为：在同一探测维度下，在不考虑其空间位置的情况下，不同探测数据的差别在于数值本身的差别，而非其物理含义的差别。在举例中，同一探测维度下的探测数据的物理学单位通常是相同的，即：在举例中，若两个探测数据物理学单位不同，则其通常属于不同的探测维度。同时，本实施例也不排除例外的情形。
此外，一个其他传感器可能得到多个探测维度的多个探测数据组，也可能得到一个探测维度的一个探测数据组。
其中一种实施方式中，所述其他传感器为以下至少之一：微波雷达、超声波雷达、激光雷达，以及红外传感器。
其中的红外传感器，可用于检测热辐射温度数据等，微波雷达可以探测到目标距离，相对速度，雷达散射截面RCS(Radar-Cross Section)数据等，基于物体间距离与其他映射关系，具体实施方式中还可进一步计算获取到矢量速度、加速度、方位等等。故而，探测数据可以包括以下至少之一：距离数据、速度数据、加速度数据、方位数据、对微波的反射特性数据，以及热辐射温度数据等，进一步可以为对探测域中目标对象的距离数据的变化、速度数据的变化、加速度数据的变化、方位数据的变化、RCS数据，以及热辐射温度数据的变化的表征。
同时，以上所涉及的距离数据，以及基于距离数据获取到的速度数据、加速度数据、方位数据等等，也可利用雷达探测到。
所述其他传感器的探测数据：雷达散射截面RCS数据；其中的RCS，具体可以为Radar-Cross Section。进一步可理解为对探测域中目标对象的RCS数据的变化的表征。
从以上描述也可见，以上所涉及的探测数据，不论是其他传感器的探测数据，还是图像采集传感器的例如RGB数据的探测数据，均可以是传感器直接探测到的，也可以是传感器直接探测再间接计算得到的。
此外，探测数据还可以是传感器在直接产生探测数据时的中间数据，例如作为图像采集传感器的摄像头的光流数据。光流表达了图像的变化，由于它包含了目标运动的信息，因此可被观察者用来确定目标的运动情况。光流数据是作为图像采集传感器的摄像头连续图像帧之间的像素关系求导处理的参数，可以是二维的矢量(X、Y方向)。故而，本实施例可以把当前的像素对应与前续帧的光流数据加入进来，对应的，图像数据矩阵可增加一个光流数据矩阵，进而，可以为系统的数据组织增加更多的数据维度，用于后续的数据综合处理。
以上所涉及的中间数据，还可例如推算出的三维矢量速度，即运动速度的矢量数据等等。具体的，在有些场景，需要对目标对象的运动速度做精确的记录，为了增加精度，可以以矢量速度而非相对速度来表征，在这种情况下，我们可以利用系统推算出目标矢量速度数据在三维立体空间X/Y/Z轴或者对应的球坐标系(ρ,φ,θ)坐标的各个分量，进而，速度数据可对应于三层传感矩阵层，分别对应表示X、Y、Z的三个分量，或者ρ,φ,θ的三个分量。进而，本实施例所涉及的多维矩阵结构也可记录了目标对象的运动速度的矢量值。
故而，以上所涉及的速度数据，可以为运动速度的矢量分量，也可以是速度值。
可见，任意经传感器检测得到的可被二维分布的数据，均可理解为以上所涉及的探测数据，即均不脱离本实施例描述的范围。
此外，例如摄像头的图像采集传感器具体可以为能够感知目标对象的纹理(形状，轮廓，光照明暗等)与色彩，记录一瞬间的图像信息的传感器。摄像头还可以录制视频信息，把记录的事件用时间轴串起来形成视频流，可用于事件的回放与时间关联的事件分析。
以上所述涉及的红外传感器，其也可理解为一种图像采集传感器，故而，红外传感器可以作为前文所涉及的图像采集传感器被使用，也可作为一种其他传感器被使用，其能捕捉目标的红外辐射信息并以图片和视频的格式来保存。
以上所涉及的微波雷达可以捕捉目标的相对距离、相对运动速度、目标的雷达散射截面RCS数据，并以热图、目标对象的相对距离、相对运动速度、目标的雷达散射截面RCS数据维度的定量表述，或者点云数据输出。
以上所涉及的激光雷达则主要通过探测目标对象的空间位置(相对距离，空间角度位置坐标信息)输出目标的点云数据。
由于激光雷达通常输出的是点云数据结构：灰度像素点云数据(X1，Y1，Z1，灰度值1)，或者彩色点云数据(X1,Y1,Z1,r1,g1,b1)，同样可将这4层或者6层的数据组合到这个目标点按照光学摄像头捕捉图像的空间采样投射模型映射到矩阵层的对应位置。
可见，各种传感器都有自己的信息感知维度，例如本领域常用的摄像头，其可以捕捉目标的图像信息，栩栩如生的记录拍摄那一刻的环境与目标的纹理与色彩信息，但是可能无法从单张图片中准确提取目标的距离、速度信息，也很难从一张传统的照片来预测事件下一刻将要会发生什么。但是视频的方式会带来巨量的数据，以及因此带来的大传输带宽和存储数据的空间的需求。其它传感器的记录方式，比如说雷达，超声波以及激光雷达等，能记录各自传感器自身感知维度的信息，比如，目标的距离、速度；他们记录的数据信息以及目前的记录方式(数据结构)对于我们需要的全面描述记录目标特征并可以直接用于环境感知与事件预测，数据的维度与完整度不足。如何把这些不同种类的传感器采集的数据信息在统一的时间与空间轴上组织起来并高效记录保存下来，现有相关技术中缺乏通用的、可行的方法。本实施例所涉及的方法可以对所采集的各种数据做高效的多层数据融合，充分利用来自不同维度的信息组合来支持更多样有效的信息采样与保存，用于目标特征提取与数据分析。
S12：形成多维矩阵结构。
此外，在形成多维矩阵结构后，还可为该结构加上时间戳记录采样的时间，其可有利于实施环境与目标的多维信息数据采集与存储。
多维矩阵结构，以包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层。所述探测数据组中的每个探测数据元素纵向对应于所述像素矩阵层中的像素元素，其可以是一一对应的，也可以是一对多或者多对一的，可见，单个探测数据矩阵可以纵向对应于单个像素元素，也可纵向对应于包含多个像素元素的区域范围。
可见，本实施例所涉及的方法增加了系统感知深度，建立了以像素为颗粒度的立体多维的深度感知的数组结构，即多维矩阵结构。简言之，本实施例可将现有相关技术中的每一个像素扩展为每一个多维的数据结构，即本实施例把不同维度的探测数据用矩阵数组的形式组合在一起，组成一个多维矩阵结构。
本实施例中，可利用多维矩阵结构来表征各探测数据与像素的映射结果，同时还可通过其他描述方式来表征探测数据与像素的映射结果，例如语义描述、表格等等。
图2是本发明实施例1中形成多维矩阵结构的一原理示意图；图3是本发明实施例1中形成多维矩阵结构的另一原理示意图；图4是本发明实施例1中形成多维矩阵结构的又一原理示意图。
本实施例中，每个像素矩阵层对应用于表征一个像素数据矩阵，其中，像素矩阵层可以为像素数据矩阵本身，例如前文所涉及的RGB三层或者YUV三层等等，以图2至图4所示实施方式为例，其可以为RGB三层像素矩阵层，其可具体理解为图3和图4中最左侧的三层数据结构，还可具体理解为图5中最靠前的三层数据结构。其他可选实施方式中，像素矩阵层也可以为根据像素数据矩阵而产生的其他矩阵层，例如可以为对像素数据矩阵进行插值后或者换算后产生的其他矩阵层。
图2至图4具体采用了图像采集传感器+雷达+红外传感器的组合来做示意，其仅为一种举例的组合，本实施例的可选方案还可再加入其他传感器，或去除部分传感器，其作用原理是一致的。
本实施例中，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据元素的数值均是根据探测数据赋值确定的。
故而，多维矩阵结构中各矩阵层的各个位置的数据元素，也可利用多维矩阵结构的横截面坐标(x，y)来表征。可见，上下文所涉及的纵向，可理解为各矩阵层之间的分布方向，上下文所涉及的横向，可理解为单个矩阵中各元素的分布方向。
以图2和图4为例，其中的L层的传感矩阵层可理解为表征雷达所探测到的距离数据的传感矩阵层，其中的S层的传感矩阵层可理解为表征雷达所探测到的速度数据的传感矩阵层，其中的R层的传感矩阵层可理解为表征以上所涉及的RCS数据的传感矩阵层，其中的H层的传感矩阵层可理解为红外传感器探测到的热辐射温度数据的传感矩阵层。
可见，在其中一种实施方式中，所述探测数据元素的数值为对应的探测数据本身，具体可理解为所述探测数据元素的数值是对应的探测数据直接赋值确定的。
在另一实施方式中，所述探测数据元素的数值也可以是根据对应的探测数据换算确定的，具体可理解为所述探测数据元素的数值也可以是利用换算确定的数值赋值确定的。
例如：R层的传感矩阵层，可输入RCS映射到雷达接收功率P的相应值，进而，可先将对应多维矩阵结构的横截面坐标位置为(x,y)对应的RCS值先做如下计算：P(x，y)＝RCS(x，y)/(L(x，y) 4)；
其中：
RCS(x,y)是指(x,y)的横截面坐标位置对应的RCS值；
L(x,y)是指雷达探测到目标对象的相对距离值。
进而，可利用换算后的P(x,y)进行复制。
可见，该换算确定的数值，可以是单个探测数据换算确定，也可以是利用多个不同类的探测数据换算确定的。
在图3所示实施方式中，其中的F1(L)中各数据元素所赋值的数值，可理解为对雷达所探测到的距离数据L进行函数换算后的数值；其中的F2(S)中各数据元素所赋值的数值，可理解为对雷达所探测到的速度数据S进行函 数换算后的数值；其中的F3(R)中各数据元素所赋值的数值，可理解为对雷达所探测到的RCS数据进行函数换算后的数值；其中的F4(H)中各数据元素所赋值的数值，可理解为对红外传感器所探测到的热辐射温度数据进行函数换算后的数值。
本实施例可选方案所涉及的换算，也可仅针对部分探测数据实施，而不限于如图3所示。
此外，以上所涉及的赋值，主要是针对于探测域中目标对象部分的赋值，即赋值本身也可表征为对目标对象的检测结果。从中也可看出，在应用本实施例及其可选方案所涉及的多维矩阵结构时，可用于进行目标检测，以及与之相关的环境感知。
图5是本发明另一实施例中多传感器融合的数据处理方法的流程示意图。
请参考图5，本实施例中，在步骤S12，可包括：
S121：根据已建立的映射关系，确定所述目标对象的每一个探测数据对应的所述目标对象的像素元素。
确定目标对象的像素元素的过程可以为得到目标像素信息的过程，该目标像素信息可理解为用于表征所述探测数据所对应的像素。具体实施过程中，可以利用横截面坐标，例如以上所表征的(x，y)来表征该目标像素信息，也可以利用任意预设的标识信息来表征该目标像素信息。
由于像素矩阵层是表征像素数据矩阵的，而像素数据矩阵本身就是表征其中像素的，故而，本实施例在形成多维矩阵结构时，只需利用映射关系来确定该对应的像素元素即可。
映射关系，可以理解为用于表征所述其他传感器的探测域中不同探测维度的不同位置的探测数据与不同像素元素之间的映射关系。该映射关系可以利用任意的数据结构来表征，例如可以利用表格的形式，或者任意的语句形式来表征。
其中一种实施方式中，所述映射关系具体用于表征所述传感器的探测域中不同探测维度的不同位置的探测数据与不同的单个像素元素之间的对应关系。
图6是本发明一实施例中建立映射关系的流程示意图；图9是本发明一实施例中投射的示意图；图10是本发明一实施例中成像面的示意图。
请参考图6，建立所述映射关系的过程，包括：
S13：通过数据处理变动各传感器的探测坐标系，并使得各传感器的探测域的中心轴线与所述图像采集传感器的光轴一致，以及：各传感器的探测域的探测顶点与所述图像采集传感器的入瞳中心重合。
其中的入瞳中心27，可参照例如摄像头的图像采集传感器的常规原理理解，其可与探测顶点28重合，该探测顶点28也可理解为传感器的探测原点。
通过以上步骤，以统一探测域并将多传感器组合的探测空间映射关系经由空间坐标变换，统一转换到例如摄像头的图像采集传感器的三维探测空间 上，进而可便于以图像采集传感器的探测方向、探测空间为依据实现投射，进而建立其所需的映射关系。其中的变动，可例如坐标系的平移、旋转和缩放。
其中一种实施方式中，该探测坐标系可以为直角坐标系，进而可具有互相垂直的X轴、Y轴与Z轴，进而，使用几何空间转换的方式可在标准的三维欧氏立体几何空间下，将各传感器的探测域(数据探测采集的物理空间区域)关联起来，通过对各自坐标系的平移、旋转和缩放，把它们统一到摄像头的3维探测空间坐标里，即通过以上步骤S13，确立系统的统一探测空间以及共同的探测视角。
具体实施过程中，由于各传感器安装的空间位置可能不同，本实施例可选方案可通过传感器安装的空间参数把各传感器各自的探测域(各自的探测空间)的中心轴线(X'/Y'/Z'轴)计算出来，进而以此为依据实施以上步骤S13。
其中一种举例中，由于真实系统往往有产品公差以及安装误差，本实施例可选方案还可结合几何标定的过程将各传感器各自的探测域的中心轴线检测出来，为了标定各中心轴线，可在多个确定空间位置坐标的位置上放置标靶，再用传感器测量到的这些标靶的位置读数，把标靶位置读数与它们在现实中的物理空间位置建立对应关系，从而确立该传感器探测目标空间的坐标读数与实际几何空间位置的空间对应关系。
另一实施方式中，该探测坐标系也可以为球坐标系。即，我们也可以用球坐标系把各传感器的探测域关联起来，建立一一对应关系。
例如，在球坐标系的坐标(ρ,φ,θ)中，具有三维变量，其中：
ρ可以理解为目标距离坐标系轴心的距离；
φ可以理解为目标相对z轴的角度；
θ可以理解为目标相对x轴的角度。
进而，其中的球坐标原点，可例如前文所涉及的入瞳中心，故而，映射关系可理解为目标对象对应该球坐标原点在径向方向上的空间映射关系。
具体的，欧式立体几何空间坐标与球坐标可以相互转换，其转换关系可表征如下：
x＝ρsinφcosθ
y＝ρsinφcosθ
z＝ρcosφ。
其中一种实施方式中，在变动后，图像采集传感器的探测域21与其他传感器的探测域，在图7中可利用一个参考平面中的位置关系来表达，具体的，其中的探测域22可理解为红外传感器的探测域，该红外传感器可具体为红外热成像仪，其对应的探测数据可例如热辐射温度数据；其中的探测域23可理解为雷达的探测域。
S14：根据所述图像采集传感器的光轴与入瞳中心，在变动后将各传感器的探测域投射到所述图像采集传感器的成像面所处的二维平面，得到每个传感器对应的投射区域。
请参考图10，图像采集传感器的探测域21，经投射后，可在成像面投射形成成像区域24，即图像采集传感器的投射区域，也可表征为成像区域；其他传感器的探测域，经投射后，可在成像面24投射形成投射区域，例如红外传感器的探测域22投射可得到投射区域26，雷达的探测域23投射可得到投射区域25。
各投射区域可处于成像区域24中，其他可选实施方式中，本实施例也不排除投射区域可部分处于成像区域24外。
S15：根据所述二维平面中所述投射区域与所述图像采集传感器的成像区域的位置关系，确定所述映射关系。
具体可以为：根据所述传感器的投射区域与当前的探测域之间的投射关系，以及所述二维平面中所述投射区域与所述图像采集传感器的成像区域的位置关系，确定所述映射关系。
本实施方式通过以上所涉及的变动与投射，可以将探测域统一到同一平面中，进而便于对各位置点之间的映射关系进行表征与确定。
具体实施过程中，若其他传感器的探测域发生了变动，则在步骤S15中，还可结合变动前后探测域的各位置的对应关系确定所述映射关系，即：根据所述传感器的投射区域与当前的探测域之间的投射关系、变动前后探测域的对应关系，以及所述二维平面中所述投射区域与所述图像采集传感器的成像区域的位置关系，确定所述映射关系。
步骤S121之后，可包括：S122：将所述探测数据赋值到其所对应的像素元素所纵向对应的探测数据元素。
图7是本发明实施例1中步骤S121与步骤S123的流程示意图。
其中，若在前述步骤S121中得到了目标像素信息，则在步骤S121之后，请参考图7，可以包括：
S123：根据所述目标像素信息，在所述探测数据对应的传感矩阵层中确定对应的一个目标探测数据元素。
该目标探测数据元素，可例如表征为特定传感矩阵层中的特定(x，y)坐标。
由于步骤S123确定的目标探测数据元素，在步骤S122中，可具体包括：根据所述探测数据，对其对应的目标探测数据元素进行赋值。
在赋值后，可理解为通过赋值的结果能够将目标对象的检测结果给表现出来，其也可进一步适于被识别。
图8是本发明实施例1中步骤S16的流程示意图。
其中一种实施方式中，在建立映射关系时，可以利用算法细微调整几何空间映射关系来减小几何映射定位误差，具体的，可在步骤S12之后，实施以下步骤S16，具体包括：
S161：对所述其他传感器的探测域中检测出来的任意的一个或多个参考目标对象进行定位计算，得到目标定位结果；所述目标定位结果用于表征所述参考目标对象在探测空间中的位置；
S162：将所述目标定位结果所表征的位置映射到像素矩阵层，得到校验定位信息；
S163：比对所述校验定位信息与原定位信息，得到定位误差信息；所述原定位信息用于表征对于同一所述参考目标对象，在形成多维矩阵结构时所确定的像素元素在所述像素矩阵层中的位置；
S164：根据所述定位误差信息，调整对应的所述传感矩阵层，以使得所述传感矩阵层与所述像素矩阵层的纵向对应关系发生变化；
S165：根据变化后的纵向对应关系调整所述传感矩阵层中探测数据元素的赋值。
具体可理解为：将探测域中的多个检出的目标对象在各自的传感器检测空间区域里做独立的定位计算，把最终的定位结果匹配到多维矩阵结构中，再将该匹配的结果与步骤S12中针对相同目标对象的检测结果进行比较，确定该相同目标对象的位置误差，具体可使用几何空间转换的方式计算该误差，进而，可通过缩放和平移的对应传感矩阵层与像素矩阵层的相对位置，即调整传感矩阵层中数据元素、像素矩阵层中数据元素的纵向对应关系，也可理解为调整传感矩阵层中数据元素与像素的对应关系，减少该误差，同时可对前文通过步骤S13至步骤S15确定的映射关系进行相应调整。
可见，以上实施方式在不使用所建立的映射关系的情况下，通过对目标对象的定位检测，以及将定位检测结果直接映射到对应的传感矩阵层，可以在传感矩阵层中再次表征出目标对象，其理应与利用预先建立映射关系即通过步骤S12通过赋值表现出的目标对象相一致，进而，可通过比对两者的差别表征出以上所涉及的误差，进而有效对目标对象的检测结果进行调整，还可对所建立的映射关系进行调整。
其可有利于进一步减小不同轴的几何空间转换引入的映射误差，使得数据结构的组合更精确。
在实际应用中，各个传感器应尽可能在空间结构上实现接近同轴安装，越靠近越好，这样有利于减少因为不同轴的几何空间转换引入的映射误差(结果类似于虚像)。
此外，由于初始输入时雷达和红外传感器的空间角度分辨率和图像采集传感器可能不同，即：所述其他传感器的分辨率与所述图像采集传感器的分辨率不匹配，在本实施例的可选方案中，还可对分辨率进行匹配处理，以保障各传感矩阵层中的探测数据元素能够与像素矩阵层中的图像数据元素一一对应。
以上的分辨率不一致，可理解为其他传感器的探测数据在投射到成像面所处二维平面后，在相同尺寸范围内，其行列分布与图像采集传感器的图像在该成像面中像素的行列分布不一致，具体可例如行数、列数至少之一不一致，也可以指探测数据经换算、计算等处理，且被投射到成像面后，在相同尺寸范围内数据的行列分布与成像面中像素的行列分布不一致。
为了实现该匹配，在其中一种实施方式中，可以通过像素平面定义宏块 Macroblock来实现，进而，所述映射关系具体用于表征所述传感器的探测域中不同位置的探测数据与不同宏块之间的对应关系，所述宏块包含至少两个像素。
具体实施过程中，可通过将画面分割成一个个大小预先定义好的宏块以后，再同低分辨率传感器探测的数据进行一一对应映射关系的数据匹配。当然，宏块的具体定义参数需要在数据组织里说明，例如，可在数据文件头或者说明里注明。
为了实现该匹配，在另一实施方式中，还可通过插值的方式来实现，例如，所述传感矩阵层中的数据元素还包括第一插值数据元素，和/或：所述像素矩阵层中的数据元素还包括第二插值数据元素。该些插值数据元素的数值，可理解为插值确定的。
以雷达探测到的探测数据为例，把雷达的探测数据转换成具备张量结构的密集的类图像数据，其可作为一个探测数据组，再利用几何投影的方式，将雷达的探测域投射到成像面的二维平面，其可作为雷达探测的目标对象的投影面；其中的分辨率可等于系统里与之匹配的图像采集传感器的像素分辨率，建立雷达的数据与摄像头数据逐点的对应映射关系；雷达探测到的目标对象，投影到投影面上，可生成雷达的探测数据的雷达感知矩阵结构；雷达感知矩阵的数据层可例如包括L(目标距离值)层，S(相对速度值)层，R(雷达散射截面值)层；同样，这些层的顺序可以交互，也可以灵活组合，例如L、S、R可以全部激活，也可只选其中的一种或者其中两种激活，对应的可形成一个或多个传感矩阵层；目前，毫米波雷达的空间分辨率比较低，目标的角分辨率不高导致其数值投影在目标的二维映射面上会有比较大的可能覆盖面积，类似于雷达的原始像素颗粒尺寸比图像采集传感器的像素尺寸大，分辨率低，为了对每一个多维像素的每一层传感矩阵层中的数据元素都对应赋值，我们需对投射面进行插值，即对当前的传感矩阵层进行插值，提高其分辨率使其与图像采集传感器的像素匹配，然后再对对应传感矩阵层中的每一个数据元素逐一赋值，故而，匹配后的传感矩阵层可以具有插值数据元素；由于雷达数据是稀疏的，在雷达的数据结构(L层、S层、R层等)里，在雷达有探测到目标对象的区域(等效像素位置)雷达的数据会一一对应地赋值过去，但是在没有探测到目标的区域，可将这些区域对应的雷达的原始数据赋值为：“0”，或者设为按照事先约定代表背景的缺省数值，保证雷达数据对应的传感矩阵层中的每一个数据元素均完成了赋值。
若其他传感器为红外热成像仪，其采集的图像也是以像素为单位的。通过适当的插值方式可放大图像分辨率，进而匹配摄像头的分辨率，把红外热成像仪采集的图像(一般为黑白的亮度像素信息)逐点对应赋值到对应的传感矩阵层。一般情况下红外热成像仪的分辨率低于普通摄像头分辨率。
此外，本实施例也不排除其他传感器的分辨率高于图像采集传感器的情况，例如红外热成像仪的图像分辨率高于系统安装的摄像头分辨率，该情况下除了对像素矩阵层进行插值的方案，还可以采用缩小热成像仪分辨率的方 式来处理；总之基本原理就是通过图像的合理缩放，使得两种传感器的分辨率相同然后对多维矩阵结构的对应矩阵层的数据元素赋值。
基于相类似的原因，也可对像素矩阵层中的数据元素进行插值。
关于以上所涉及的插值，可例如：最近邻插值、双线性插值、三次卷积法等。
其中一种实施方式中，所述多维矩阵结构中的矩阵层能够被有选择地激活。其中的激活，可理解为仅在激活时才能够使用到该矩阵层，且该矩阵层的数据会被更新迭代。同时，矩阵层可以是在预先编写程序中写入而存在，进而，其可以是被激活的，也可以是不被激活的。矩阵层也可以是在预先编写程序时不存在之后被添加的，还可以是根据预先定义的规则自动生成而存在的。可见，矩阵层的是否存在，以及是否被激活，可以被区分理解。
诚如前文所提及的，所述其他传感器为以下至少之一：微波雷达、超声波雷达、激光雷达，红外传感器、以及太赫兹图像传感器；所述其他传感器的探测数据包括以下至少之一：距离数据、速度数据、加速度数据、方位数据、雷达散射截面RCS数据，以及热辐射温度数据；所述图像数据矩阵包括以下至少之一：亮度数据矩阵、RGB三层数据矩阵、YUV三层数据矩阵，以及光流数据矩阵。
根据当前场景的不同，所应用的传感器，所存在的矩阵层，以及所激活的矩阵层均可以是变化的。
例如：具体举例中，可采用摄像头作为图像采集传感器，采用微波雷达以及红外热成像仪分别作为其他传感器，摄像头输出彩色图像(RGB或者YUV数据)，微波雷达输出目标对象的距离数据、相对的速度数据、例如方位角的方位数据以及目标对象的RCS数据，红外热成像仪输出目标的热辐射温度分布图像，其中各像素可理解为热辐射温度数据。该举例的组合可以从多个维度感知与探测目标，可以有效的工作在多种工况下(白天，黑夜，大雾雨天等恶劣环境)。
具体举例中，多种传感器可以灵活搭配组合，例如可以三者全用(摄像头+微波雷达+红外热成像仪)，又可以采用两两组合的方式：摄像头加微波雷达，摄像头加红外热成像仪，或者是微波雷达加红外热成像仪的组合。由于把多个维度的目标探测数据用多维矩阵结构的形式组合在一起，系统可以根据硬件配置或者场景(白天，夜晚等不同工况下)动态地调整用于目标识别的传感器输入参数的维度，即调整所激活的矩阵层，使用多维像素的子集来检测。比如：在夜晚行车中我们要检测车灯照射范围之外的目标，我们可以只激活雷达的输入矩阵而得到的传感矩阵层以及红外热成像仪的输入矩阵而得到的传感矩阵层。还可以采取硬件动态配置的方式，动态地加入或者移除一种传感器，不论何种方式，本实施例所涉及的方法均能保持工作，即系统依旧能保持工作。
该灵活组合，还可具有以下积极效果：在某种场景下系统的其中一种传感器失效(或者损坏)，可通过调整保持本实施例所涉及的方法及实施该方法 的系统仍能保持有效运作，增强了系统运行的鲁棒性。在一些应用领域，比如ADAS或者是自动驾驶应用，增加系统的鲁棒性非常必要。
此外，针对多传感器的组合，同类传感器的数量也可以是灵活配置的，例如可以采用多个摄像头、多个雷达，以及多个热成像仪。在选择“摄像头+毫米波雷达+红外热成像仪”组合之上，具体还可以引入其他种类的传感器，比如激光雷达等，新的传感器带入的探测参数也可以附加在我们的“摄像头+毫米波雷达+红外热成像仪”系统的多维矩阵结构中，成为我们多维的一部分。
不论何种组合，均可有利于使得探测的目标对象可以在超越单一传感器探测维度被检测、分类、识别，系统能有更高的目标检出率和更好的识别能力与识别质量。
由于各传感器的探测视野可能不同，它们之间的映射关系可能会部分重叠。如图10所示，所标示的A类区域是三种传感器共同探测的区域，所标示的B类区域与C类区域是雷达与摄像头共同探测的区域，所标示的D类区域与E类区域是摄像头与红外热成像仪共同探测的区域，G类区域为只有摄像头探测的区域。
其中，A类区域为三种传感器共同探索的区域，这个区域可以最充分地利用多传感器融合带来的多数据维度的精确探测，故而，该区域的传感器融合较为重要。摄像头和雷达重叠探测的区域，例如B类区域与C类区域，摄像头与雷达的探测维度互补性很强，这两种传感器的融合也较为重要。再一个重要的融合区域是摄像头与红外热成像仪重叠的区域，在这个区域里，红外热成像仪可以弥补摄像头在夜晚、大雾天等工况下的不足；但是由于两者都是产生图像信息，他们之间的信息融合，更需要前文所涉及的分辨率匹配的过程，例如需要用图像插值的方式把红外传感器的图像分辨率(像素数目)放大，或者把摄像头图像分辨率缩小，以达到相互的匹配，然后，把红外传感器的图像(我们标记为H维度)组合摄像头的RGB(或者YUV)，形成所需的多维矩阵结构。
在步骤S12之后，本实施例可选方案可以采用各种传统的特征值+分类器的方式对各层矩阵层进行分析处理，来检测目标对象；也可以采用神经网络的方式做后续的检测处理；或者两种方式混合处理。无论那种方式，由于本实施例将多维度的探测信息统一到了一个数据结构中，并以像素为单元做映射组合起来，这样的深度融合数据组合，可有效提升对目标对象的检出质量。
其中，针对神经网络，由于输入为多维矩阵结构，网络会相应产生更多层的特征图(feature map)，有了更丰富的多层多维度的特性提取，有利于更加高效、高质量地做目标的检测与定位。多维矩阵对算法的匹配性较佳，目前流行的神经网络算法，比如R-CNN、Faster R-CNN、SSD等，均可匹配适用于本实施例所涉及的多维矩阵结构的应用。
对于目标对象的检测，本实施例中如果采用机器学习的方式，这就会涉及到目标样本的采集与系统的训练。本实施例所涉及的多维度测量参数的矩阵描述方式，能够对目标对象的样本采集与训练带来便利；每层矩阵层的信 息相对是独立的，每层的结构可以增减，在后续处理中可以动态地对其中的一层或者是多层激活(这些层的信息参与目标的测定与描述)或者不激活(这些层的信息不参与目标的测定与描述)，但是总体上不妨碍以多维像素结构来对目标的探测与标记。我们建议在目标样本采集时，激活多维像素的所有矩阵数组层，但是在训练时可以根据特定的场景(白天、夜晚、传感器探测视野的重叠状态等不同情况)动态的组合其中的特定矩阵数组层的激活来用于训练，以匹配相应的场景。对于多传感器探测区域，无论是几种的组合，我们都可以利用多维矩阵结构把它们有机地结合在一起，用相同的处理算法框架来检测目标；他们的训练方式(包含数据样本的采集)也可以融合在一起，一次实现。
本实施例利用多传感器从多个探测维度来识别目标，这个多维矩阵结构能够有机结合多维度感知的探测数据，而且对后续的数据处理(无论采用传统的特征值加分类器的方法，还是神经网络的方法，或者是两者的结合)，以及训练样本的采样都带来了巨大的便利。这样的数据装配对与空间导航与定位(比如SLAM)也非常有效，具体的，利用该结构对目标对象的描述，不仅具备特征数据(用于分类识别)，还具备三维的位置空间信息(XY轴上的空间角度以及探测器到目标的距离)，目标识别的结果可以直接用来做空间定位。
本实施例可选方案中，来自不同维度的信息组合产生更多样有效的数据挖掘与特征提取潜力，不仅提供了一种更高效的事件记录方法(格式)，更有效的提高系统环境感知与目标检测能力，还可以大大节省传输数据所需的带宽和存储数据的空间，方便将有效并充分的数据流传输给不同的系统使用，同时降低系统(处理单元)做事件预测与分析的数据量与实时运算处理能力的要求，有效降低环境感知系统的成本。
由于本方法在多维矩阵结构里可以包含目标相对距离与速度信息，使用本方法可以在一帧像素对应的数据里就做出目标的意图描述以及预测事件将要发生的场景分析。
故而，本实施例及其可选方案所涉及的方法可适用于车辆驾驶辅助系统(ADAS)和自动驾驶系统、机器人、无人搬运车(AGV)，以及各种需要有环境感知与目标检测能力的设备与系统中。
若将本实施涉及的方法应用于及其学习，由于机器学习需要巨量的样本集，这些样本集还需要是针对自身多传感器组合系统量身定制采集的，本发明的数据组织方法，可以非常有针对性地用于特定组合多传感器系统采集多维度丰富有效的数据样本，非常适合多传感器融合的样本集(正、负样本集以及语义样本集)收集，而且大幅节省存储空间。当机器学习系统涉及到“云”+“端”或者“边缘计算”时，需要系统采样数据在本地与云端的传输，本可选方案涉及的方法可以实现更高效地传输多维度信息同时避免不必要的冗余数据占用数据传输带宽；另外，在一些特定领域(比如安防、监控、保险取证)的应用，系统要求用尽可能少的存储空间来保存尽可能多的信息，本发 明的数据采样与存储方法能把多维度的目标信息存储在一帧数据矩阵上(可包含目标距离和运行速度矢量这些信息)，可大幅提升取证信息的保存效率。
图11是本发明实施例1中多传感器融合的数据处理装置的结构示意图。
请参考图11，多传感器融合的数据处理装置300，包括：
获取模块301，用于获取目标对象的图像数据与至少一组探测数据组；所述图像数据是图像采集传感器探测到的，所述探测数据组是其他传感器探测到的；所述图像数据用于利用至少一个图像数据矩阵表征所述图像采集传感器采集到的目标图像；不同探测数据组为不同探测维度的探测数据；
形成模块302，用于形成多维矩阵结构；其中：
所述多维矩阵结构包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层，每个像素矩阵层对应用于表征一个图像数据矩阵，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据组中的每个探测数据元素纵向对应于所述像素矩阵层中的一个像素元素；所述探测数据元素的数值均是根据探测数据赋值确定的。
所述形成模块，具体用于：
根据已建立的映射关系，确定所述目标对象的每一个探测数据对应的所述目标对象的像素元素，所述映射关系用于表征所述其他传感器的探测域中不同探测维度的不同位置的探测数据与不同像素元素之间的映射关系；
将所述探测数据赋值到其所对应的像素元素所纵向对应的探测数据元素。
可选的，建立所述映射关系的过程，包括：
通过数据处理变动各传感器的探测坐标系，并使得各传感器的探测域的中心轴线与所述图像采集传感器的光轴一致，以及：各传感器的探测域的探测顶点与所述图像采集传感器的入瞳中心重合；
根据所述图像采集传感器的光轴与入瞳中心，在变动后将各传感器的探测域投射到所述图像采集传感器的成像面所处的二维平面，得到每个传感器对应的投射区域；
根据所述传感器的投射区域与当前的探测域之间的投射关系，以及所述二维平面中所述投射区域与所述图像采集传感器的成像区域的位置关系，确定所述映射关系。
可选的，所述探测坐标系为空间直角坐标系，或者球坐标系。
可选的，所述映射关系具体用于表征所述传感器的探测域中不同位置的探测数据与不同的单个像素之间的对应关系，或者：
所述其他传感器的分辨率与所述图像采集传感器的分辨率不匹配，所述映射关系具体用于表征所述传感器的探测域中不同位置的探测数据与不同宏块之间的对应关系，所述宏块包含至少两个像素。
可选的，若所述其他传感器的分辨率与所述图像采集传感器的分辨率不匹配；则：
所述传感矩阵层中的数据元素还包括第一插值数据元素，和/或：所述像 素矩阵层中的数据元素还包括第二插值数据元素。
可选的，所述探测数据元素的数值为对应的探测数据本身，或者所述探测数据元素的数值是根据对应的探测数据换算确定的。
可选的，所述多维矩阵结构中的矩阵层能够被有选择地激活。
可选的，所述其他传感器为以下至少之一：微波雷达、超声波雷达、激光雷达、红外传感器，以及太赫兹成像传感器；所述其他传感器的探测数据包括以下至少之一：距离数据、速度数据、加速度数据、方位数据、雷达散射截面RCS数据，以及热辐射温度数据；
所述像素数据矩阵包括以下至少之一：亮度数据矩阵、灰度数据矩阵、RGB三层数据矩阵、R层数据矩阵、G层数据矩阵、B层数据矩阵、YUV三层数据矩阵Y层数据矩阵、U层数据矩阵、V层数据矩阵，以及光流数据矩阵。
本实施例提供的多传感器融合的数据处理装置，能够将不同传感器测得的多个不同维度的数据以像素元素为基础，用多维矩阵结构的形式组合在一起，进而，可有利于对取得的数据做多层面的数据融合与深度学习，其可有利于实现更多样更有效的数据挖掘与特征提取，从而产生更有效的环境感知与目标检测的能力。
实施例2
本实施例提供了一种多传感器融合方法，包括：
把来自多传感器的多个维度的探测数据用多维像素矩阵的形式组合在一起，建立以摄像头像素为颗粒度的立体多维深度感知矩阵数组；
在多维像素矩阵中，把每个像素包含的信息做了纵向扩展，除了其原本包含的亮度与颜色信息，还为每个像素增加了多个纵向维度，在增加的纵向维度上能够输入该像素在摄像头探测空间映射的目标对象被其它传感器探测到的多种对应维度的探测信息，所述探测信息包括以下至少之一：相对距离、相对运动速度、目标的雷达散射截面RCS数据以及目标的热辐射温度分布等数据；其中，把多维度的探测信息以分层的方式装配到原本以图像像素为单元的目标对象描述之上，产生多维像素，其在数学上表现为统一结构的矩阵数组，以使得原来的每一个像素变成了一个多维像素，得到多维像素矩阵。其中的多维像素矩阵，可理解为因其是以像素为单元的，故而，表述成多维像素矩阵，其实际是由多个矩阵装配在一起的，可见，其可对应理解为实施例1所涉及的多维矩阵结构，即多维像素矩阵与多维矩阵结构是表征同一含义的。
可见，本实施例把多个维度的目标探测数据用矩阵数组(类似一个立体矩阵)的形式组合在一起。在摄像头成像的二维像面空间基础上，本实施例把每个像素包含的信息做了扩展，除了其原本包含的亮度与颜色信息，本实施例还为每个像素增加了多个纵向维度，在增加的纵向维度上输入该像素在摄像头探测空间(物方空间)映射的目标物体单元被其它传感器探测到的多种对应维度的信息，如相对距离、相对运动速度、目标的雷达散射截面RCS 数据以及目标的热辐射温度分布等数据，把多维度信息以分层的方式装配到原本以图像像素为单元的目标物体描述子之上，在数学上表现为统一结构的矩阵数组。在本文中本实施例把这种目标的“多维度测量参数”的矩阵数组描述称之为“多维像素”结构。在摄像头成像的每个像素为基础维度上增加了其它传感器如雷达和红外传感器带来的距离、相对速度、目标的雷达散射截面RCS数据以及目标的热辐射温度分布等数据，增加系统感知深度，建立了以摄像头像素为颗粒度的立体多维深度感知矩阵数组，原来的每一个像素变成了本实施例中的每一个多维像素。本实施例把不同维度的探测数据用矩阵数组的形式组合在一起，组成一个“多维度测量参数”的矩阵数组，简称多维像素矩阵。
“多维度测量参数”的矩阵数组(既多维像素矩阵)描述示意图2至图4所示。本实施例还可以在此基础上增加更多的数据维度(带入更多传感器的数据封装)，组合方式相同。另外，纵向的矩阵数组顺序可以改变(当然，顺序的改变可能意味着机器学习要重新再次训练)。
以上描述，结合图2和图3的示意，可以毫无疑义地得到方案的实施需获取到图像数据与探测数据组，进而需基于图像数据与探测数据组得到多维矩阵结构，其中图像数据是利用像素矩阵表征的；进而，可得到包含多个矩阵维度的多维矩阵结构，该多维矩阵结构即以上所涉及的多维像素矩阵，多维像素矩阵必然包含探测数据对应的传感矩阵层，即以上所涉及的纵向维度上分层的内容，多维像素矩阵必然也包含像素对应的像素矩阵层，即以上所涉及的原本包含的亮度与颜色信息的像素，其中的每个像素可表述为像素元素，而像素元素与探测数据元素必然是纵向对应的，综上可见，通过以上描述，可毫无疑义地推知实施例1中图1所示的步骤S11与步骤S12的内容。
在有些特殊情况下，本实施例会只有单色摄像头参与到多传感器的组合中，比如摄像头只用于红外成像场景，摄像头只输出单色图像；在这种情况下，本实施例的多维像素结构仍然有效，只是RGB(或者YUV)三层输入数据更改为单层Y(像素亮度)数据结构，在单层像素亮度的基础上用上述相同的多维像素结构方法来组合其它传感器的输入信息，产生“多维度测量参数”的矩阵数组，既多维像素矩阵。
即：所述摄像头为彩色摄像头或者单色摄像头；
若所述摄像头为彩色摄像头，则：彩色摄像头输出的数据矩阵为RGB或者YUV三层，所述多维像素矩阵是将其他传感器的探测信息封装在这三层数据矩阵之上得到的；
若所述摄像头为单色摄像头，则：摄像头只输出单色图像，所述多维像素矩阵是在单层像素亮度的基础上组合其它传感器的探测信息得到的。
多维像素的横截面坐标对等于摄像头像平面的像素坐标，因为多维像素是基于摄像头像平面的像素信息做了扩展，每个像素增加了多个纵向维度的信息组合而来；本实施例把从摄像头像素平面坐标位置为(x,y)的像素(既“像素(x,y)”)扩展而来的多维像素的个体称之为“多维像素(x,y)”。
对于雷达等其他传感器带来的目标相对距离、相对速度、目标的雷达散射截面RCS数据，本实施例可直接把相对距离、相对速度、目标的雷达散射截面RCS数据值直接赋值到各多维像素对应的矩阵层上，也可以把这些赋值先经过相应的公式计算后把计算结果再赋值到各多维像素对应的矩阵层上，比如：本实施例希望在多维像素的R(RCS)层输入的是RCS映射到雷达接收功率P的相应值，那本实施例可以先将对应多维像素横截面坐标位置为(x,y)上的“多维像素(x,y)”之RCS值先做如下计算：P(x，y)＝RCS(x，y)/(L(x，y) 4),其中，RCS(x,y)是指“多维像素(x,y)”的RCS值，L(x,y)是指“多维像素(x,y)”中雷达探测到目标的相对距离值。所以，多维像素在每个像素为基础维度上增加了其他传感器如雷达和红外传感器带来的距离、相对速度、目标的雷达散射截面RCS数据以及目标的热辐射温度分布图像等数据，既可以是探测数据的直接赋值，也可以是这些数据经过函数换算后再赋值，如图2示意图所示。
系统数据处理的流程以及数据结构如下(多层结构的矩阵数组)：首先，本实施例要把来自不同传感器的探测目标数据组合起来。在本实施例里第一步本实施例先使用几何空间转换的方式在标准的三维欧氏立体几何空间(空间坐标系标注为X/Y/Z轴)把各传感器的探测域(数据探测采集的物理空间区域)关联起来，建立一一对应关系。
由于各传感器安装的空间位置可能不同，本实施例通过传感器安装的空间参数把各传感器各自的探测域(各自的探测空间)的中心轴线(X'/Y'/Z'轴)计算出来，然后通过对各自坐标系的平移、旋转和缩放，把它们统一到摄像头的3维探测空间(物方视场)坐标里，使得其与摄像头的光轴对齐一致，以确定系统的统一探测空间以及共同的探测视角，然后根据雷达、红外热成像仪等其它传感器探测域在摄像头成像面对应之2维物面上建立映射关系(2维物面在物方空间)。
进而，把它们探测到的目标根据此映射关系与摄像头成像的各像素建立对接，将目标探测数据一一对应地赋值到多维像素矩阵中的相应位置上。
通过以上描述，可以毫无疑义地获悉，由于映射关系是建立在探测域中与物面的，必然表征探测域中各点与物面各点的映射关系，故而，对映射关系的使用必然是用其确定探测域中的探测数据映射到那个像素，即其可毫无疑义地推知需实施步骤S121，同时，由于赋值必然是需赋值到传感矩阵层的，所以，其可毫无疑义地推知需实施步骤S122，故而，通过以上内容，可毫无疑义地推知图5所示步骤S121与步骤S122的内容。
由于真实系统往往有产品公差以及安装误差，本实施例也可结合几何标定的方法把各传感器各自的探测域(各自的探测空间)的中心轴线(X'/Y'/Z'轴)检测出来，再通过对各自坐标系的平移、旋转和缩放，把它们统一到同一个坐标系里，建立起系统的统一探测域(三维的几何空间)，把各传感器各自独立的探测区域在系统的统一区域里建立起一一对应关系。(几何标定的方法是：本实施例在多个确定空间位置坐标的位置上放置标靶，再用传感 器测量到的这些标靶的位置读数，把标靶位置读数与它们在现实中的物理空间位置建立映射关系，从而确立该传感器探测目标空间的坐标读数与实际几何空间位置的对应关系)。
本实施例也可以再进一步，利用算法细微调整几何空间映射关系来减小几何映射定位误差。原理如下：在系统最终目标检测出来后，本实施例把探测域中的多个检出目标在各自的传感器检测空间区域里做独立的定位计算，把最终的结果映射到“多维像素”的对应矩阵数组(层)上，在这个维度(当前层)去比较(计算)与之前使用几何空间转换方式产生的相同目标之定位位置的误差(本实施例称之为“几何映射定位误差”)，然后缩放和平移的对应矩阵数组(层)与摄像头的像素图层的相对位置，减少误差，即减少“几何映射定位误差”，同时对“多维像素”的对应数据矩阵数组(层)里的值按照新的像素纵向对应关系做调整。这样本实施例可以进一步减小不同轴的几何空间转换引入的映射误差，使得“多维像素”矩阵数组组合地更精确。
通过以上描述，可以毫无疑义地获悉，误差的确定，是通过对对象进行定位计算，再将定位结果映射到像素矩阵层中，将之与利用映射关系映射识别同一对象的结果进行比较，进而得到的，可见，以上实施方式的内容可毫无疑义地推知实施例1中步骤S16的方案。
建立起系统的统一探测域后，多传感器组合的探测域空间投射关系如图9所示。由于本实施例的多维像素的组合是“每个像素增加了多个纵向维度，在增加的纵向维度上输入该像素在摄像头探测空间(物方空间)映射的目标物体单元被其它传感器探测到的多种对应维度的信息”，所以把其它各种传感器的探测空间统一映射到摄像头探测空间上与其光轴对齐一致，并且其它各种传感器探测域的顶点与摄像头的入瞳中心重合，参照图9和图10，本实施例可选方案可采用了摄像头+雷达+红外热成像仪的组合来做示意，这是一种比较典型的组合，如果再加入其他传感器，多维像素数据映射的原理也是一样的。
在实际应用中，各个传感器应尽可能在空间结构上实现接近同轴安装，越靠近越好，这样有利于减少因为不同轴的几何空间转换引入的映射误差(结果类似于虚像)。
由于初始输入时雷达和红外传感器的空间角度分辨率和摄像头可能不同(目前雷达的空间角度分辨率还不高)，在数据装配的时候本实施例会采用插值的方法解决这个问题。
当多维像素对应的矩阵数组生成后，后续本实施例可以采用各种传统的特征值+分类器的方式对各层数据进行分析处理，来检测目标；也可以采用神经网络的方式做后续的检测处理；或者两种方式混合处理。无论那种方式，由于本实施例把多维度的目标探测信息统一到了一个矩阵数组中，以像素为单元做映射组合起来，这样的深度融合数据组合，对目标的检出质量的提升有巨大的帮助。特别是针对神经网络，由于输入多维像素矩阵，网络会相应产生更多层的feature map，有了更丰富的多层多维度的特性提取，本实施例 可以更加高效高质量地做目标的检测与定位。多维像素矩阵对算法的匹配性非常好，目前流行的神经网络算法，比如R-CNN、Faster R-CNN、SSD等，对应本实施例的多维像素矩阵输入(多层输入)做相应的改动即可适用。
本实施例对于目标的检测往往采用机器学习的方式，这就会涉及到目标样本的采集与系统的训练。本实施例之多维度测量参数的矩阵数组描述方式(多维像素)对于目标样本的采集与训练带来便利；多维像素每层的信息相对是独立的，每层的结构可以增减，在后续处理中可以动态地对其中的一层或者是多层激活(这些层的信息参与目标的测定与描述)或者不激活(这些层的信息不参与目标的测定与描述)，但是总体上不妨碍以多维像素结构来对目标的探测与标记。本实施例建议在目标样本采集时，激活多维像素的所有矩阵数组层，但是在训练时可以根据特定的场景(白天、夜晚、传感器探测视野的重叠状态等不同情况)动态的组合其中的特定矩阵数组层的激活来用于训练，以匹配相应的场景。对于多传感器探测区域，无论是几种的组合，本实施例都可以利用“多维像素”矩阵数组把它们有机地结合在一起，用相同的处理算法框架来检测目标；他们的训练方式(包含数据样本的采集)也可以融合在一起，一次实现。这是本实施例的另一个技术特点，也是使用“多维像素”结构来描述目标物体的方法带来的好处。
本实施例采用摄像头，微波雷达以及红外热成像仪组成多传感器融合系统。这是一种常用的多传感器组合，摄像头输出彩色图像(RGB或者YUV数据)，微波雷达输出探测目标的距离、相对速度、方位角以及目标的雷达散射截面RCS(Radar Cross-Section)数据，红外热成像仪输出目标的热辐射温度分布图像。这样的组合可以从多个维度感知与探测目标，可以有效的工作在多种工况下(白天，黑夜，大雾雨天等恶劣环境)。在本实施例里，多种传感器可以灵活搭配组合，所以在这里可以一个系统里三者全用(摄像头+微波雷达+红外热成像仪)，又可以采用两两组合的方式：摄像头加微波雷达，摄像头加红外热成像仪，甚至是微波雷达加红外热成像仪的组合。由于把多个维度的目标探测数据用矩阵数组(类似一个立体矩阵)的形式组合在一起，系统可以根据硬件配置或者场景(白天，夜晚等不同工况下)动态地调整用于目标识别的传感器输入参数的维度(调整多维度目标探测数据矩阵数组的激活层数)，使用多维像素的子集来检测，比如在夜晚行车中本实施例要检测车灯照射范围之外的目标，本实施例可以只激活雷达的输入矩阵以及红外热成像仪的输入矩阵。甚至还可以采取硬件动态配置的方式，动态地加入或者移除一种传感器，系统仍能工作。系统硬件的灵活组合能力，除了提供系统硬件配置成本选择的灵活性之外，还能给用户带来的好处是：在某种场景下系统的其中一种传感器失效(或者损坏)，系统通过软件配置的调整仍能保持有效运作，增强了系统运行的鲁棒性。在一些应用领域，比如ADAS或者是自动驾驶应用，增加系统的鲁棒性非常必要。
以上有关激活的描述，必然可得到部分矩阵层被激活，部分未被激活的结果，故而，可以毫无疑义推知矩阵层能够被有选择地激活的结论。
由于各传感器的探测视野可能不同，它们之间的映射关系可能会部分重叠。如图10所示，外面的单实线框表示摄像头成像区域，里面的单实线框内是红外热成像仪成像区域，虚线框内标识区域是雷达探测区域，它们部分重叠组成交错的探测区域：1)A类区域是三种传感器共同探测的区域，2)B与C类区域是雷达与摄像头共同探测的区域，3)D与E类区域是摄像头与红外热成像仪共同探测的区域，4)G类区域为只有摄像头探测的区域。本实施例最关心的是三种传感器共同探索的区域(A类区域)，这个区域可以最充分地利用多传感器融合带来的多数据维度的精确探测。第二个重要的区域是摄像头和毫米波雷达重叠的区域(B与C类区域)，摄像头与毫米波雷达的探测维度互补性很强，这两种传感器的融合也非常有意义。第三个重要的融合区域是摄像头与红外热成像仪重叠的区域，在这个区域里，红外热成像仪可以弥补摄像头在夜晚、大雾天等工况下的不足；但是由于两者都是产生图像信息，他们之间的信息融合，更多的技术挑战来自与分辨率上面的匹配，需要用图像插值的方式把红外传感器的图像分辨率(像素数目)放大，或者把摄像头图像分辨率缩小，以达到相互的匹配，然后，把红外传感器的图像(本实施例标记为H维度)附加到有摄像头的RGB(或者YUV)数据结构的矩阵数组中。
在A区域(三种传感器共同探测到的区域)，本实施例把摄像头把采集到的数据，按照RGB颜色(顺序可以交换)三层排列，假设每层的像数尺寸(既分辨率)为X*Y(例如：1920*1080–对应1080P分辨率的摄像头)，如果原始数据输入是YUV格式，也可以按照YUV三层排列，但是本实施例建议最好转换为RGB数据结构(YUV转RGB)，因为这样可以减少各层数据的关联，有利于后续独立的特征提取。本实施例把这三层立体的数据结构(大小为：X*Y*3)作为“摄像头的原始数据输入层”，然后，在摄像头的原始数据输入层上按照多维像素的结构加入微波雷达采集的数据。
雷达采集的数据如果和摄像头像素数据直接匹配的话，雷达数据过于稀疏，如果要同摄像头影像数据按照像素关系来直接逐点匹配，需要先做处理，把雷达数据转换成具备张量结构的密集的类图像数据。在本实施例里本实施例设计了如下的方法把雷达的数据输入到本实施例的系统“多维像素矩阵”：1)利用几何投影的方式，把雷达的探测空间投影到摄像头成像面对应的2维物面之上，作为雷达目标的2维映射面(如图3示意图所示)；其2维空间分辨率等于系统里与之匹配的摄像头的像素分辨率，建立雷达的数据与摄像头数据逐点的一一对应映射关系；雷达探测到的目标(像)，映射到雷达目标的2维映射面上，生成“雷达感知矩阵”；在矩阵层(深度)上雷达感知矩阵的数据(层)由以下“雷达原始数据输入”分门别类复合组成：L(目标距离值)层，S(相对速度值)层，R(雷达散射截面值)层；同样，这些层的顺序可以交互，也可以灵活组合(L、S、R全部激活),或者只选其中的一种(L或者S或者R)或者其中2种(L+S、S+R等)。目前毫米波雷达的空间分辨率比较低，目标的角分辨率不高导致其数值投影在目标的2维映射面 上会有比较大的“可能覆盖面积”，类似于雷达的原始像素颗粒尺寸比摄像头的像素尺寸大，分辨率低，为了把每一个“多维像素”的每一层数据矩阵都对应赋值，本实施例需要对“雷达二维映射面”插值，提高其分辨率使其与摄像头的像素匹配，然后再对每一个“多维像素”都逐一赋值。常用的插值方法，比如：最近邻插值、双线性插值、三次卷积法等都可以采用。2)由于雷达数据是稀疏的，在雷达的数据结构(矩阵数据层L、S、R等)里，在雷达有探测到目标的区域，雷达的数据会一一对应地赋值过去。但是在没有探测到目标的区域，本实施例把这些区域对应的雷达的原始数据赋值为“0”，或者按照事先设定的代表背景的缺省数值，以保证雷达数据矩阵中的每一个矩阵单元都有赋值。
红外热成像仪采集的图像也是以像素为单位的。本实施例通过适当的插值方式放大图像分辨率来匹配摄像头的分辨率，把红外热成像仪采集的图像(一般为黑白的亮度像素信息)逐点对应赋值到“多维像素”数据结构里对应的矩阵，在这里本实施例把它称为“H”矩阵。一般情况下红外热成像仪的分辨率低于普通摄像头分辨率，本实施例用插值放大分辨率的方式来处理；当然也不排除特殊情况，红外热成像仪的图像分辨率高于系统安装的摄像头分辨率，这样的话本实施例可以采用缩小热成像仪分辨率的方式来处理；总之基本原理就是通过图像的合理缩放，使得两种传感器的分辨率相同然后对多维像素的对应数据层赋值。
通过以上列举可知，在插值后，矩阵中必然就包含了插值数据元素，即其可毫无疑义地得到在矩阵中可包含插值数据元素的内容。
针对多传感器的组合，目前本实施例选择最通用的摄像头+毫米波雷达+红外热成像仪；这个组合是灵活多样的，可以三种传感器都选择组成一个系统，也可以选择其中的两样传感器(摄像头+其他)组成一个系统，传感器的数量也是灵活配置，可以是多个摄像头加多个雷达加多个热成像仪组成一套系统。但是各种组合不变的原则是：摄像头输出彩色图像(RGB或者YUV数据)，微波雷达输出探测目标的距离、相对速度、方位角以及目标的雷达散射截面RCS数据，红外热成像仪输出目标的温度分布图像，本实施例把这些物理探测维度映射到探测目标上，使得探测目标可以在超越单一传感器探测维度被检测、分类、识别，系统能有更高的目标检出率和更好的识别能力与识别质量。
在本实施例的系统里，在选择“摄像头+毫米波雷达+红外热成像仪”组合之上，本实施例还可以引入其他种类的传感器，比如激光雷达等，新的传感器带入的探测参数也可以附加在本实施例的“摄像头+毫米波雷达+红外热成像仪”系统的数据结构组合中，成为本实施例“多维像素”的一部分。
本实施例中，本实施例使用多传感器从多个探测维度来识别目标，本实施例把多个维度的目标探测数据用矩阵数组(类似一个立体矩阵)的形式组合在一起，这个数据的矩阵数组的每一个“多维像素”都有机结合了多维度感知的探测信息，而且在一个单元里(一个“多维像素”里)，这样的结构 对后续的数据处理(无论采用传统的特征值加分类器的方法，还是神经网络的方法，或者是两者的结合)，以及训练样本的采样都带来了巨大的便利。这样的数据装配对与空间导航与定位(比如SLAM)也非常有效，因为在本实施例这种“多维像素”描述体系里，目标物体的直接描述不仅具备特征数据(用于分类识别)，还具备三维的位置空间信息(XY轴上的空间角度以及探测器到目标的距离)，目标识别的结果可以直接用来做空间定位。
本实施例适用于车辆驾驶辅助系统(ADAS)和自动驾驶系统、机器人、无人搬运车(AGV)，以及各种需要有环境感知与目标检测能力的设备与系统中。
图12是本发明一实施例中电子设备的结构示意图。
请参考图12，电子设备40，可以包括存储器42与处理器41。所述存储器42，用于存储感知数据，中间运行数据，系统输出数据和所述处理器41的可执行指令。所述处理器41配置为经由执行所述可执行指令来执行实施例1与实施例2及其可选方案所涉及的方法。
其中，存储器42与处理器41可通过总线43进行通讯。
图13是本发明一实施例中传感设备的结构示意图。
请参考图13，传感设备50，包括存储器52、处理器51与传感器54；所述存储器52，用于存储感知数据，中间运行数据，系统输出数据和所述处理器51的可执行指令；所述处理器51配置为经由执行所述可执行指令来执行实施例1与实施例2及其可选方案所涉及的方法。
其中，存储器52、处理器51与传感器54可通过总线53进行通讯。
本发明一实施例还提供了一种存储介质，其上存储有感知数据，中间运行数据，系统输出数据和程序，该程序被处理器执行时实现实施例1与实施例2及其可选方案所涉及的方法。
最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (10)

  1. 一种多传感器融合的数据处理方法，其特征在于，包括：
    获取目标对象的图像数据与至少一组探测数据组；所述图像数据是图像采集传感器探测到的，所述探测数据组是其他传感器探测到的；所述图像数据用于利用至少一个像素数据矩阵表征所述图像采集传感器采集到的目标图像；不同探测数据组为不同探测维度的探测数据；
    形成多维矩阵结构；其中：
    所述多维矩阵结构包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层，每个像素矩阵层对应用于表征一个像素数据矩阵，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据组中的探测数据元素纵向对应于所述像素矩阵层中的像素元素；所述探测数据元素的数值均是根据探测数据赋值确定的。
  2. 根据权利要求1所述的方法，其特征在于，所述形成多维矩阵结构，包括：
    根据已建立的映射关系，确定所述目标对象的每一个探测数据对应的所述目标对象的像素元素，所述映射关系用于表征所述其他传感器的探测域中不同探测维度的不同位置的探测数据与不同像素元素之间的映射关系；
    将所述探测数据赋值到其所对应的像素元素所纵向对应的探测数据元素；
    建立所述映射关系的过程，包括：
    通过数据处理变动各传感器的探测坐标系，并使得各传感器的探测域的中心轴线与所述图像采集传感器的光轴一致，以及：各传感器的探测域的探测顶点与所述图像采集传感器的入瞳中心重合；
    根据所述图像采集传感器的光轴与入瞳中心，在变动后将各传感器的探测域投射到所述图像采集传感器的成像面所处的二维平面，得到每个传感器对应的投射区域；
    根据所述二维平面中所述投射区域与所述图像采集传感器的成像区域的位置关系，确定所述映射关系。
  3. 根据权利要求1所述的方法，其特征在于，若所述其他传感器的分辨率与所述图像采集传感器的分辨率不匹配；则：
    所述传感矩阵层中的数据元素还包括第一插值数据元素，和/或：所述像素矩阵层中的数据元素还包括第二插值数据元素；
    所述探测数据元素的数值为对应的探测数据本身，或者所述探测数据元素的数值是根据对应的探测数据换算确定的；
    所述多维矩阵结构中的矩阵层能够被有选择地激活。
  4. 根据权利要求1所述的方法，其特征在于，所述形成多维矩阵结构之后，还包括：
    对所述其他传感器的探测域中检测出来的任意的一个或多个参考目标对象进行定位计算，得到目标定位结果；所述目标定位结果用于表征所述参考目标对象在探测空间中的位置；
    将所述目标定位结果所表征的位置映射到像素矩阵层，得到校验定位信息；
    比对所述校验定位信息与原定位信息，得到定位误差信息；所述原定位信息用于表征对于同一所述参考目标对象，在形成多维矩阵结构时所确定的像素元素在所述像素矩阵层中的位置；
    根据所述定位误差信息，调整对应的所述传感矩阵层，以使得所述传感矩阵层与所述像素矩阵层的纵向对应关系发生变化；
    根据变化后的纵向对应关系调整所述传感矩阵层中探测数据元素的赋值。
  5. 根据权利要求1至4任一项所述的方法，其特征在于，所述其他传感器为以下至少之一：微波雷达、超声波雷达、激光雷达、红外传感器，以及太赫兹成像传感器；所述其他传感器的探测数据包括以下至少之一：距离数据、速度数据、加速度数据、方位数据、雷达散射截面RCS数据，以及热辐射温度数据；
    所述像素数据矩阵包括以下至少之一：亮度数据矩阵、灰度数据矩阵、RGB三层数据矩阵、R层数据矩阵、G层数据矩阵、B层数据矩阵、YUV三层数据矩阵，Y层数据矩阵、U层数据矩阵、V层数据矩阵以及光流数据矩阵。
  6. 一种多传感器融合的数据处理装置，其特征在于，包括：
    获取模块，用于获取目标对象的图像数据与至少一组探测数据组；所述 图像数据是图像采集传感器探测到的，所述探测数据组是其他传感器探测到的；所述图像数据用于利用至少一个像素数据矩阵表征所述图像采集传感器采集到的目标图像；不同探测数据组为不同探测维度的探测数据；
    形成模块，用于形成多维矩阵结构；其中：
    所述多维矩阵结构包括纵向分布的多个矩阵层，所述多个矩阵层包括至少一层像素矩阵层与至少一层传感矩阵层，每个像素矩阵层对应用于表征一个像素数据矩阵，每个所述传感矩阵层用于表征一组探测数据组，所述探测数据组中的探测数据元素纵向对应于所述像素矩阵层中的像素元素；所述探测数据元素的数值均是根据探测数据赋值确定的。
  7. 一种多传感器融合方法，其特征在于：包括：
    把来自多传感器的多个维度的探测数据用多维像素矩阵的形式组合在一起，建立以摄像头像素为颗粒度的立体多维深度感知矩阵数组；
    在多维像素矩阵中，把每个像素包含的信息做了纵向扩展，除了其原本包含的亮度与颜色信息，还为每个像素增加了多个纵向维度，在增加的纵向维度上能够输入该像素在摄像头探测空间映射的目标对象被其它传感器探测到的多种对应维度的探测信息，所述探测信息包括以下至少之一：相对距离、相对运动速度、目标的雷达散射截面RCS数据以及目标的热辐射温度分布等数据；其中，把多维度的探测信息以分层的方式装配到原本以图像像素为单元的目标对象描述之上，产生多维像素，其在数学上表现为统一结构的矩阵数组，以使得原来的每一个像素变成了一个多维像素，得到多维像素矩阵。
  8. 根据权利要求1所述的多传感器融合方法，其特征在于，所述摄像头为彩色摄像头或者单色摄像头；
    若所述摄像头为彩色摄像头，则：彩色摄像头输出的数据矩阵为RGB或者YUV三层，所述多维像素矩阵是将其他传感器的探测信息封装在这三层数据矩阵之上得到的；
    若所述摄像头为单色摄像头，则：摄像头只输出单色图像，所述多维像素矩阵是在单层像素亮度的基础上组合其它传感器的探测信息得到的。
  9. 根据权利要求1所述的多传感器融合方法，其特征在于，把来自多传感器的多个维度的探测数据用多维像素矩阵的形式组合在一起，建立以摄像头像素为颗粒度的立体多维深度感知矩阵数组之前，还包括：
    建立起系统的统一探测域并将多传感器组合的探测空间映射关系经由空间坐标变换，统一转换到摄像头的三维探测空间上，其具体为：使用几何空间转换的方式在标准的三维欧氏立体几何空间把各传感器的探测域关联起来，其中：
    通过对各自坐标系的平移、旋转和缩放，把它们统一到摄像头的3维探测空间坐标里，使得其与摄像头的光轴对齐一致，以确立系统的统一探测空间以及共同的探测视角；
    然后根据除摄像头以外其它传感器探测域在摄像头成像面对应之2维物面上的映射关系；
    把来自多传感器的多个维度的探测数据用多维像素矩阵的形式组合在一起，建立以摄像头像素为颗粒度的立体多维深度感知矩阵数组，包括：
    把其他传感器探测到的目标对象根据所述映射关系与摄像头成像的各个像素建立对应关系，将像素在摄像头探测空间映射的目标对象被其它传感器探测到的多种对应维度的探测信息一一对应地赋值到多维像素矩阵中的相应位置上。
  10. 根据权利要求1所述的多传感器融合方法，其特征在于，在目标对象检测出来后，把探测域中的多个检出目标在各自的传感器检测空间区域里做独立的定位计算，把最终的结果映射到多维像素的对应矩阵数组上，在这个维度去比较与之前使用几何空间转换方式产生的相同目标之定位位置的误差，然后缩放和平移的对应矩阵数组与摄像头的像素图层的相对位置，减少误差，同时对多维像素的对应数据矩阵数组(层)里的值按照新的像素纵向对应关系做调整。
PCT/CN2019/078001 2018-03-29 2019-03-13 多传感器融合的数据处理方法、装置与多传感器融合方法 WO2019184709A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19776961.5A EP3779867A4 (en) 2018-03-29 2019-03-13 Data processing method and device based on multi-sensor fusion, and multi-sensor fusion method
US17/040,191 US11675068B2 (en) 2018-03-29 2019-03-13 Data processing method and device based on multi-sensor fusion, and multi-sensor fusion method

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201810280358.2A CN108663677A (zh) 2018-03-29 2018-03-29 一种多传感器深度融合提高目标检测能力的方法
CN201810280358.2 2018-03-29
CN201910127976.8 2019-02-20
CN201910127976.8A CN109655825B (zh) 2018-03-29 2019-02-20 多传感器融合的数据处理方法、装置与多传感器融合方法

Publications (1)

Publication Number Publication Date
WO2019184709A1 true WO2019184709A1 (zh) 2019-10-03

Family

ID=63782936

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/078001 WO2019184709A1 (zh) 2018-03-29 2019-03-13 多传感器融合的数据处理方法、装置与多传感器融合方法

Country Status (4)

Country Link
US (1) US11675068B2 (zh)
EP (1) EP3779867A4 (zh)
CN (2) CN108663677A (zh)
WO (1) WO2019184709A1 (zh)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111060881A (zh) * 2020-01-10 2020-04-24 湖南大学 一种毫米波雷达外参数在线标定方法
CN111324594A (zh) * 2020-02-17 2020-06-23 武汉轻工大学 用于粮食加工业的数据融合方法、装置、设备及存储介质
CN112180373A (zh) * 2020-09-18 2021-01-05 纵目科技（上海）股份有限公司 一种多传感器融合的智能泊车系统和方法
CN112285709A (zh) * 2020-05-19 2021-01-29 陕西理工大学 基于深度学习的大气臭氧遥感激光雷达数据融合方法
CN112689842A (zh) * 2020-03-26 2021-04-20 华为技术有限公司 一种目标检测方法以及装置
CN112911265A (zh) * 2021-02-01 2021-06-04 北京都视科技有限公司 融合处理器及融合处理系统
CN113205559A (zh) * 2021-04-12 2021-08-03 华中科技大学 一种粉末床熔融的红外热像仪标定方法
CN115374880A (zh) * 2022-10-10 2022-11-22 北京邮电大学 一种面向海上目标识别的多级增量数据融合系统
CN116047463A (zh) * 2023-04-03 2023-05-02 中国科学院空天信息创新研究院 多角度sar目标散射各向异性推演方法、装置、设备及介质
CN116579965A (zh) * 2023-05-22 2023-08-11 北京拙河科技有限公司 一种多图像融合方法及装置
CN116823674A (zh) * 2023-08-24 2023-09-29 湖南省水务规划设计院有限公司 一种跨模态融合的水下图像增强方法
CN116883998A (zh) * 2023-06-20 2023-10-13 珠海微度芯创科技有限责任公司 基于毫米波图像的物品标注方法、装置、电子设备

Families Citing this family (68)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3367659A1 (en) * 2017-02-28 2018-08-29 Thomson Licensing Hue changing color gamut mapping
CN108663677A (zh) 2018-03-29 2018-10-16 上海智瞳通科技有限公司 一种多传感器深度融合提高目标检测能力的方法
CN109522951A (zh) * 2018-11-09 2019-03-26 上海智瞳通科技有限公司 一种环境与目标的多维信息数据采集与存储的方法
CN109444985B (zh) * 2018-12-14 2024-04-26 华诺星空技术股份有限公司 多传感融合的便携式隐匿物成像探测系统
CN111382774A (zh) * 2018-12-31 2020-07-07 华为技术有限公司 一种数据处理方法及装置
CN111523353A (zh) * 2019-02-02 2020-08-11 顾泽苍 一种机器理解雷达数据处理的方法
EP3702802A1 (en) * 2019-03-01 2020-09-02 Aptiv Technologies Limited Method of multi-sensor data fusion
US11276189B2 (en) * 2019-03-06 2022-03-15 Qualcomm Incorporated Radar-aided single image three-dimensional depth reconstruction
BR112021014296A2 (pt) * 2019-03-26 2021-10-05 Telefonaktiebolaget Lm Ericsson (Publ) Método para determinar uma transformação entre um primeiro sistema de coordenadas de um dispositivo háptico ultrassônico e um segundo sistema de coordenadas de um dispositivo de sensor visual, determinante de transformação, programa de computador, e, produto de programa de computador
CN110033649A (zh) * 2019-03-28 2019-07-19 华南理工大学 基于多传感器信息的车辆运行环境感知技术协同处理方法
CN111856445B (zh) * 2019-04-11 2023-07-04 杭州海康威视数字技术股份有限公司 一种目标检测方法、装置、设备及系统
WO2020241954A1 (ko) * 2019-05-31 2020-12-03 엘지전자 주식회사 차량용 전자 장치 및 차량용 전자 장치의 동작 방법
CN112208529B (zh) * 2019-07-09 2022-08-02 毫末智行科技有限公司 用于目标检测的感知系统、驾驶辅助方法和无人驾驶设备
CN110412986A (zh) * 2019-08-19 2019-11-05 中车株洲电力机车有限公司 一种车辆障碍物检测方法及系统
CN110543850B (zh) * 2019-08-30 2022-07-22 上海商汤临港智能科技有限公司 目标检测方法及装置、神经网络训练方法及装置
CN112447058B (zh) * 2019-09-03 2022-09-06 比亚迪股份有限公司 泊车方法、装置、计算机设备及存储介质
CN110703732B (zh) * 2019-10-21 2021-04-13 北京百度网讯科技有限公司 相关性检测方法、装置、设备及计算机可读存储介质
CN110515092B (zh) * 2019-10-23 2020-01-10 南京甄视智能科技有限公司 基于激光雷达的平面触摸方法
WO2021092805A1 (zh) * 2019-11-13 2021-05-20 中新智擎科技有限公司 一种多模态数据融合方法、装置及智能机器人
CN111078014B (zh) * 2019-12-16 2023-11-24 深圳数拓科技有限公司 一种多维数据采集应用方法及系统
CN111209943B (zh) * 2019-12-30 2020-08-25 广州高企云信息科技有限公司 数据融合方法、装置及服务器
CN111208503B (zh) * 2020-01-13 2022-04-01 佛山市云米电器科技有限公司 一种雷达定位方法及系统
CN113378867B (zh) * 2020-02-25 2023-08-22 北京轻舟智航智能技术有限公司 一种异步数据融合的方法、装置、存储介质及电子设备
CN111398989A (zh) * 2020-04-02 2020-07-10 昆易电子科技(上海)有限公司 驾驶辅助系统的性能分析方法和测试设备
US11823458B2 (en) * 2020-06-18 2023-11-21 Embedtek, LLC Object detection and tracking system
US11574100B2 (en) * 2020-06-19 2023-02-07 Micron Technology, Inc. Integrated sensor device with deep learning accelerator and random access memory
CN113866763A (zh) * 2020-06-30 2021-12-31 华为技术有限公司 一种分布式微波雷达的成像方法及装置
CN112083400A (zh) * 2020-08-21 2020-12-15 达闼机器人有限公司 运动物体及其传感器的标定方法、装置、存储介质
CN112345084B (zh) * 2020-11-05 2021-09-28 北京易达恩能科技有限公司 基于数字孪生环境的三维温度场构建方法及装置
CN112711061B (zh) * 2020-11-18 2023-10-10 南京航空航天大学 针对射线源的监测系统及监测方法
CN112528763A (zh) * 2020-11-24 2021-03-19 浙江大华汽车技术有限公司 一种目标检测方法、电子设备和计算机存储介质
CN112697075B (zh) * 2020-12-03 2022-08-02 中国科学院光电技术研究所 一种交会对接激光雷达合作目标的投影面积分析方法
CN112617813B (zh) * 2020-12-15 2023-02-14 南京邮电大学 一种基于多传感器的非侵入式跌倒检测方法及系统
CN112541475B (zh) * 2020-12-24 2024-01-19 北京百度网讯科技有限公司 感知数据检测方法及装置
US11663832B2 (en) * 2021-01-19 2023-05-30 Micromax International Corp. Method and system for detecting and analyzing objects
CN112946684B (zh) * 2021-01-28 2023-08-11 浙江大学 基于光学目标信息辅助的电磁遥感智能成像系统与方法
CN112562093B (zh) * 2021-03-01 2021-05-18 湖北亿咖通科技有限公司 目标检测方法、电子介质和计算机存储介质
CN113286311B (zh) * 2021-04-29 2024-04-12 沈阳工业大学 基于多传感器融合的分布式周界安防环境感知系统
CN113362395A (zh) * 2021-06-15 2021-09-07 上海追势科技有限公司 一种基于传感器融合的环境感知方法
CN113435515B (zh) * 2021-06-29 2023-12-19 青岛海尔科技有限公司 图片识别方法和装置、存储介质及电子设备
CN113203424B (zh) * 2021-07-02 2021-10-26 中移(上海)信息通信科技有限公司 多传感器的数据融合方法、装置及相关设备
CN113627473B (zh) * 2021-07-06 2023-09-29 哈尔滨工程大学 基于多模态传感器的水面无人艇环境信息融合感知方法
CN113721240B (zh) * 2021-08-27 2024-03-15 中国第一汽车股份有限公司 一种目标关联方法、装置、电子设备及存储介质
CN113608186B (zh) * 2021-09-13 2023-10-20 中国工程物理研究院应用电子学研究所 一种雷达系统与光电成像系统的标校方法
CN114018236B (zh) * 2021-09-30 2023-11-03 哈尔滨工程大学 一种基于自适应因子图的激光视觉强耦合slam方法
WO2023070312A1 (zh) * 2021-10-26 2023-05-04 宁德时代新能源科技股份有限公司 图像处理方法
US20230237783A1 (en) * 2022-01-26 2023-07-27 Ford Global Technologies, Llc Sensor fusion
CN114613037B (zh) * 2022-02-15 2023-07-18 中国电子科技集团公司第十研究所 一种机载融合信息引导传感器提示搜索方法及装置
CN114677655A (zh) * 2022-02-15 2022-06-28 上海芯物科技有限公司 多传感器目标检测方法、装置、电子设备以及存储介质
CN114887898B (zh) * 2022-04-06 2024-02-13 厦门陆海环保股份有限公司 气动分选设备的气阀控制方法及系统
CN114813037B (zh) * 2022-04-21 2023-06-20 中国船舶科学研究中心 一种空化流动结构频率分布特征分析方法
CN114965872B (zh) * 2022-04-27 2023-10-13 重庆科技学院 一种多传感器数据融合的电子鼻及方法
CN114998567B (zh) * 2022-07-18 2022-11-01 中国科学院长春光学精密机械与物理研究所 一种基于多模态特征判别的红外点群目标识别方法
CN115457351B (zh) * 2022-07-22 2023-10-20 中国人民解放军战略支援部队航天工程大学 一种多源信息融合不确定性判别方法
CN114998425B (zh) * 2022-08-04 2022-10-25 吉奥时空信息技术股份有限公司 一种基于人工智能的目标物体地理坐标定位方法和装置
CN115452823B (zh) * 2022-08-26 2023-07-18 北京理工大学 一种基于混合感知的防护材料质量检测方法及装置
CN115345256B (zh) * 2022-09-16 2023-10-27 北京国联视讯信息技术股份有限公司 应用于智能制造的工业产品测试系统
CN115473921B (zh) * 2022-09-16 2023-08-04 云峰管业股份有限公司 一种用于智能化装配式综合管廊的监控系统及方法
CN115331190B (zh) * 2022-09-30 2022-12-09 北京闪马智建科技有限公司 一种基于雷视融合的道路隐患识别方法及装置
CN115327657B (zh) * 2022-10-13 2022-12-20 维飒科技(西安)有限公司 探测目标定位方法及装置
CN115526216B (zh) * 2022-11-24 2023-04-07 西安永安建筑科技有限责任公司 一种聚氨酯泡沫板生产设备的运行状态数据存储方法
TWI831643B (zh) * 2023-03-13 2024-02-01 鴻海精密工業股份有限公司 交通標誌識別方法及相關設備
CN116757981A (zh) * 2023-06-19 2023-09-15 北京拙河科技有限公司 一种多终端图像融合方法及装置
CN117115583B (zh) * 2023-08-09 2024-04-02 广东工业大学 基于交叉融合注意力机制的危险品检测方法及装置
CN116740681B (zh) * 2023-08-10 2023-11-21 小米汽车科技有限公司 目标检测方法、装置、车辆和存储介质
CN116912649B (zh) * 2023-09-14 2023-11-28 武汉大学 基于相关注意力引导的红外与可见光图像融合方法及系统
CN117274503B (zh) * 2023-11-17 2024-02-09 中国科学院国家空间科学中心 一种面向空间环境数据的多粒度动态可视化方法与系统
CN117329928B (zh) * 2023-11-30 2024-02-09 武汉阿内塔科技有限公司 一种基于多元情报融合的无人机综合探测方法和系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102789640A (zh) * 2012-07-16 2012-11-21 中国科学院自动化研究所 一种将可见光全色图像与红外遥感图像进行融合的方法
CN102842124A (zh) * 2012-07-16 2012-12-26 西安电子科技大学 基于矩阵低秩分解的多光谱图像与全色图像融合方法
CN104008533A (zh) * 2014-06-17 2014-08-27 华北电力大学 基于分块自适应特征跟踪的多传感器图像融合方法
CN105372203A (zh) * 2015-11-04 2016-03-02 江南大学 基于多传感器融合的新鲜苹果损伤敏感度无损检测方法
KR20170088202A (ko) * 2016-01-22 2017-08-01 서울시립대학교 산학협력단 이종의 위성영상 융합가능성 평가방법 및 그 장치
CN108663677A (zh) * 2018-03-29 2018-10-16 上海智瞳通科技有限公司 一种多传感器深度融合提高目标检测能力的方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1851728A1 (en) * 2005-02-08 2007-11-07 Seegrid Corporation Multidimensional evidence grids and system and methods for applying same
CN101652627A (zh) * 2007-01-14 2010-02-17 微软国际控股私有有限公司 一种用于成像的方法、设备和系统
CN109522280B (zh) * 2012-09-16 2022-04-19 哈尔滨华强电力自动化工程有限公司 一种图像文件格式及生成方法及装置及应用
US10890648B2 (en) * 2014-10-24 2021-01-12 Texas Instruments Incorporated Method and apparatus for generating alignment matrix for camera-radar system
WO2016100816A1 (en) * 2014-12-19 2016-06-23 United Technologies Corporation Sensor data fusion for prognostics and health monitoring
CN104881637B (zh) * 2015-05-09 2018-06-19 广东顺德中山大学卡内基梅隆大学国际联合研究院 基于传感信息及目标追踪的多模信息系统及其融合方法
CN104933444B (zh) * 2015-06-26 2019-01-01 南京邮电大学 一种面向多维属性数据的多层聚类融合机制的设计方法
JP6321202B2 (ja) * 2015-07-14 2018-05-09 エスゼット ディージェイアイ テクノロジー カンパニー リミテッドＳｚ Ｄｊｉ Ｔｅｃｈｎｏｌｏｇｙ Ｃｏ．，Ｌｔｄ モバイルプラットフォームの運動を決定する方法、装置、及びシステム
CN106485274B (zh) * 2016-10-09 2019-05-10 湖南穗富眼电子科技有限公司 一种基于目标特性图的物体分类方法
CN110136183B (zh) * 2018-02-09 2021-05-18 华为技术有限公司 一种图像处理的方法、装置以及摄像装置
CN109522951A (zh) * 2018-11-09 2019-03-26 上海智瞳通科技有限公司 一种环境与目标的多维信息数据采集与存储的方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102789640A (zh) * 2012-07-16 2012-11-21 中国科学院自动化研究所 一种将可见光全色图像与红外遥感图像进行融合的方法
CN102842124A (zh) * 2012-07-16 2012-12-26 西安电子科技大学 基于矩阵低秩分解的多光谱图像与全色图像融合方法
CN104008533A (zh) * 2014-06-17 2014-08-27 华北电力大学 基于分块自适应特征跟踪的多传感器图像融合方法
CN105372203A (zh) * 2015-11-04 2016-03-02 江南大学 基于多传感器融合的新鲜苹果损伤敏感度无损检测方法
KR20170088202A (ko) * 2016-01-22 2017-08-01 서울시립대학교 산학협력단 이종의 위성영상 융합가능성 평가방법 및 그 장치
CN108663677A (zh) * 2018-03-29 2018-10-16 上海智瞳通科技有限公司 一种多传感器深度融合提高目标检测能力的方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JIANG ET AL.: "Key technologies in registration and fusion for infrared and vivble images", vol. 35, 31 October 2006 (2006-10-31), pages 1 - 6, XP055743457, ISSN: 1007-2276 *

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111060881B (zh) * 2020-01-10 2022-05-13 湖南大学 一种毫米波雷达外参数在线标定方法
CN111060881A (zh) * 2020-01-10 2020-04-24 湖南大学 一种毫米波雷达外参数在线标定方法
CN111324594A (zh) * 2020-02-17 2020-06-23 武汉轻工大学 用于粮食加工业的数据融合方法、装置、设备及存储介质
CN111324594B (zh) * 2020-02-17 2023-08-11 武汉轻工大学 用于粮食加工业的数据融合方法、装置、设备及存储介质
CN112689842A (zh) * 2020-03-26 2021-04-20 华为技术有限公司 一种目标检测方法以及装置
CN112285709A (zh) * 2020-05-19 2021-01-29 陕西理工大学 基于深度学习的大气臭氧遥感激光雷达数据融合方法
CN112285709B (zh) * 2020-05-19 2022-07-26 陕西理工大学 基于深度学习的大气臭氧遥感激光雷达数据融合方法
CN112180373A (zh) * 2020-09-18 2021-01-05 纵目科技（上海）股份有限公司 一种多传感器融合的智能泊车系统和方法
CN112180373B (zh) * 2020-09-18 2024-04-19 纵目科技（上海）股份有限公司 一种多传感器融合的智能泊车系统和方法
CN112911265B (zh) * 2021-02-01 2023-01-24 北京都视科技有限公司 融合处理器及融合处理系统
CN112911265A (zh) * 2021-02-01 2021-06-04 北京都视科技有限公司 融合处理器及融合处理系统
CN113205559A (zh) * 2021-04-12 2021-08-03 华中科技大学 一种粉末床熔融的红外热像仪标定方法
CN113205559B (zh) * 2021-04-12 2022-08-02 华中科技大学 一种粉末床熔融的红外热像仪标定方法
CN115374880A (zh) * 2022-10-10 2022-11-22 北京邮电大学 一种面向海上目标识别的多级增量数据融合系统
CN116047463A (zh) * 2023-04-03 2023-05-02 中国科学院空天信息创新研究院 多角度sar目标散射各向异性推演方法、装置、设备及介质
CN116579965A (zh) * 2023-05-22 2023-08-11 北京拙河科技有限公司 一种多图像融合方法及装置
CN116579965B (zh) * 2023-05-22 2024-01-19 北京拙河科技有限公司 一种多图像融合方法及装置
CN116883998A (zh) * 2023-06-20 2023-10-13 珠海微度芯创科技有限责任公司 基于毫米波图像的物品标注方法、装置、电子设备
CN116883998B (zh) * 2023-06-20 2024-04-05 珠海微度芯创科技有限责任公司 基于毫米波图像的物品标注方法、装置、电子设备
CN116823674A (zh) * 2023-08-24 2023-09-29 湖南省水务规划设计院有限公司 一种跨模态融合的水下图像增强方法
CN116823674B (zh) * 2023-08-24 2024-03-12 湖南省水务规划设计院有限公司 一种跨模态融合的水下图像增强方法

Also Published As

Publication number Publication date
US20210012165A1 (en) 2021-01-14
EP3779867A1 (en) 2021-02-17
CN108663677A (zh) 2018-10-16
CN109655825B (zh) 2020-07-03
EP3779867A4 (en) 2021-12-29
US11675068B2 (en) 2023-06-13
CN109655825A (zh) 2019-04-19

Similar Documents

Publication Publication Date Title
WO2019184709A1 (zh) 多传感器融合的数据处理方法、装置与多传感器融合方法
WO2021227359A1 (zh) 一种无人机投影方法、装置、设备及存储介质
US9736451B1 (en) Efficient dense stereo computation
CN108789421B (zh) 基于云平台的云机器人交互方法和云机器人及云平台
CN114092780A (zh) 基于点云与图像数据融合的三维目标检测方法
CN109522951A (zh) 一种环境与目标的多维信息数据采集与存储的方法
US20220028114A1 (en) Method and System for Calibrating a Camera and Localizing Objects Within the Camera Field of View
CN106256124A (zh) 结构化立体
WO2021114777A1 (en) Target detection method, terminal device, and medium
Singh Surround-view vision-based 3d detection for autonomous driving: A survey
CN113192182A (zh) 一种基于多传感器的实景重建方法及系统
CN112055192B (zh) 图像处理方法、图像处理装置、电子设备及存储介质
CN113984037B (zh) 一种基于任意方向目标候选框的语义地图构建方法
TW202247108A (zh) 視覺定位方法、設備及電腦可讀儲存媒體
US11430146B2 (en) Two-stage depth estimation machine learning algorithm and spherical warping layer for EQUI-rectangular projection stereo matching
US11665330B2 (en) Dynamic-baseline imaging array with real-time spatial data capture and fusion
CN115598744A (zh) 一种基于微透镜阵列的高维光场事件相机及提取方法
WO2022011560A1 (zh) 图像裁剪方法与装置、电子设备及存储介质
CN102959937B (zh) 用于稳健颜色传送的方法和设备
KR20180098565A (ko) 픽셀 빔을 표현하는 데이터를 생성하는 방법 및 장치
Short 3-D Point Cloud Generation from Rigid and Flexible Stereo Vision Systems
Gao et al. A novel self-calibration method for a stereo-tof system using a kinect V2 and two 4k gopro cameras
Zhou et al. Improved YOLOv7 models based on modulated deformable convolution and swin transformer for object detection in fisheye images
Peng et al. A low-cost implementation of a 360 vision distributed aperture system
TWI819613B (zh) 物件的雙感測方法及用於物件感測的運算裝置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19776961

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2019776961

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2019776961

Country of ref document: EP

Effective date: 20201029