WO2022028576A1 - 图像配准方法及装置、计算机设备、介质 - Google Patents

图像配准方法及装置、计算机设备、介质 Download PDF

Info

Publication number
WO2022028576A1
WO2022028576A1 PCT/CN2021/111207 CN2021111207W WO2022028576A1 WO 2022028576 A1 WO2022028576 A1 WO 2022028576A1 CN 2021111207 W CN2021111207 W CN 2021111207W WO 2022028576 A1 WO2022028576 A1 WO 2022028576A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
registration
target
frame
event
Prior art date
Application number
PCT/CN2021/111207
Other languages
English (en)
French (fr)
Inventor
吴臻志
徐茂轩
祝夭龙
Original Assignee
北京灵汐科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202010784874.6A external-priority patent/CN111951312A/zh
Priority claimed from CN202010785636.7A external-priority patent/CN111951313B/zh
Application filed by 北京灵汐科技有限公司 filed Critical 北京灵汐科技有限公司
Publication of WO2022028576A1 publication Critical patent/WO2022028576A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular, to an image registration method, an image registration apparatus, a computer device, and a computer-readable medium.
  • CMOS Complementary Metal Oxide Semiconductor
  • CMOS Complementary Metal Oxide Semiconductor
  • Embodiments of the present disclosure provide an image registration method, an image registration apparatus, a computer device, and a computer-readable medium.
  • an embodiment of the present disclosure provides an image registration method, including:
  • the dynamic vision sensor and the image sensor are temporally registered to obtain temporal registration parameters, where the temporal registration parameters represent the temporal correlation between the sequence of events collected by the dynamic vision sensor and the image frames collected by the image sensor.
  • the step of performing temporal registration on the dynamic vision sensor and the image sensor, and obtaining temporal registration parameters includes:
  • the temporal registration parameters are determined from a plurality of the time stamps and a plurality of the frame numbers.
  • acquiring a plurality of time stamps of a sequence of events acquired by the dynamic vision sensor in response to a plurality of registration events comprises:
  • An average value of timestamps of a plurality of event data is calculated as the timestamp.
  • the step of determining the temporal registration parameter according to a plurality of the time stamps and a plurality of the frame numbers includes:
  • An average value of a plurality of the relative ratios is calculated as the temporal registration parameter.
  • an embodiment of the present disclosure provides an image registration method, including:
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the spatial registration parameters are obtained.
  • the spatial registration parameters represent the sequence of events collected by the dynamic vision sensor and the space of the image frames collected by the image sensor. connection relation;
  • the temporal registration parameters are obtained according to any one of the image registration methods described in the first aspect.
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the step of obtaining the spatial registration parameters includes:
  • time registration parameters obtain the target event sequence and at least one target image frame in the target registration time domain
  • the spatial registration parameters are obtained by performing spatial registration according to the target event sequence and the at least one target image frame.
  • performing spatial registration according to the target event sequence and the at least one target image frame, the step of obtaining the spatial registration parameters includes:
  • Image matching is performed on the event frame to be registered and the feature description sub-image frame to obtain the spatial registration parameter.
  • the step of constructing the event frame to be registered according to the target event sequence includes:
  • Time accumulation is performed on multiple event data in the target event sequence to obtain the to-be-registered event frame.
  • the step of obtaining a registration result image according to the spatial registration parameters, the sequence of events and the image frame includes:
  • the spatial registration parameters and the at least one target image frame the detailed information of the moving target in the event frame to be registered is determined, and the registration result image is obtained.
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the step of obtaining the spatial registration parameters includes:
  • the image frames to be registered in the target registration time domain are acquired, and the to-be-registered image frames represent the detailed features of the moving objects in the image frames;
  • the step of acquiring the to-be-registered image frame in the target registration temporal domain includes:
  • Pixel data of the at least one moving object is acquired, and the image frame to be registered is generated.
  • the step of acquiring the pixel data of the at least one moving object, and generating the image frame to be registered includes:
  • the initial weight matrix is assigned to obtain a target assignment weight matrix
  • the initial weight matrix is assigned to obtain a target assignment weight matrix, including:
  • the target assignment weight matrix is generated by combining the first assignment weight matrix and the second assignment weight matrix.
  • the step of obtaining a registration result image according to the spatial registration parameters, the sequence of events and the image frame includes:
  • the motion information of at least one moving object in the image frame to be registered is determined, and the registration result image is obtained.
  • an image registration apparatus including:
  • time registration module used for performing time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, the time registration parameters representing the sequence of events collected by the dynamic vision sensor and the image frames collected by the image sensor time relationship.
  • the temporal registration module includes:
  • an event data processing unit configured to acquire multiple time stamps of event sequences collected by the dynamic vision sensor in response to multiple registration events, each of the registration events corresponding to one of the time stamps;
  • an image frame processing unit configured to acquire frame numbers of multiple image frames acquired by the image sensor in response to multiple registration events
  • the time registration unit determines the time registration parameter according to a plurality of the time marks and a plurality of the frame numbers.
  • the event data processing unit is configured to acquire timestamps of multiple event data in an event sequence corresponding to the same registration event; calculate an average value of timestamps of multiple event data as the time stamp.
  • the time registration unit is configured to determine a relative ratio between the change amount of the time stamp and the frame number change amount corresponding to any two adjacent registration events; calculate a plurality of the relative ratios The average value of is used as the temporal registration parameter.
  • an image registration apparatus including:
  • the spatial registration module is used for performing spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration parameters to obtain spatial registration parameters, and the spatial registration parameters represent the event sequence and the image collected by the dynamic vision sensor The spatial correlation of the image frames collected by the sensor;
  • an image registration module configured to obtain a registration result image according to the spatial registration parameter, the event sequence and the image frame
  • the temporal registration parameters are obtained according to any one of the image registration methods described in the first aspect.
  • the spatial registration module includes:
  • a first data processing unit configured to acquire a target event sequence and at least one target image frame in the target registration time domain according to the time registration parameter
  • a first spatial registration unit configured to perform spatial registration according to the target event sequence and the at least one target image frame to obtain the spatial registration parameter.
  • the first data processing unit is configured to construct an event frame to be registered according to the target event sequence; acquire feature description sub-image frames according to the at least one target image frame; The frame is image-matched with the feature description sub-image frame to obtain the spatial registration parameter.
  • the first data processing unit is configured to perform time accumulation of multiple event data in the target event sequence to obtain the to-be-registered event frame.
  • the image registration module is configured to determine the detailed information of the moving object in the event frame to be registered according to the spatial registration parameter and the at least one target image frame, and obtain the registration result image.
  • the spatial registration module includes:
  • a second data processing unit configured to acquire the target event sequence in the target registration time domain according to the time registration parameter
  • the second data processing unit is further configured to acquire image frames to be registered in the target registration time domain according to the time registration parameters, and the image frames to be registered represent the moving target in the image frame. detail features;
  • the second spatial registration unit is configured to perform spatial registration according to the target event sequence and the to-be-registered image frame to obtain the spatial registration parameter.
  • the second data processing unit is configured to determine at least one moving object according to a plurality of the image frames; acquire pixel data of the at least one moving object to generate the image frame to be registered.
  • the second data processing unit is configured to construct an initial weight matrix corresponding to the size of the image frame collected by the image sensor; according to the current distribution area and prediction of the at least one moving object in the image frame In the distribution area, the initial weight matrix is assigned to obtain a target assignment weight matrix; the target assignment weight matrix is dot-multiplied with the image frame to obtain the to-be-registered image frame.
  • the second data processing unit is configured to assign a value to a first area in the initial weight matrix that matches the current distribution area of the at least one moving object in the image frame, to obtain a first assigning a weight matrix; assigning a value to a second area in the initial weight matrix that matches the predicted distribution area of the at least one moving object in the image frame to obtain a second assignment weight matrix; combining the first An assignment weight matrix and the second assignment weight matrix are used to generate the target assignment weight matrix.
  • the image registration module is configured to determine motion information of at least one moving object in the to-be-registered image frame according to the target event sequence and the spatial registration parameter to obtain the registration result image.
  • an embodiment of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. 1. the method, and/or implement the method according to any one of the second aspects.
  • embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements any of the methods described in the first aspect, and/or implements the second method. The method of any of the aspects.
  • temporal registration is performed first, and then the temporal registration result is spatially registered, and then the temporal registration result between the dynamic vision sensor and the image sensor and the
  • the result of spatial registration is to use the image frame output by the image sensor to perform image registration on the data output by the dynamic vision sensor, so as to realize the image registration between the dynamic vision sensor and the image sensor.
  • the object perception ability is stronger; it can also perform image registration on the image frame to be registered generated by the moving target extracted from the image frame output by the image sensor according to the data output by the dynamic vision sensor, and generate accurate data of the moving target at a certain moment.
  • FIG. 1 is a flowchart of an image registration method in an embodiment of the present disclosure
  • FIG. 3 is a flowchart of some steps in yet another image registration method according to an embodiment of the present disclosure.
  • FIG. 6 is a flowchart of some steps in another image registration method in an embodiment of the present disclosure.
  • FIG. 7 is a flowchart of some steps in yet another image registration method in an embodiment of the present disclosure.
  • FIG. 8 is a flowchart of some steps in still another image registration method according to an embodiment of the present disclosure.
  • FIG. 9 is a flowchart of some steps in still another image registration method according to an embodiment of the present disclosure.
  • FIG. 10 is a flowchart of some steps in still another image registration method according to an embodiment of the present disclosure.
  • FIG. 11 is a flowchart of some steps in still another image registration method according to an embodiment of the present disclosure.
  • FIG. 13 is a flowchart of some steps in still another image registration method according to an embodiment of the present disclosure.
  • 15 is a block diagram of an image registration apparatus in an embodiment of the present disclosure.
  • 16 is a block diagram of an image registration apparatus in an embodiment of the present disclosure.
  • 17 is a flowchart of an embodiment of an image registration method in an embodiment of the present disclosure.
  • 19 is a schematic diagram of feature-based image matching in an embodiment of the present disclosure.
  • FIG. 20 is a flowchart of an embodiment of an image registration method in an embodiment of the present disclosure.
  • 21 is a flowchart of an embodiment of an image registration method in an embodiment of the present disclosure.
  • 22 is a flowchart of an embodiment of an image registration method in an embodiment of the present disclosure.
  • 25 is a schematic structural diagram of an embodiment of an image registration apparatus in an embodiment of the present disclosure.
  • FIG. 26 is a schematic structural diagram of an embodiment of an image registration apparatus in an embodiment of the present disclosure.
  • FIG. 27 is a schematic structural diagram of a computer device in an embodiment of the present disclosure.
  • the dynamic vision sensor is an event-driven photoelectric sensor. According to the photoelectric changes experienced by each pixel unit independently, it encodes and outputs information such as the position of the activated pixel, event (ON/OFF), time stamp, etc. The output is event data. As shown in ⁇ xd, yd, t, c>, where ⁇ xd, yd> represents the spatial position of the pixel where the light intensity changes, t represents the timestamp, and c represents the amount of change in light intensity enhancement or reduction.
  • the dynamic vision sensor has the characteristics of high sensitivity. It only captures the moving target without recording the background information, which reduces the amount of data generated, thereby reducing the requirements for data storage, data computing power and transmission bandwidth.
  • CMOS image sensor As an example, it has high image resolution, but the temporal resolution is relatively low, which is prone to motion blur and generates a large amount of data.
  • the traditional image sensor adopts a full-frame trigger mechanism to output images in units of points and frame by frame. All pixel units are exposed for a fixed length of time, and all output at one time. It has the characteristics of high resolution (number of pixels) and high image quality, but Due to the large amount of information, it has high requirements for data storage, data computing power and transmission bandwidth.
  • the combination of dynamic vision sensor and traditional image sensor can effectively achieve dual-high perception with high static spatial resolution and high dynamic temporal resolution.
  • due to the differences between the two sensors in terms of focal length, field of view, optical devices, exposure principles, and data transmission methods it is difficult for the data collected by the two sensors to accurately correspond at the pixel level, which affects their collaborative work. Accuracy of spatial-temporal perception of objects. Therefore, how to realize the image registration between the dynamic vision sensor and the traditional image sensor is an urgent problem to be solved. Registration is performed to improve object perception of dynamic vision sensor images.
  • an image registration method which includes: performing temporal registration on a dynamic vision sensor and an image sensor; and performing spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration result ; Based on the spatial registration results and the temporal registration results, use the image frames collected by the image sensor to perform image registration on the event data collected by the dynamic vision sensor, or use the dynamic vision sensor to perform image registration on the image frames collected by the image sensor.
  • an embodiment of the present disclosure provides an image registration method, including:
  • the image sensor is a CMOS image sensor.
  • step S100 when performing step S100 to perform time registration on the dynamic vision sensor and the image sensor, it is necessary to first determine the use device and scene configuration of the dynamic vision sensor and the image sensor.
  • the configured scene should be consistent with the actual scene.
  • the multi-mode temporal registration of the dynamic vision sensor and the image sensor can be realized through step S100, which is beneficial to further perform spatial registration of the dynamic vision sensor and the image sensor on the basis of the temporal registration, and finally realize the Image registration.
  • the embodiments of the present disclosure do not specifically limit how to perform time registration on the dynamic vision sensor and the image sensor.
  • the steps of performing time registration on the dynamic vision sensor and the image sensor, and obtaining the time registration parameters include:
  • Time registration refers to registering the output information of the dynamic vision sensor and the output information of the image sensor with the image acquisition time as the dimension.
  • the multiple registration events are registration events that occur repeatedly. For example, using a blinking light source to emit light in pulses, one blink of the light source corresponding to each pulse is a registration event.
  • each time a registration event occurs the dynamic vision sensor generates an event sequence consisting of a series of event data, and the time stamp of the event sequence is used to generally represent the time information of the event data corresponding to the registration event. .
  • the event sequence and the image frame corresponding to the same registration event are associated in time.
  • the temporal registration parameters determined in step S130 can represent the temporal relationship between the event sequence corresponding to the same registration event and the image frame, that is, characterize the time sequence of the event sequence collected by the dynamic vision sensor and the image frame collected by the image sensor. connection relation.
  • time stamp of the first event data in the event sequence can be used as the time stamp of the event sequence; the time stamp of the last event data in the event sequence can also be used as the time stamp of the event sequence;
  • the timestamp of an event data serves as the timestamp of the event sequence.
  • the step of acquiring a plurality of time stamps of a sequence of events acquired by the dynamic vision sensor in response to a plurality of registration events includes:
  • S112. Calculate an average value of timestamps of multiple event data as the timestamp.
  • This embodiment of the present disclosure does not specifically limit how the time registration parameter is determined according to multiple time stamps and multiple frame numbers.
  • the step of determining the time registration parameter includes:
  • the time correlation between the event sequence collected by the dynamic vision sensor and the image frames collected by the image sensor represented by the time parameter can be used.
  • the event sequences and image frames that satisfy the temporal correlation in the same time domain are obtained, which facilitates the spatial registration of the dynamic vision sensor and the image sensor in the same time domain.
  • the following is an example to illustrate the time registration of the dynamic vision sensor and the image sensor in the embodiment of the present disclosure.
  • the scene was recorded with a dynamic vision sensor and an image sensor using a flashing light source that emits light in pulses.
  • the dynamic vision sensor detects a flickering light source, it outputs a sequence of events at the instant (increase or decrease) of the light intensity change.
  • the time stamp of the event sequence corresponding to the registration event can be obtained by taking the mean value of the time stamps t of multiple event data in the event sequence corresponding to one registration event.
  • an embodiment of the present disclosure provides an image registration method, including:
  • step S200 and step S300 are spatial registration performed on the basis of temporal registration of the dynamic vision sensor and the image sensor.
  • Spatial registration refers to registering the output information of the dynamic vision sensor and the output information of the image sensor with image information (eg, pixels) as the dimension.
  • the registration result image obtained through step S300 may be obtained by registering an image output by a dynamic vision sensor with an image frame collected by an image sensor, or may be an event sequence collected by a dynamic vision sensor It is obtained by registering the images output by the image sensor.
  • This embodiment of the present disclosure makes no special limitation on this.
  • the image frames collected by the image sensor are used to register the images output by the dynamic vision sensor, which can make up for the lack of the dynamic vision sensor's ability to capture static images.
  • the registration of the output images can make up for the lack of the temporal resolution of the image sensor, so as to achieve dual-high perception with high static spatial resolution and high dynamic temporal resolution.
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the steps of obtaining the spatial registration parameters include:
  • the target registration time domain refers to the same time domain corresponding to the dynamic vision sensor and the image sensor.
  • the image sources collected by the dynamic vision sensor and the image sensor are the same. That is, the target event sequence acquired through step S210 and the target image frame are registered in the time dimension.
  • the embodiments of the present disclosure do not specifically limit how to obtain spatial registration parameters by performing spatial registration according to a target event sequence and at least one target image frame.
  • performing spatial registration according to the target event sequence and the at least one target image frame, and the step of obtaining the spatial registration parameters includes:
  • This embodiment of the present disclosure does not specifically limit how to construct the event frame to be registered according to the target event sequence.
  • the step of constructing the event frame to be registered according to the target event sequence includes:
  • the spatial registration parameters obtained through steps S210 to S220 can be used to register the images output by the dynamic vision sensor with the image frames collected by the image sensor.
  • the step of obtaining a registration result image according to the spatial registration parameters, the event sequence and the image frame includes:
  • S310 Determine the detailed information of the moving object in the event frame to be registered according to the spatial registration parameter and the at least one target image frame, and obtain the registration result image.
  • the registration result image obtained through step S310 is a DVS image that integrates the detailed information of the moving object.
  • the detailed information of the moving object includes outline, edge, color, etc. of the moving object. This embodiment of the present disclosure makes no special limitation on this.
  • the registration result image is obtained through feature-based image matching.
  • the registration result image is obtained by feature-based image matching, including:
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the steps of obtaining the spatial registration parameters include:
  • an image sensor is used to track a moving object, and pixel data of the moving object and its accessories are acquired to obtain image frames to be registered. Only the pixel data of the moving target and its accessories are retained in the image frame to be registered.
  • the moving target is first identified and the moving target is tracked, or the visual field is concentrated on the target accessories for sparse tracking, which can eliminate redundancy. information, saving bandwidth and computing power.
  • This embodiment of the present disclosure does not specifically limit how to acquire the image frame to be registered.
  • the step of acquiring the to-be-registered image frame in the target registration temporal domain includes:
  • the step of generating the image frame to be registered includes:
  • the initial weight matrix is assigned to obtain a target assignment weight matrix, including :
  • S2422c Combine the first assignment weight matrix and the second assignment weight matrix to generate the target assignment weight matrix.
  • the step of obtaining a registration result image according to the spatial registration parameters, the event sequence and the image frame includes:
  • S320 Determine motion information of at least one moving object in the to-be-registered image frame according to the target event sequence and the spatial registration parameter, and obtain the registration result image.
  • the motion information of the moving object may be the motion trajectory of the moving object.
  • the registration result image obtained through step S320 is an image frame in which the motion information of the moving object is integrated.
  • an embodiment of the present disclosure provides an image registration apparatus, including:
  • the time registration module 100 is configured to perform time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, where the time registration parameters represent the sequence of events collected by the dynamic vision sensor and the images collected by the image sensor The temporal relationship of the frames.
  • the temporal registration module includes:
  • an event data processing unit configured to acquire multiple time stamps of event sequences collected by the dynamic vision sensor in response to multiple registration events, each of the registration events corresponding to one of the time stamps;
  • an image frame processing unit configured to acquire frame numbers of multiple image frames acquired by the image sensor in response to multiple registration events
  • the time registration unit determines the time registration parameter according to a plurality of the time marks and a plurality of the frame numbers.
  • the event data processing unit is configured to acquire timestamps of multiple event data in an event sequence corresponding to the same registration event; calculate an average value of timestamps of multiple event data as the time stamp.
  • the time registration unit is configured to determine a relative ratio between the change amount of the time stamp and the frame number change amount corresponding to any two adjacent registration events; calculate a plurality of the relative ratios The average value of is used as the temporal registration parameter.
  • an embodiment of the present disclosure provides an image registration apparatus, including:
  • the spatial registration module 200 is configured to perform spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration parameters, and obtain spatial registration parameters, wherein the spatial registration parameters represent the sequence of events collected by the dynamic vision sensor and the The spatial correlation of the image frames collected by the image sensor;
  • an image registration module 300 configured to obtain a registration result image according to the spatial registration parameter, the event sequence and the image frame;
  • the temporal registration parameters are obtained according to any one of the image registration methods described in the first aspect.
  • the spatial registration module includes:
  • a first data processing unit configured to acquire a target event sequence and at least one target image frame in the target registration time domain according to the time registration parameter
  • a first spatial registration unit configured to perform spatial registration according to the target event sequence and the at least one target image frame to obtain the spatial registration parameter.
  • the first data processing unit is configured to construct an event frame to be registered according to the target event sequence; acquire feature description sub-image frames according to the at least one target image frame; The frame is image-matched with the feature description sub-image frame to obtain the spatial registration parameter.
  • the first data processing unit is configured to perform time accumulation of multiple event data in the target event sequence to obtain the to-be-registered event frame.
  • the image registration module is configured to determine the detailed information of the moving object in the event frame to be registered according to the spatial registration parameter and the at least one target image frame, and obtain the registration result image.
  • the spatial registration module includes:
  • a second data processing unit configured to acquire the target event sequence in the target registration time domain according to the time registration parameter
  • the second data processing unit is further configured to acquire image frames to be registered in the target registration time domain according to the time registration parameters, and the image frames to be registered represent the moving target in the image frame. detail features;
  • the second spatial registration unit is configured to perform spatial registration according to the target event sequence and the to-be-registered image frame to obtain the spatial registration parameters.
  • the second data processing unit is configured to determine at least one moving object according to a plurality of the image frames; acquire pixel data of the at least one moving object to generate the image frame to be registered.
  • the second data processing unit is configured to construct an initial weight matrix corresponding to the size of the image frame collected by the image sensor; according to the current distribution area and prediction of the at least one moving object in the image frame In the distribution area, the initial weight matrix is assigned to obtain a target assignment weight matrix; the target assignment weight matrix is dot-multiplied with the image frame to obtain the to-be-registered image frame.
  • the second data processing unit is configured to assign a value to a first area in the initial weight matrix that matches the current distribution area of the at least one moving object in the image frame, to obtain a first assigning a weight matrix; assigning a value to a second area in the initial weight matrix that matches the predicted distribution area of the at least one moving object in the image frame to obtain a second assignment weight matrix; combining the first An assignment weight matrix and the second assignment weight matrix are used to generate the target assignment weight matrix.
  • the image registration module is configured to determine motion information of at least one moving object in the to-be-registered image frame according to the target event sequence and the spatial registration parameter to obtain the registration result image.
  • the device can be implemented by means of software and/or hardware, and can generally be integrated in computer equipment, for example, it can be a computer device that establishes a connection with a dynamic vision sensor and a traditional image sensor, and the computer device can receive and process Receive data from dynamic vision sensors as well as traditional image sensors.
  • the image registration method provided by this embodiment includes:
  • E110 Perform time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, where the time registration parameters represent the time correlation between the event sequence collected by the dynamic vision sensor and the image frames collected by the image sensor.
  • the dynamic vision sensor and the image sensor respectively refer to a dynamic vision sensor and an image sensor that require image registration, and the shooting scenes of the two are the same.
  • the image sensor outputs image frames after collecting image information.
  • image sensors are CMOS image sensors.
  • Time registration refers to registering the output information of the dynamic vision sensor and the output information of the image sensor with the image acquisition time as the dimension.
  • the event sequence composed of the image frame output by the image sensor and the event data output by the dynamic vision sensor when the image information is collected for the same shooting scene is registered in terms of time sequence, so as to realize the time registration of the dynamic vision sensor and the image sensor. allow.
  • the event sequence number information of the event sequence (composed of a plurality of event data) output by the dynamic vision sensor can be registered with the frame number of the image frame output by the image sensor, so as to realize the comparison between the dynamic vision sensor and the image.
  • the sensors are time-registered.
  • the event sequence number is the sequence number of the timestamp of the event data output by the dynamic vision sensor.
  • the temporal registration of the dynamic vision sensor and the image sensor is performed by determining the temporal registration parameters between the dynamic vision sensor and the image sensor.
  • the time registration parameter may be the corresponding proportional relationship between the dynamic vision sensor and the amount of output information of the image sensor when the image acquisition time is the statistical dimension.
  • E110 may include: in response to at least two shooting scene changes, respectively acquiring an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each shooting scene change, and an event sequence number corresponding to each shooting scene change The frame number of a group of image frames output by the image sensor; according to the event sequence number of each event sequence and the frame number of each group of image frames, the time registration parameters between the dynamic vision sensor and the image sensor are determined, and the time registration parameters as a temporal registration result.
  • the change of the shooting scene refers to the change of the image information collected by the dynamic vision sensor or the image sensor, for example, there may be a moving target object in the shooting scene, for example, the light intensity (or light source, etc.) in the shooting scene may also change, etc.
  • each shooting scene changes, determine the event sequence output by the dynamic vision sensor during the change process, obtain the event sequence number corresponding to each event data in the event sequence, and determine the image sensor during the change process.
  • a set of output image frames (wherein a set of image frames may include one or more image frames), and the frame number of each image frame is obtained.
  • the time registration parameters between the dynamic vision sensor and the image sensor are determined, that is, the time registration parameters are determined.
  • the image acquisition time as the dimension
  • the corresponding proportional relationship between the number of events output by the dynamic vision sensor and the number of image frames output by the image sensor, such as the m event data output by the dynamic vision sensor in the same shooting scene corresponds to the output of the image sensor of an image frame.
  • statistical analysis is performed on the event sequence number information of each event sequence and the frame number of each group of image frames obtained during the process of multiple shooting scenes changing, and the relationship between the dynamic vision sensor and the image sensor is determined according to the statistical analysis result. time registration parameters.
  • determining the time registration parameters between the dynamic vision sensor and the image sensor according to the event sequence numbers of each event sequence and the frame numbers of each group of image frames may include:
  • the event sequence numbers of each event in an event sequence obtained in the first shooting scene change are t10, t11, ..., t1m, respectively, and take the mean value of t10, t11, ..., t1m t1avg is used as the mean value of the event sequence numbers of the event sequence.
  • the calculated mean values of event sequence numbers of each event sequence are t1avg, t2avg, ..., tnavg.
  • the amount of change in the mean value of the event sequence number refers to the difference between the mean values of the event sequence numbers of two consecutive event sequences corresponding to two consecutive scene changes, namely tnavg-t(n-1)avg;
  • the amount of change in the frame number refers to the difference between the frame numbers of the last image frame in the two groups of video frames corresponding to two consecutive shooting scene changes, assuming that the frame numbers of the last image frame in the n groups of video frames are n1, n2, ... , nn, then the variation of the frame numbers of two consecutive groups of image frames is nn-n(n-1).
  • the frame numbers of each group of image frames are continuous.
  • the frame numbers of each image frame in a group of image frames obtained during the first shooting scene change are 1, 2, ..., n1, then The frame numbers of a group of image frames acquired in the second shooting scene change are counted from n1+1.
  • the ratio of the change of the mean value of the event serial number to the change of the frame number refers to the ratio of the change of the mean value of the event serial number to the change of the frame number corresponding to the change of the two consecutive shooting scenes, namely (tnavg-t(n- 1) avg)/(nn-n(n-1)).
  • the corresponding ratio of the change in the mean value of the (n-1) event sequence numbers and the change in the frame number can be obtained, the mean value of the (n-1) corresponding ratios can be calculated, and the mean value can be used as Registration parameters.
  • a flashing light source is used to effect the change of the captured scene, the light source emits light in pulses, and the scene is recorded using a dynamic vision sensor and an image sensor, respectively.
  • the dynamic vision sensor detects a flickering light source, it will output an event sequence at the moment of light intensity change (increase or weaken), and take the average of the event numbers of each event in the event sequence as the time stamp of the dynamic vision sensor.
  • the image sensor records and outputs the frame number of each image frame, and correlates the frame number of the last image frame with the time stamp of the dynamic vision sensor, and then the time stamp of a dynamic vision sensor and the image frame number of the image sensor can be obtained. corresponding relationship.
  • the mean value of the mean value change of the event sequence number and the mean value of the frame number change amount can also be calculated respectively, and the ratio of the mean value of the mean value of the event sequence number mean value change amount to the mean value of the frame number change amount is taken as the time Registration parameters.
  • the time registration parameter is the ratio of the change of the event sequence number to the change of the frame number, wherein the change of the frame number is 1, that is, the time registration parameter instructs the image sensor to output a corresponding image frame.
  • E120 Perform spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration parameters to obtain spatial registration parameters, where the spatial registration parameters represent the sequence of events collected by the dynamic vision sensor and the images collected by the image sensor The spatial association of frames.
  • the time registration result refers to the registration details of the dynamic vision sensor and the image sensor regarding the image acquisition time.
  • the time registration result is the time registration parameter.
  • Spatial registration refers to registering the output information of the dynamic vision sensor and the image sensor with the image information as the dimension.
  • the image information output by the dynamic vision sensor and the image sensor in the target registration time domain is registered in terms of pixels, so as to realize the spatial registration of the dynamic vision sensor and the image sensor.
  • the output data of the dynamic vision sensor and the output data of the image sensor after time registration are obtained, and the spatial registration of the dynamic vision sensor and the image sensor can be realized by spatially registering these output data.
  • performing spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration result may include:
  • the target event sequence output by the dynamic vision sensor in the target registration time domain and at least one target image frame output by the image sensor in the target registration time domain are obtained; according to the target event sequence and the at least one target image frame, The dynamic vision sensor and the image sensor are spatially registered to obtain the spatial registration parameters.
  • the target registration time domain refers to the same time domain corresponding to the dynamic vision sensor and the image sensor.
  • the image sources collected by the dynamic vision sensor and the image sensor are the same. That is, the event data (also referred to as event stream data) output by the dynamic vision sensor in the target registration time domain and the image frames output by the image sensor in the target registration time domain are registered in the time dimension.
  • the target registration time domain can be determined according to the event sequence number of the output event data; for the image sensor, the target registration time domain can be determined according to the number of output image frames.
  • the variation of the event sequence number corresponding to the dynamic vision sensor is obtained, and then output the frame number of the image frame according to the selected image sensor, and the dynamic visual sensor.
  • the time sequence of the event data output by the vision sensor is obtained, and each event data output by the dynamic vision sensor corresponding to the change of the event sequence number is obtained, which is the target event sequence output by the dynamic vision sensor in the target registration time domain.
  • the selected output image frame of the image sensor is the target image frame output by the image sensor in the target registration time domain.
  • the number of image frames output by the image sensor in the target registration time domain determined in this step is at least one.
  • the event frame to be registered is image-matched with the target image frame to obtain the spatial registration parameters, so as to realize the spatial registration of the dynamic vision sensor and the image sensor .
  • E130 Acquire a registration result image according to the spatial registration parameter, the event sequence, and the image frame.
  • the spatial registration result refers to the registration details of the image information between the dynamic vision sensor and the image sensor.
  • the spatial registration result is the spatial registration parameter.
  • the data output by the dynamic vision sensor includes at least one of event data and an event stream feature frame constructed according to the event data.
  • the dynamic vision sensor When using the image frame output by the image sensor to perform image registration on the event data output by the dynamic vision sensor, first determine the image frame output by the image sensor corresponding to the event data to be registered based on the temporal registration result, and then based on the spatial registration result , and use the image frame to perform image registration on the event data to be registered. By repeating the above process, all event data output by the dynamic vision sensor can be registered.
  • the image frame may be used to perform image registration on multiple event data respectively.
  • the target time period for example, the image frame corresponding to the image sensor output
  • the multiple event data in the time period are constructed into event stream feature frames, and then the image frames output by the image sensor corresponding to the target time period are obtained based on the temporal registration results.
  • Feature frames for image registration By repeating the above process, the event stream feature frames constructed by multiple event data in each target time period of the dynamic vision sensor can be registered.
  • the dynamic vision sensor and the image sensor firstly perform temporal registration, and then perform spatial registration based on the temporal registration result, and then based on the temporal registration result between the dynamic vision sensor and the image sensor And the spatial registration result, the image frame output by the image sensor is used to perform image registration on the data output by the dynamic vision sensor, so as to realize the image registration between the dynamic vision sensor and the image sensor. After registration, the dynamic vision sensor image object perception ability is stronger.
  • FIG. 18 is a flowchart of an image registration method according to Embodiment 2 of the present disclosure. This embodiment is embodied on the basis of the above-mentioned embodiment, wherein, according to the target event data and the target image frame, performing spatial registration on the dynamic vision sensor and the image sensor may include:
  • the target event data construct an event stream feature frame corresponding to at least one target image frame; determine a feature description sub-image frame corresponding to the at least one target image frame; Perform feature-based image registration with the feature description sub-image frame to obtain the spatial registration parameters between the dynamic vision sensor and the image sensor, and use the spatial registration parameters as the spatial registration result.
  • the image registration method provided by this embodiment includes:
  • E210 Perform time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, where the time registration parameters represent the time correlation between the event sequence collected by the dynamic vision sensor and the image frames collected by the image sensor.
  • E220 Acquire, according to the time registration parameters, a target event sequence output by the dynamic vision sensor in the target registration time domain, and at least one target image frame output by the image sensor in the target registration time domain.
  • E230 Construct an event frame to be registered corresponding to at least one target image frame according to the target event sequence.
  • the target event sequence output by the dynamic vision sensor in the target registration time domain is the combination of multiple event data, that is, the event sequence output by the dynamic vision sensor in the target registration time domain.
  • the event data in a certain time range are collected together and constructed into image frames, which are called event stream feature frames, or event frames to be registered.
  • the event data in the time range corresponding to the output of one image frame by the image sensor is collected together to construct the event frame to be registered.
  • E240 Acquire a feature description sub-image frame according to at least one target image frame.
  • a feature description sub-image frame corresponding to the at least one target image frame is obtained by a preset method.
  • the target image frame output by the image sensor in the target registration time domain is acquired, the time sequence feature feature extraction is performed on the target image frame, and the feature description sub-image frame corresponding to the target image frame is generated.
  • the edge contour is extracted from the image collected by the image sensor according to the frame, and the corresponding feature description sub-image frame is generated.
  • the frame difference method or the filter filtering method may be used to extract the image edge contour, which is not specifically limited in this embodiment.
  • E250 Perform image matching between the event frame to be registered and the sub-image frame for feature description to obtain spatial registration parameters between the dynamic vision sensor and the image sensor, and use the spatial registration parameters as a spatial registration result.
  • the image edge contour determined by the target image frame output by the image sensor is similar to the light intensity change frame data collected by the dynamic vision sensor, that is, the feature description sub-image frame is similar to the constructed event flow feature frame.
  • the feature-based image registration method determines the image registration parameters of two image frames as the spatial registration parameters between the dynamic vision sensor and the image sensor.
  • the feature description sub-image frame is used as the reference image
  • the event stream feature frame is used as the image to be registered.
  • the core steps of the feature-based image registration method are:
  • the image registration parameters can be determined based on the spatial transformation model used during registration, the similarity measure criterion for registration and the spatial transformation matrix;
  • Image conversion registration based on the determined image registration parameters, use the reference image to perform image registration on the image to be registered.
  • the event data acquired in the target registration time domain can be divided into multiple event sequences according to the number of image frames output by the image sensor. , and build the event frame to be registered based on one of the event sequences. Further, image registration is performed on the to-be-registered event frame by using the feature description sub-image frame of an image frame corresponding to the event sequence to determine spatial registration parameters between the dynamic vision sensor and the image sensor.
  • E260 Determine the detailed information of the moving target in the event frame to be registered according to the spatial registration parameter and the at least one target image frame, and obtain the registration result image.
  • using the image frame output by the image sensor to perform image registration on the event data output by the dynamic vision sensor may include:
  • the data output by the dynamic vision sensor includes at least one of event data and an event frame to be registered constructed according to the event data.
  • the image frame output by the image sensor and the event data output by the dynamic vision sensor corresponding to the image frame are determined.
  • the image frame output by the image sensor and the event data output by the dynamic vision sensor belong to the same time domain.
  • the target image frame is used to perform image registration on the event data output by the dynamic vision sensor.
  • the number of image frames output by the image sensor can be one or more. When the number is multiple, each image frame can be used in turn to register the corresponding event data output by the dynamic vision sensor until the image sensor output is processed. of all image frames.
  • the time registration result for example, the time registration parameter
  • determine multiple event data collected by the dynamic vision sensor corresponding to the image frame secondly, based on the relationship between the dynamic vision sensor and the image sensor using the image frame to perform image registration on the event data, or use the image frame to perform image registration on the event frame to be registered constructed from the event data.
  • the above technical solution realizes the image registration between the dynamic vision sensor and the image sensor, and the image of the dynamic vision sensor has stronger object perception ability after registration; at the same time, the feature-based image registration method is applied to the event stream data, An implementation method for processing event stream data is provided, which solves the problem that most existing image processing methods and image detection methods cannot be directly used for processing event streams.
  • FIG. 20 is a flowchart of an image registration method according to Embodiment 3 of the present disclosure.
  • This embodiment provides an optional implementation, wherein the image sensor may be a CMOS image sensor.
  • the image registration method provided by this embodiment includes:
  • E410 In response to at least two shooting scene changes, obtain an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each shooting scene change, and a group of images output by the CMOS image sensor corresponding to each shooting scene change Like the frame number of the frame, and determine the event sequence number mean of each event sequence separately.
  • E420 According to the mean value of the event sequence numbers of each event sequence and the frame numbers of each group of image frames, respectively determine the corresponding ratios of the variation of the mean value of the event sequence numbers and the variation of the frame numbers, and use the mean value of each corresponding ratio as the dynamic vision sensor and the frame number.
  • the mean value of the event sequence numbers of each event sequence is taken as the time stamp of the dynamic vision sensor, and the frame number of the last image frame in each group of image frames is taken as the time stamp of the CMOS image sensor.
  • E430 based on the time registration parameters, acquire the target event sequence output by the dynamic vision sensor in the target registration time domain, and the target image frame output by the CMOS image sensor in the target registration time domain.
  • the spatial registration between the dynamic vision sensor and the CMOS image sensor is performed based on the temporal registration parameters.
  • E440 Construct an event flow feature frame (an event frame to be registered) according to multiple event data output by the dynamic vision sensor in the target registration time domain.
  • the feature description sub-image frame is obtained by a frame difference method or a filter filtering method.
  • E460 Perform feature-based image registration on the event stream feature frame and the feature description sub-image frame to obtain spatial registration parameters between the dynamic vision sensor and the CMOS image sensor.
  • the spatial registration parameters between the event stream feature frame and the feature description sub-image frames are determined as the spatial registration parameters between the dynamic vision sensor and the CMOS image sensor.
  • E470 Based on the spatial registration result and the temporal registration result, use the image frame output by the CMOS image sensor to perform image registration on the event stream feature frame constructed according to the event data output by the dynamic vision sensor.
  • the image registration of the dynamic vision sensor is performed through the image frames output by the CMOS image sensor, which improves the object perception capability of the dynamic vision sensor, such as color, edge, etc.
  • the combination of the two different modes of sensors realizes high static Spatial resolution - dual-high perception with high dynamic temporal resolution, while also reducing the requirements of CMOS image sensors for data storage, data computing power and transmission bandwidth.
  • the device can be implemented by means of software and/or hardware, and generally can be integrated in computer equipment, for example, it can be a computer equipment connected to a dynamic vision sensor and a traditional image sensor, and the computer equipment can receive and process Receive data from dynamic vision sensors as well as traditional image sensors.
  • the image registration method provided by this embodiment includes:
  • E510 In response to at least two shooting scene changes, obtain an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each shooting scene change, and a group of images output by the image sensor corresponding to each shooting scene change The frame number of the frame.
  • E520 Determine a time registration parameter between the dynamic vision sensor and the image sensor according to the event sequence numbers of each event sequence and the frame numbers of each group of image frames.
  • the E520 may include:
  • the technical solution provided in this embodiment realizes time registration between sensors of different modes, and is suitable for application scenarios that require time registration between a dynamic vision sensor and a traditional image sensor.
  • FIG. 22 is a flowchart of an image registration method provided by the fifth embodiment of the present disclosure, which can be applied to the case of how to combine a dynamic vision sensor and a traditional image sensor to achieve smooth tracking of moving objects.
  • the method can be provided by the embodiment of the present disclosure. It can be implemented by an image registration device, which can be implemented in software and/or hardware, and can generally be integrated in computer equipment, such as a computer device that is connected to a dynamic vision sensor and a traditional image sensor. Capable of receiving and processing data acquired from dynamic vision sensors as well as conventional image sensors.
  • the image registration method provided by this embodiment includes:
  • E610 Perform time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, where the time registration parameters represent the time correlation between the event sequence collected by the dynamic vision sensor and the image frames collected by the image sensor.
  • the dynamic vision sensor and the image sensor respectively refer to a dynamic vision sensor and an image sensor that require image registration, and the shooting scenes of the two are the same.
  • the image sensor outputs image frames after collecting image information.
  • this type of image sensor is a CMOS image sensor.
  • Time registration refers to registering the output information of the dynamic vision sensor and the output information of the image sensor with the image acquisition time as the dimension.
  • the event sequence composed of the image frame output by the image sensor and the event data output by the dynamic vision sensor when the image information is collected for the same shooting scene is registered in terms of time sequence, so as to realize the time registration of the dynamic vision sensor and the image sensor. allow.
  • the event sequence number information of the event sequence (composed of a plurality of event data) output by the dynamic vision sensor can be registered with the frame number of the image frame output by the image sensor, so as to realize the comparison between the dynamic vision sensor and the image.
  • the sensors are time-registered.
  • the event sequence number is the sequence number of the timestamp of the event data output by the dynamic vision sensor.
  • the temporal registration of the dynamic vision sensor and the image sensor is performed by determining the temporal registration parameters between the dynamic vision sensor and the image sensor.
  • the time registration parameter may be the corresponding proportional relationship between the dynamic vision sensor and the amount of output information of the image sensor when the image acquisition time is the statistical dimension.
  • E610 may include: in response to at least two shooting scene changes, respectively acquiring an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each shooting scene change, and an event sequence number corresponding to each shooting scene change The frame number of a group of image frames output by the image sensor; according to the event sequence number of each event sequence and the frame number of each group of image frames, the time registration parameters between the dynamic vision sensor and the image sensor are determined, and the time registration parameters as a temporal registration result.
  • the change of the shooting scene refers to the change of the image information collected by the dynamic vision sensor or the image sensor, for example, there may be a moving target object in the shooting scene, for example, the light intensity (or light source, etc.) in the shooting scene may also change, etc.
  • each shooting scene changes, determine the event sequence output by the dynamic vision sensor during the change process, obtain the event sequence number corresponding to each event data in the event sequence, and determine the image sensor during the change process.
  • a set of output image frames (wherein a set of image frames may include one or more image frames), and the frame number of each image frame is obtained.
  • the time registration parameters between the dynamic vision sensor and the image sensor are determined, that is, the time registration parameters are determined.
  • the image acquisition time as the dimension
  • the corresponding proportional relationship between the number of events output by the dynamic vision sensor and the number of image frames output by the image sensor, such as the m event data output by the dynamic vision sensor in the same shooting scene corresponds to the output of the image sensor of an image frame.
  • statistical analysis is performed on the event sequence number information of each event sequence and the frame number of each group of image frames obtained during the process of multiple shooting scenes changing, and the relationship between the dynamic vision sensor and the image sensor is determined according to the statistical analysis result. time registration parameters.
  • determining the time registration parameters between the dynamic vision sensor and the image sensor according to the event sequence numbers of each event sequence and the frame numbers of each group of image frames may include:
  • the event sequence numbers of each event in an event sequence obtained in the first shooting scene change are t10, t11, ..., t1m, respectively, and take the mean value of t10, t11, ..., t1m t1avg is used as the mean value of the event sequence numbers of the event sequence.
  • the calculated mean values of event sequence numbers of each event sequence are t1avg, t2avg, ..., tnavg.
  • the amount of change in the mean value of the event sequence number refers to the difference between the mean values of the event sequence numbers of two consecutive event sequences corresponding to two consecutive scene changes, namely tnavg-t(n-1)avg;
  • the amount of change in the frame number refers to the difference between the frame numbers of the last image frame in the two groups of video frames corresponding to two consecutive shooting scene changes, assuming that the frame numbers of the last image frame in the n groups of video frames are n1, n2, ... , nn, then the variation of the frame numbers of two consecutive groups of image frames is nn-n(n-1).
  • the frame numbers of each group of image frames are continuous.
  • the frame numbers of each image frame in a group of image frames obtained during the first shooting scene change are 1, 2, ..., n1, then The frame numbers of a group of image frames acquired in the second shooting scene change are counted from n1+1.
  • the ratio of the change of the mean value of the event serial number to the change of the frame number refers to the ratio of the change of the mean value of the event serial number to the change of the frame number corresponding to the change of the two consecutive shooting scenes, namely (tnavg-t(n- 1) avg)/(nn-n(n-1)).
  • the corresponding ratio of the change in the mean value of the (n-1) event sequence numbers and the change in the frame number can be obtained, the mean value of the (n-1) corresponding ratios can be calculated, and the mean value can be used as Registration parameters.
  • a flashing light source is used to effect the change of the captured scene, the light source emits light in pulses, and the scene is recorded using a dynamic vision sensor and an image sensor, respectively.
  • the dynamic vision sensor detects a flickering light source, it will output an event sequence at the moment of light intensity change (increase or weaken), and take the average of the event numbers of each event in the event sequence as the time stamp of the dynamic vision sensor.
  • the image sensor records and outputs the frame number of each image frame, and correlates the frame number of the last image frame with the time stamp of the dynamic vision sensor, and then the time stamp of a dynamic vision sensor and the image frame number of the image sensor can be obtained. corresponding relationship.
  • the mean value of the mean value change of the event sequence number and the mean value of the frame number change amount can also be calculated respectively, and the ratio of the mean value of the mean value of the event sequence number mean value change amount to the mean value of the frame number change amount is taken as the time Registration parameters.
  • the time registration parameter is the ratio of the change of the event sequence number to the change of the frame number, wherein the change of the frame number is 1, that is, the time registration parameter instructs the image sensor to output a corresponding image frame.
  • E620 Perform spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration parameters to obtain spatial registration parameters, where the spatial registration parameters represent the sequence of events collected by the dynamic vision sensor and the images collected by the image sensor The spatial association of frames.
  • the time registration result refers to the registration details of the dynamic vision sensor and the image sensor regarding the image acquisition time.
  • the time registration result is the time registration parameter.
  • Spatial registration refers to registering the output information of the dynamic vision sensor and the image sensor with the image information as the dimension.
  • the image information output by the dynamic vision sensor and the image sensor in the target registration time domain is registered in terms of pixels, so as to realize the spatial registration of the dynamic vision sensor and the image sensor.
  • the output data of the dynamic vision sensor and the output data of the image sensor after time registration are obtained, and the spatial registration of the dynamic vision sensor and the image sensor can be realized by spatially registering these output data.
  • performing spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration result may include:
  • the target event sequence output by the dynamic vision sensor in the target registration time domain and at least one target image frame output by the image sensor in the target registration time domain are obtained; according to the target event sequence and the at least one target image frame, The dynamic vision sensor and the image sensor are spatially registered to obtain the spatial registration parameters.
  • the target registration time domain refers to the same time domain corresponding to the dynamic vision sensor and the image sensor.
  • the image sources collected by the dynamic vision sensor and the image sensor are the same. That is, the event data (also referred to as event stream data) output by the dynamic vision sensor in the target registration time domain and the image frames output by the image sensor in the target registration time domain are registered in the time dimension.
  • the target registration time domain can be determined according to the event sequence number of the output event data; for the image sensor, the target registration time domain can be determined according to the number of output image frames.
  • the variation of the event sequence number corresponding to the dynamic vision sensor is obtained, and then output the frame number of the image frame according to the selected image sensor, and the dynamic visual sensor.
  • the time sequence of the event data output by the vision sensor is obtained, and each event data output by the dynamic vision sensor corresponding to the change of the event sequence number is obtained, which is the target event sequence output by the dynamic vision sensor in the target registration time domain.
  • the selected output image frame of the image sensor is the target image frame output by the image sensor in the target registration time domain.
  • the number of image frames output by the image sensor in the target registration time domain determined in this step is at least one.
  • the to-be-registered event frame is image-matched with the target image frame to obtain spatial registration parameters, so as to realize the spatial registration of the dynamic vision sensor and the image sensor .
  • E630 Acquire a registration result image based on the spatial registration parameter, the temporal registration parameter, the event sequence, and the image frame. That is, using the event data output by the dynamic vision sensor to perform image registration on the to-be-registered image frame, wherein the to-be-registered image frame is generated according to at least one moving object extracted from the image frame acquired by the image sensor.
  • a moving target refers to a moving target that needs to be tracked.
  • the image frame to be registered is generated according to at least one moving object extracted from the image frame output by the image sensor, and the image frame needs to be registered.
  • the image frame to be registered may include one or more moving objects, and the number of moving objects may be determined according to actual tracking requirements.
  • the image frame to be registered is generated by performing image processing on the image frame output by the image sensor.
  • the background area in the image frame output by the image sensor can be eliminated, so that only Including the moving target and the pixel data near the moving target, excluding the background pixel data in the image frame output by the image sensor.
  • the image frames to be registered can also be processed by intercepting the image frames output by the image sensor, for example, performing moving target recognition on the image frames output by the image sensor, and predicting the movement trend of the moving target.
  • the image frame output by the sensor is intercepted, and the obtained image frame includes the moving target and the area near the moving target to be registered.
  • the image frame to be registered only includes contour data of the moving object and pixel data near the contour of the moving object.
  • the data output by the dynamic vision sensor includes at least one of event data and an event stream feature frame constructed according to the event data.
  • the multiple event data are first screened based on the position coordinates to obtain part of the event data corresponding to the image frame to be registered, that is, part of the event data corresponding to each moving target in the image frame to be registered, and Image registration is performed on the image frame to be registered according to this part of the event data.
  • multiple data frames corresponding to the original video frame corresponding to the to-be-registered image frame can be obtained based on the temporal registration result.
  • event data, and construct multiple event data into event stream feature frames, based on the spatial registration result use event stream feature frames to perform image registration on to-be-registered image frames.
  • the filter out the multiple event data corresponding to each event data After determining the multiple event data collected by the dynamic vision sensor corresponding to the image frame to be registered according to the time registration result (for example, the time registration parameter), filter out the multiple event data corresponding to each event data. Part of the event data matching the position of the moving target, and according to this part of the event data, an event stream feature frame is constructed, and based on the spatial registration parameters, the event stream feature frame is used to perform image registration on the image frame to be registered.
  • the corresponding image frames to be registered are generated in real time.
  • the event data output by the dynamic vision sensor or the event stream feature frame pairs constructed from multiple event data are used.
  • Image registration is performed on each corresponding image frame to be registered, and a plurality of registered image frames corresponding to the moving target to be tracked can be obtained.
  • the motion track information of the at least one moving object may be determined according to a plurality of registered image frames obtained after performing image registration on the to-be-registered image frame using the data output by the dynamic vision sensor.
  • the moving target After obtaining multiple registered image frames corresponding to the moving target to be tracked, the moving target can be tracked by detecting the moving target in each registered image frame.
  • the smooth motion trajectory of the moving target to be tracked can be obtained through multiple registration image frames, that is, the accurate data of the moving target to be tracked at each moment can be obtained, so as to obtain the real-time online accurate data of the moving target to be tracked.
  • the image sensor and the dynamic vision sensor are temporally registered, the temporal registration result is determined, and the image sensor and the dynamic vision sensor are spatially registered according to the temporal registration result, and the spatial registration result is determined Then, based on the temporal registration result and the spatial registration result, the data output by the dynamic vision sensor can be used to perform image registration on the to-be-registered image frame generated by the moving target extracted from the image frame output by the image sensor.
  • image registration is performed on the image frame to be registered generated by the moving target extracted from the image frame output by the image sensor according to the data output by the dynamic vision sensor, and a certain image frame is generated.
  • FIG. 23 is a flowchart of an image registration method provided in Embodiment 6 of the present disclosure. This embodiment is embodied on the basis of the above-mentioned embodiment, wherein the image sensor and the dynamic vision sensor are spatially registered according to the target image frame and the target event data, and the spatial registration result is determined, Can include:
  • an event stream feature frame corresponding to at least one target image frame
  • the image registration method provided in this embodiment may include:
  • E710. Perform time registration on the dynamic vision sensor and the image sensor to obtain time registration parameters, where the time registration parameters represent the time correlation between the event sequence collected by the dynamic vision sensor and the image frames collected by the image sensor.
  • E720 Determine, according to the time registration result, the target image frame output by the image sensor and the target event data output by the dynamic vision sensor in the target registration time domain.
  • E730 Acquire a target event sequence in the target registration time domain according to the time registration parameter.
  • the target event sequence output by the dynamic vision sensor in the target registration time domain is a combination of multiple event data, that is, the event stream data output by the dynamic vision sensor in the target registration time domain.
  • the event data within a certain time period is collected together and constructed into image frames, which are called event stream feature frames.
  • the event data in the time range corresponding to the output of one image frame by the image sensor is collected together to construct the event flow feature frame.
  • E740 Acquire image frames to be registered in the target registration time domain according to the temporal registration parameters, where the to-be-registered image frames represent detailed features of moving objects in the image frames.
  • an image frame to be registered corresponding to the at least one target image frame is obtained by a preset method.
  • the target image frame output by the image sensor in the target registration time domain is acquired, the time sequence feature feature extraction is performed on the target image frame, and the to-be-registered image frame corresponding to the target image frame is generated.
  • the edge contour is extracted from the image collected by the image sensor according to the frame, and the corresponding image frame to be registered is generated.
  • the frame difference method or the filter filtering method may be used to extract the image edge contour, which is not specifically limited in this embodiment.
  • E750 Perform spatial registration according to the target event sequence and the to-be-registered image frame to obtain the spatial registration parameters.
  • the image edge contour determined by the target image frame output by the image sensor is similar to the light intensity change frame data collected by the dynamic vision sensor, that is, the image frame to be registered is similar to the constructed event flow feature frame.
  • the feature-based image registration method determines the image registration parameters of two image frames as the spatial registration parameters between the dynamic vision sensor and the image sensor.
  • the event stream feature frame is used as the reference image, and the feature description sub-image frame is used as the image to be registered.
  • the core steps of the feature-based image registration method are:
  • the image registration parameters can be determined based on the spatial transformation model used during registration, the similarity measure criterion for registration and the spatial transformation matrix;
  • Image conversion registration based on the determined image registration parameters, use the reference image to perform image registration on the image to be registered.
  • the event data acquired in the target registration time domain can be divided into multiple event sequences according to the number of image frames output by the image sensor. , and build an event stream feature frame based on one of the event sequences. Further, image registration is performed on the event stream feature frame using an image frame to be registered of an image frame corresponding to the event sequence, so as to determine the spatial registration parameters between the image sensor and the dynamic vision sensor.
  • E760 Determine motion information of at least one moving object in the image frame to be registered according to the target event sequence and the spatial registration parameter, and obtain the registration result image.
  • using the data output by the dynamic vision sensor to perform image registration on the image frame to be registered may include: based on the image acquisition spatial registration result, determining At least one piece of data output by the dynamic vision sensor, and an image frame output by the image sensor corresponding to the at least one piece of data; based on the spatial registration parameters, image registration is performed on the image frame using the at least one piece of data.
  • the data output by the dynamic vision sensor includes at least one of event data and an event flow feature frame constructed according to the event data.
  • the method provided in this embodiment before performing image registration on the to-be-registered image frame, the method provided in this embodiment further includes:
  • an initial weight matrix corresponding to the size can be constructed separately for each image frame output by the image sensor, and all regional weights in the initial weight matrix are set to zero; or a size can be constructed for each image frame output by the image sensor. Corresponding and general initial weight matrix, and all area weights in the initial weight matrix are set to zero.
  • an initial weight matrix to obtain a target assignment weight matrix including:
  • the second area matching the predicted distribution area in the frame is assigned to obtain a second assignment weight matrix; and the target assignment weight matrix is generated by combining the first assignment weight matrix and the second assignment weight matrix.
  • the method can generate image frames to be registered corresponding to each target image frame.
  • the moving target to be tracked in the target image frame and assign a value to the matching area (ie, the first area) in the initial weight matrix according to the current distribution area of the moving target to be tracked in the target image frame.
  • the first assignment weight matrix For example, the frame difference method can be used to obtain the moving target to be tracked in the target image frame, and then the area weight corresponding to the moving target to be tracked obtained based on the difference method is set to 1, which is not specifically limited in this implementation.
  • the target decomposition is performed on each moving object in the target image frame, and multiple moving closed target regions are extracted through the motion spatial continuity feature, and then the motion of each moving object is obtained through the spatial position of the frame before and after the video image.
  • Vector field For a moving target to be tracked, through the motion vector field of the moving target to be tracked, combined with the historical motion trajectory (such as direction and speed, etc.) of the moving target to be tracked, the weighted prediction of the center point of the moving target to be tracked at the next target
  • the spatial position in the image frame is combined with its mesh division (moving objects of different sizes correspond to mesh divisions of different sizes) to obtain the predicted distribution area of the moving object to be tracked in the next target image frame.
  • the OR operation is performed on the first assignment weight matrix and the second assignment weight matrix to obtain the target assignment weight matrix.
  • the distribution area that is, the predicted distribution area
  • the matching area ie, the second area
  • the assignment is 1, the above target assignment weight matrix can be directly obtained.
  • a dot product operation is performed on the target assignment weight matrix and the target image frame to obtain the registered image frame for the moving target to be tracked.
  • the registration image frame includes each moving object to be tracked and its nearby pixel data, or the outline data of each moving object to be tracked and the pixel data near the outline, that is, the registration image frame extracts the data from the target image frame.
  • the above technical solution can track and identify moving objects under low storage and low transmission requirements, and achieves smooth tracking of the moving object's trajectory; at the same time, the feature-based image registration method is applied to event stream data, providing a
  • the implementation of processing event stream data solves the problem that most existing image processing methods and image detection methods cannot be directly used for processing event streams.
  • FIG. 24 is a flowchart of an image registration method provided in Embodiment 7 of the present disclosure. This embodiment provides an optional implementation, wherein the image sensor is a CMOS image sensor.
  • the image registration method provided in this embodiment may include:
  • E810 Perform time registration on the CMOS image sensor and the dynamic vision sensor, and determine the time registration parameters between the CMOS image sensor and the dynamic vision sensor.
  • CMOS image sensor and dynamic vision sensor should be consistent with the actual use scene.
  • the spatial registration between the CMOS image sensor and the dynamic vision sensor is performed based on the temporal registration parameters.
  • CMOS image sensor and dynamic vision sensor should be consistent with the actual use scene, that is, consistent with the E810 time registration scene.
  • E830 Construct an event flow feature frame according to the event data output by the dynamic vision sensor in the target registration time domain.
  • the feature description sub-image frame is obtained by a frame difference method or a filter filtering method.
  • E850 Perform feature-based image registration on the feature description sub-image frame and the event stream feature frame to obtain spatial registration parameters between the CMOS image sensor and the dynamic vision sensor.
  • E860 based on the time registration parameters, determine the event data output by the dynamic vision sensor and the image frame output by the CMOS image sensor in the same time domain.
  • E870 Construct an initial weight matrix corresponding to the image frame size output by the CMOS image sensor.
  • the obtained image frames to be registered are in one-to-one correspondence with the target image frames output by the CMOS image sensor in the target time domain.
  • E890 Acquire one image frame to be registered and event data collected by a dynamic vision sensor corresponding to the image frame to be registered in sequence.
  • E8100 Construct an event stream feature frame according to the event data, and based on the spatial registration parameters, use the event stream feature frame to perform image registration on the image frame to be registered, generate a corresponding registered image frame, and return to execute E890.
  • CMOS image sensor is used to track and extract moving objects and local detail features (and only pixel data near the moving object is considered), and a dynamic vision sensor is used to supplement the moving objects. Motion details and local behavior micro-movement feature information, thus forming a real-time online accurate feature information description of the moving target to be tracked.
  • FIG. 25 is a schematic structural diagram of an image registration device provided in Embodiment 8 of the present disclosure, which can be applied to the case of image registration between a dynamic vision sensor and a traditional image sensor, and the device can be implemented by software and/or hardware. , and can generally be integrated in a computer device, for example, a computer device connected to a dynamic vision sensor and a traditional image sensor, and the computer device can receive and process the data collected by the dynamic vision sensor and the traditional image sensor.
  • the image registration apparatus includes: an inter-sensor temporal registration module 610 , an inter-sensor spatial registration module 620 and an image registration module 630 . in,
  • the inter-sensor time registration module 610 is configured to perform time registration on the dynamic vision sensor and the image sensor;
  • the inter-sensor spatial registration module 620 is configured to perform spatial registration on the dynamic vision sensor and the image sensor according to the temporal registration result;
  • the image registration module 630 is configured to perform image registration on the data output by the dynamic vision sensor using the image frames output by the image sensor based on the spatial registration result and the temporal registration result.
  • the dynamic vision sensor and the image sensor firstly perform temporal registration, and then perform spatial registration based on the temporal registration result, and then based on the temporal registration result between the dynamic vision sensor and the image sensor Using the image frame output by the image sensor to perform image registration on the event stream feature frame constructed by the event data output by the dynamic vision sensor, the image registration between the dynamic vision sensor and the image sensor is realized.
  • the object perception ability of dynamic vision sensor images is stronger after registration.
  • the inter-sensor spatial registration module 620 includes:
  • a spatial registration data acquisition unit configured to acquire, according to the temporal registration result, the target event data output by the dynamic vision sensor in the target registration time domain, and the target image output by the image sensor in the target registration time domain frame;
  • the inter-sensor spatial registration unit is configured to perform spatial registration on the dynamic vision sensor and the image sensor according to the target event data and the target image frame.
  • the inter-sensor time registration module 610 includes:
  • the sensor image acquisition timing information acquisition unit is configured to, in response to at least two shooting scene changes, respectively acquire an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each of the shooting scene changes, and an event sequence number corresponding to each of the shooting scene changes. the frame number of a group of image frames output by the image sensor corresponding to the change of the shooting scene;
  • the inter-sensor time registration unit is configured to determine the time registration parameters between the dynamic vision sensor and the image sensor according to the event sequence number of each of the event sequences and the frame number of each group of the image frames, and to The time registration parameter is used as the time registration result.
  • the inter-sensor time registration unit is configured to determine the event sequence number mean value of each of the event sequences respectively; according to the event sequence number mean value of each of the event sequences and the frame numbers of each group of the image frames, respectively determine the event sequence number.
  • Each corresponding ratio of the variation of the mean value to the variation of the frame number, and the mean value of the respective corresponding ratios is used as the time registration parameter.
  • the inter-sensor spatial registration unit is configured to construct, according to the target event data, an event stream feature frame corresponding to at least one target image frame; determine a feature description sub-image frame corresponding to the at least one target image frame; The event stream feature frame corresponding to the at least one target image frame is subjected to feature-based image registration with the feature description sub-image frame to obtain the spatial registration parameters between the dynamic vision sensor and the image sensor, and the The spatial registration parameters are used as the spatial registration result.
  • the image registration module 630 may be configured to determine, based on the temporal registration result, an image frame output by the image sensor and at least one piece of data collected by the dynamic vision sensor corresponding to the image frame; The at least one data is image-registered using the image frame based on the spatial registration parameters.
  • the data output by the dynamic vision sensor includes at least one of event data and an event flow feature frame constructed according to the event data.
  • the above image registration apparatus can execute the image registration method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the executed image registration method.
  • FIG. 26 is a schematic structural diagram of an image registration device provided in Embodiment 9 of the present disclosure, which can be applied to the case of temporal registration of a dynamic vision sensor and a traditional image sensor.
  • the device can be implemented by software and/or hardware. , and can generally be integrated in a computer device, for example, a computer device connected to a dynamic vision sensor and a traditional image sensor, and the computer device can receive and process the data collected by the dynamic vision sensor and the traditional image sensor.
  • the image registration apparatus includes: a sensor image acquisition timing information acquisition module 710 and an inter-sensor time registration module 720 . in,
  • the sensor image acquisition timing information acquisition module 710 is configured to, in response to at least two shooting scene changes, acquire an event sequence number of an event sequence output by the dynamic vision sensor corresponding to each of the shooting scene changes, and the event sequence number corresponding to each of the shooting scene changes.
  • the inter-sensor time registration module 720 is configured to determine time registration parameters between the dynamic vision sensor and the image sensor according to the event sequence numbers of each of the event sequences and the frame numbers of each group of the image frames.
  • the technical solution provided in this embodiment realizes time registration between sensors of different modes, and is suitable for application scenarios where time registration is required.
  • the inter-sensor time registration module 720 is configured to determine the mean value of the event sequence numbers of each of the event sequences respectively; according to the mean value of the event sequence numbers of each of the event sequences and the frame numbers of each group of the image frames, determine the events respectively.
  • Each corresponding ratio of the variation of the mean value of the serial number and the variation of the frame number, and the mean value of the respective corresponding ratios is used as the time registration parameter.
  • the above image registration apparatus can execute the image registration method provided by any embodiment of the first aspect of the present disclosure, and has functional modules and beneficial effects corresponding to the executed image registration method of the first aspect.
  • FIG. 27 is a schematic structural diagram of a computer device according to Embodiment 10 of the present disclosure.
  • the computer device includes a processor 810, a memory 820, an input device 830 and an output device 840; the number of processors 810 in the computer device can be one or more, and one processor 810 is taken as an example in FIG. 27 ;
  • the processor 810, the memory 820, the input device 830 and the output device 840 in the computer equipment may be connected by a bus or in other ways. In FIG. 27, the connection by a bus is taken as an example.
  • the memory 820 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the image registration method described in the first aspect of the embodiments of the present disclosure (for example, FIG.
  • the temporal registration module 100 in the image registration device shown in FIG. 15 ) is also like the program instructions/modules corresponding to the image registration method described in the second aspect of the embodiment of the present disclosure (for example, the image registration shown in FIG. 16 )
  • the processor 810 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 820 , that is, to implement the above-mentioned image registration method or the above-mentioned image registration method.
  • the memory 820 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of computer equipment, and the like. Additionally, memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some instances, memory 820 may further include memory located remotely from processor 810, which may be connected to the computer device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • the input device 830 may be used to receive input numerical or character information, and to generate key signal input related to user settings and function control of the computer device.
  • the output device 840 may include a display device such as a display screen.
  • the eleventh embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, where the computer program is used to execute an image registration method when executed by a computer processor, including:
  • time registration parameters represent the time correlation between the sequence of events collected by the dynamic vision sensor and the image frames collected by the image sensor
  • the dynamic vision sensor and the image sensor are spatially registered according to the temporal registration parameters, and the spatial registration parameters are obtained.
  • the spatial registration parameters represent the sequence of events collected by the dynamic vision sensor and the space of the image frames collected by the image sensor. connection relation;
  • a registration result image is acquired according to the spatial registration parameters, the sequence of events and the image frame.
  • the computer-readable storage medium storing the computer program provided by the embodiment of the present disclosure is not limited to the above method operations, and can also perform related operations in the image registration method provided by any embodiment of the present disclosure.
  • ROM read only memory
  • RAM random access memory
  • FLASH flash memory
  • hard disk or optical disk etc., including several instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute this
  • a computer device which can be a personal computer, server, or network device, etc.
  • the units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized That is all; in addition, the specific names of the functional units are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)

Abstract

本公开提供一种图像配准方法，包括：对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。本公开还提供一种图像配准装置、一种计算机设备、一种计算机可读介质。

Description

图像配准方法及装置、计算机设备、介质 技术领域
本公开涉及图像处理技术领域，尤其涉及一种图像配准方法、一种图像配准装置、一种计算机设备、一种计算机可读介质。
背景技术
动态视觉传感器(Dynamic Vision Sensor，DVS)擅长捕获运动物体的时域信息，具有很强的时间灵敏度；互补金属氧化物半导体(CMOS，Complementary Metal Oxide Semiconductor)图像传感器擅长捕捉帧为单位的图像信息，具有高分辨率、高成像质量等特点。
如何克服DVS和CMOS图像传感器各自的弱点和局限是亟待解决的问题。
发明内容
本公开实施例提供一种图像配准方法、一种图像配准装置、一种计算机设备、一种计算机可读介质。
第一方面，本公开实施例提供一种图像配准方法，包括：
对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
在一些实施例中，对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数的步骤包括：
获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
在一些实施例中，获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记的步骤包括：
获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；
计算多个事件数据的时间戳的平均值，作为所述时间标记。
在一些实施例中，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数的步骤包括：
确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；
计算多个所述相对比例的平均值作为所述时间配准参数。
第二方面，本公开实施例提供一种图像配准方法，包括：
根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；
其中，所述时间配准参数为根据第一方面任意一项所述的图像配准方法获得的。
在一些实施例中，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数的步骤包括：
根据所述目标事件序列构建待配准事件帧；
根据所述至少一个目标图像帧获取特征描述子图像帧；
对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
在一些实施例中，根据所述目标事件序列构建待配准事件帧的步骤包括：
对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
在一些实施例中，根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像的步骤包括：
根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
在一些实施例中，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧的步骤包括：
根据多个所述图像帧确定至少一个运动目标；
获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
在一些实施例中，获取所述至少一个运动目标的像素数据，生成所述待配准图像帧的步骤包括：
构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；
根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；
将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
在一些实施例中，根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵，包括：
对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；
对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；
结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
在一些实施例中，根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像的步骤包括：
根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
第三方面，本公开实施例提供一种图像配准装置，包括：
时间配准模块，用于对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
在一些实施例中，所述时间配准模块包括：
事件数据处理单元，用于获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
图像帧处理单元，用于获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
时间配准单元，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
在一些实施例中，所述事件数据处理单元用于获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；计算多个事件数据的时间戳的平均值，作为所述时间标记。
在一些实施例中，所述时间配准单元用于确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；计算多个所述相对比例的平均值作为所述时间配准参数。
第四方面，本公开实施例提供一种图像配准装置，包括：
空间配准模块，用于根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
图像配准模块，用于根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；
其中，所述时间配准参数为根据第一方面任意一项所述的图像配准方法获得的。
在一些实施例中，所述空间配准模块包括：
第一数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
第一空间配准单元，用于根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，所述第一数据处理单元用于根据所述目标事件序列构建待配准事件帧；根据所述至少一个目标图像帧获取特征描述子图像帧；对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
在一些实施例中，所述第一数据处理单元用于对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
在一些实施例中，所述图像配准模块用于根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
在一些实施例中，所述空间配准模块包括：
第二数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
所述第二数据处理单元还用于根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
第二空间配准单元，用于根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，所述第二数据处理单元用于根据多个所述图像帧确定至少一 个运动目标；获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
在一些实施例中，第二数据处理单元用于构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
在一些实施例中，第二数据处理单元，用于对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
在一些实施例中，所述图像配准模块用于根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
第五方面，本公开实施例提供一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如第一方面任一所述的方法，和/或实现如第二方面任一所述的方法。
第六方面，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如第一方面任一所述的方法，和/或实现如第二方面任一所述的方法。
本公开实施例提供的技术方案，针对动态视觉传感器和图像传感器，首先进行时间配准，然后时间配准结果进行空间配准，进而可以基于动态视觉传感器和图像传感器之间的时间配准结果和空间配准结果，使用图像传感器输出的图像帧对由动态视觉传感器输出的数据进行图像配准，以此实现了动态视觉传感器和图像传感器之间的图像配准，配准后动态视觉传感器图像的物体感知能力更强；还能够根据动态视觉传感器输出的数据对根据图像传感器输出的图像帧提取的运动目标生成的待配准图像帧进行图像配准，生成某一时刻运动目标的精准数据，以此提高了进行运动目标跟踪时所采用图像的精准度，进而可以在低存储低传输要求下基于配准后的图像进行运动目标跟踪识别，减小了对数据算力、传输带宽、数据存储的要求，也减少了使用目标检测跟踪目标而造成算力浪费的问题，还能够实现对运动目标运动轨迹的平滑跟踪，得到待跟踪的运动目标的实时在线精准数据。
附图说明
图1是本公开实施例中一种图像配准方法的流程图；
图2是本公开实施例中另一种图像配准方法中部分步骤的流程图；
图3是本公开实施例中又一种图像配准方法中部分步骤的流程图；
图4是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图5是本公开实施例中一种图像配准方法的流程图；
图6是本公开实施例中另一种图像配准方法中部分步骤的流程图；
图7是本公开实施例中又一种图像配准方法中部分步骤的流程图；
图8是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图9是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图10是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图11是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图12是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图13是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图14是本公开实施例中再一种图像配准方法中部分步骤的流程图；
图15是本公开实施例中一种图像配准装置的组成框图；
图16是本公开实施例中一种图像配准装置的组成框图；
图17是本公开实施例中图像配准方法的一种实施例的流程图；
图18是本公开实施例中图像配准方法的一种实施例的流程图；
图19是本公开实施例中基于特征的图像匹配的示意图；
图20是本公开实施例中图像配准方法的一种实施例的流程图；
图21是本公开实施例中图像配准方法的一种实施例的流程图；
图22是本公开实施例中图像配准方法的一种实施例的流程图；
图23是本公开实施例中图像配准方法的一种实施例的流程图；
图24是本公开实施例中图像配准方法的一种实施例的流程图；
图25是本公开实施例中图像配准装置的一种实施例的结构示意图；
图26是本公开实施例中图像配准装置的一种实施例的结构示意图；
图27是本公开实施例中一种计算机设备的结构示意图。
具体实施方式
下面结合附图和实施例对本公开作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本公开，而非对本公开的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本公开相关的部分而非全部结构。
在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将各项操作(或步骤)描述成顺序的处理，但是其中的许多操作可以被并行地、并发地或者同时实施。此外，各项操作的顺序可以被重新安排。当其操作完成时所述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。所述处理可以对应于方法、函数、规程、子例程、子程序等等。
为了便于理解，将本公开实施例的主要发明构思进行简述。
动态视觉传感器是一种事件驱动型光电传感器，根据每个像素单元独立感受的光电变化，将激活像素点的位置、事件(ON/OFF)、时间戳等信息编码输出，输出的是事件数据，如<xd,yd,t,c>所示，其中，<xd,yd>代表光强变化的像素的空间位置，t代表时间戳，c代表光强增强或减弱的变化量。动态视觉传感器具有高灵敏度的特点，仅捕捉运动变化的目标，不记录背景信息，减少了生成的数据量，从而降低了对数据存储、数据算力和传输带宽的要求。
传统的图像传感器擅长捕获以帧为单位的图像信息，以COMS图像传感器为例，其具有很高的图像分辨率，但时间分辨率比较低，容易造成运动模糊，且生成的数据量大。其中，传统的图像传感器采用全幅触发机制，以点为单位，按帧输出图像，所有像素单元曝光固定时长，并全部一次性输出，具有高分辨率(像素数)、成像质量高的特点，但由于信息量大，所以对数据存储、数据算力和传输带宽要求高。
动态视觉传感器与传统图像传感器结合可以有效地实现高静态空间分辨率和高动态时间分辨率的双高感知。然而，由于两种传感器在焦距、视野范围、光学器件、曝光原理、数据传输方式等各方面均存在不同，导致两种传感器采集的数据很难在像素级别进行精确对应，影响了二者协同工作时对物体空时感知的精确性。因此，如何实现动态视觉传感器和传统图像传感器之间的图像配准是亟待解决的问题， 在完成二者之间的图像配准之后，即可根据传统图像传感器输出的图像帧对动态视觉传感器图像进行配准，进而提升动态视觉传感器图像的物体感知能力。
基于上述思考，发明人创造性地提出了一种图像配准方法，该方法包括：对动态视觉传感器和图像传感器进行时间配准；根据时间配准结果，对动态视觉传感器和图像传感器进行空间配准；基于空间配准结果以及时间配准结果，使用图像传感器采集的图像帧对动态视觉传感器采集的事件数据进行图像配准，或使用动态视觉传感器对图像传感器采集的图像帧进行图像配准。
第一方面，参照图1，本公开实施例提供一种图像配准方法，包括：
S100、对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
在一些实施例中，图像传感器为CMOS图像传感器。
需要说明的是，在本公开实施例中，执行步骤S100对动态视觉传感器和图像传感器进行时间配准时，需要先确定动态视觉传感器和图像传感器的使用设备、场景配置。其中，配置的场景应于实际使用的场景保持一致。
在本公开实施例中，通过步骤S100能够实现对动态视觉传感器和图像传感器的多模时间配准，有利于在时间配准的基础上进一步对动态视觉传感器和图像传感器进行空间配准，最终实现图像配准。
本公开实施例对于如何对动态视觉传感器和图像传感器进行时间配准不做特殊限定。
在一些实施例中，参照图2，对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数的步骤包括：
S110、获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
S120、获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
S130、根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
需要说明的是，需要进行图像配准的动态视觉传感器与图像传感器的拍摄场景相同。时间配准，指的是将动态视觉传感器的输出信息和图像传感器的输出信息以图像采集时间为维度进行配准。
在本公开实施例中，多个配准事件为多次重复发生的配准事件。例如，采用一个闪烁的光源以脉冲形式发光，对应于每一个脉冲的光源的一次闪烁为一次配准事件。
在本公开实施例中，每发生一次配准事件，动态视觉传感器都产生由一系列事件数据组成的事件序列，事件序列的时间标记用于总体上表征对应于配准事件的事件数据的时间信息。其中，对应于同一配准事件的事件序列和图像帧在时间上存在关联关系。在步骤S130中确定的时间配准参数能够表征对应于同一配准事件的事件序列和图像帧在时间上的关联关系，也即表征动态视觉传感器采集的事件序列和图像传感器采集的图像帧的时间关联关系。
本公开实施例对于如何确定事件序列的时间标记不做特殊限定。例如，可以将事件序列中第一个事件数据的时间戳作为事件序列的时间标记；也可以将事件序列中最后一个事件数据的时间戳作为事件序列的时间标记；还可以将事件序列中间的任意一个事件数据的时间戳作为事件序列的时间标记。
在一些实施例中，参照图3，获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记的步骤包括：
S111、获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；
S112、计算多个事件数据的时间戳的平均值，作为所述时间标记。
本公开实施例对于如何根据多个时间标记和多个帧号确定所述时间配准参数不做特殊限定。
在一些实施例中，参照图4，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数的步骤包括：
S131、确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；
S132、计算多个所述相对比例的平均值作为所述时间配准参数。
需要说明的是，在本公开实施例中，通过时间配准确定了时间配准参数后，即可以根据时间参数表征的动态视觉传感器采集的事件序列和图像传感器采集的图像帧的时间关联关系，获取同一时间域内满足该时间关联关系的事件序列和图像帧，从而有利于在同一时间域内对动态视觉传感器和图像传感器进行空间配准。
下面举例对本公开实施例中对动态视觉传感器和图像传感器进行时间配准进行说明。
采用一个闪烁的光源以脉冲形式发光，分别用动态视觉传感器和图像传感器录制此场景。当动态视觉传感器检测到闪烁光源时，会在光强变化瞬间(增强或减弱)输出事件序列。取对应于一个配准事件的事件序列中多个事件数据的时间戳t的均值即可得到对应于该配准事件的事件序列的时间标记。同时记录图像传感器输出的对应于该配准事件的图像帧的帧号n，将帧号n与时间标记进行关联，即可得到时间标记与帧号n的对应关系。之后光源再次闪烁时，记录第二次的时间标记和帧号n的对应关系，从而得到相邻两次配准事件之间时间标记的变化量与帧号的变化量的关系，进而得到两者的相对比例。多次执行上述过程，将得到的多个相对比例取平均值，作为时间配准参数。
第二方面，参照图5，本公开实施例提供一种图像配准方法，包括：
S200、根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
S300、根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；其中，所述时间配准参数为根据第一方面任意一项所述的图像配准方法获得的。
需要说明的是，本公开实施例中，步骤S200和步骤S300是在对动态视觉传感器和图像传感器进行时间配准的基础上进行的空间配准。空间配准，指的是将动态视觉传感器的输出信息和图像传感器的输出信息以图像信息(例如像素)为维度进行配准。
在本公开实施例中，通过步骤S300得到的配准结果图像，可以是用图像传感器采集的图像帧对动态视觉传感器输出的图像进行配准得到的，也可以是用动态视觉传感器采集的事件序列对图像传感器输出的图像进行配准得到的。本公开实施例对此不做特殊限定。
在本公开实施例中，用图像传感器采集的图像帧对动态视觉传感器输出的图像进行配准，能够弥补动态视觉传感器对静态图像的捕获能力的不足，用动态视觉传感器采集的事件序列对图像传感器输出的图像进行配准能够弥补图像传感器时间分辨率的不足，从而实现高静态空间分辨率、高动态时间分辨率的双高感知。
在一些实施例中，参照图6，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
S210、根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
S220、根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
需要说明的是，目标配准时间域，指的是动态视觉传感器和图像传感器对应的一个相同时间域，在目标配准时间域内动态视觉传感器和图像传感器采集的图像源是相同的。即，通过步骤S210获取的目标事件序列和目标图像帧在时间维度是配准的。
本公开实施例对于如何根据目标事件序列和至少一个目标图像帧进行空间配准得到空间配准参数不做特殊限定。
在一些实施例中，参照图7，根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数的步骤包括：
S221、根据所述目标事件序列构建待配准事件帧；
S222、根据所述至少一个目标图像帧获取特征描述子图像帧；
S223、对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
本公开实施例对于如何根据目标事件序列构建待配准事件帧不做特殊限定。
在一些实施例中，参照图8，根据所述目标事件序列构建待配准事件帧的步骤包括：
S2211、对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
在本公开实施例中，通过步骤S210至步骤S220得到的空间配准参数，能够用于用图像传感器采集的图像帧对动态视觉传感器输出的图像进行配准。
相应地，在一些实施例中，参照图9，根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像的步骤包括：
S310、根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
在本公开实施例中，通过步骤S310得到的配准结果图像为整合了运动目标的细节信息的DVS图像。在一些实施例中，运动目标的细节信息包括运动目标的轮廓、边缘、颜色等。本公开实施例对此不做特殊限定。
在本公开实施例中，因为待配准事件帧与特征描述子图像帧在同一个时间域内具有类似性，在一些实施例中，通过基于特征的图像匹配，得到配准结果图像。
在一些实施例中，通过基于特征的图像匹配得到配准结果图像，包括：
1)特征检测，提取图像特征，如边缘、轮廓等；
2)特征匹配，用特征描述符、相似性度量等建立基准图像和待配准图像之间的相关性；
3)转换模型估计，获得配准参数；
4)图像转换配准。
在一些实施例中，参照图10，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
S230、根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
S240、根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
S250、根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，用图像传感器进行运动目标跟踪，获取运动目标及其附件的像素数据，得到待配准图像帧。待配准图像帧中仅保留运动目标及其附件的像素数 据，相比于一些相关技术中先识别运动目标在对运动目标进行跟踪，或将视野集中在目标附件进行稀疏跟踪，能够消除冗余信息，节省带宽和算力。
本公开实施例对于如何获取待配准图像帧不做特殊限定。
在一些实施例中，参照图11，根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧的步骤包括：
S241、根据多个所述图像帧确定至少一个运动目标；
S242、获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
在一些实施例中，参照图12，获取所述至少一个运动目标的像素数据，生成所述待配准图像帧的步骤包括：
S2421、构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；
S2422、根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；
S2423、将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
在一些实施例中，参照图13，根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵，包括：
S2422a、对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；
S2422b、对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；
S2422c、结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
在一些实施例中，参照图14根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像的步骤包括：
S320、根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
在一些实施例中，运动目标的运动信息可以是运动目标的运动轨迹。通过步骤S320得到的配准结果图像是整合了运动目标的运动信息的图像帧。
第三方面，参照图15，本公开实施例提供一种图像配准装置，包括：
时间配准模块100，用于对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
在一些实施例中，所述时间配准模块包括：
事件数据处理单元，用于获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
图像帧处理单元，用于获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
时间配准单元，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
在一些实施例中，所述事件数据处理单元用于获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；计算多个事件数据的时间戳的平均值，作为所述时间标记。
在一些实施例中，所述时间配准单元用于确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；计算多个所述相对比例的平均 值作为所述时间配准参数。
第四方面，参照图16，本公开实施例提供一种图像配准装置，包括：
空间配准模块200，用于根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
图像配准模块300，用于根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；
其中，所述时间配准参数为根据第一方面任意一项所述的图像配准方法获得的。
在一些实施例中，所述空间配准模块包括：
第一数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
第一空间配准单元，用于根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，所述第一数据处理单元用于根据所述目标事件序列构建待配准事件帧；根据所述至少一个目标图像帧获取特征描述子图像帧；对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
在一些实施例中，所述第一数据处理单元用于对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
在一些实施例中，所述图像配准模块用于根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
在一些实施例中，所述空间配准模块包括：
第二数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
所述第二数据处理单元还用于根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
第二空间配准单元，用于根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
在一些实施例中，所述第二数据处理单元用于根据多个所述图像帧确定至少一个运动目标；获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
在一些实施例中，第二数据处理单元用于构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
在一些实施例中，第二数据处理单元，用于对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
在一些实施例中，所述图像配准模块用于根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
实施例一
图17是本公开实施例一提供的一种图像配准方法的流程图，可适用于对动态视觉传感器与传统图像传感器进行图像配准的情况，该方法可以由本公开实施例提 供的图像配准装置来执行，该装置可采用软件和/或硬件的方式实现，并一般可集成在计算机设备中，例如可以是与动态视觉传感器以及传统图像传感器建立连接的计算机设备，该计算机设备能够接收并处理接收动态视觉传感器以及传统图像传感器的采集数据。
如图17所示，本实施例提供的图像配准方法，包括：
E110、对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
动态视觉传感器和图像传感器，分别指的是需要进行图像配准的一个动态视觉传感器与一个图像传感器，且二者的拍摄场景是相同的。其中，图像传感器采集图像信息后输出的是图像帧，可选的，此类图像传感器为COMS图像传感器。
时间配准，指的是将动态视觉传感器的输出信息和图像传感器的输出信息以图像采集时间为维度进行配准。可选的，将针对同一拍摄场景采集图像信息时图像传感器输出的图像帧与动态视觉传感器输出的事件数据组成的事件序列在时序方面进行配准，以实现对动态视觉传感器和图像传感器进行时间配准。
在一些实施例例中，可以将动态视觉传感器输出的事件序列(由多个事件数据组成)的事件序号信息与图像传感器输出的图像帧的帧号进行配准，以实现对动态视觉传感器和图像传感器进行时间配准。其中，事件序号即为动态视觉传感器输出事件数据的时间戳的序号。
可选的，通过确定动态视觉传感器和图像传感器之间的时间配准参数来实现对动态视觉传感器和图像传感器进行时间配准。其中，时间配准参数可以是以图像采集时间为统计维度时动态视觉传感器和图像传感器的输出信息数量之间的对应比例关系。
在些实施例中，E110可以包括：响应于至少两次拍摄场景变化，分别获取与每次拍摄场景变化对应的动态视觉传感器输出的一个事件序列的事件序号，以及与每次拍摄场景变化对应的图像传感器输出的一组图像帧的帧号；根据各个事件序列的事件序号以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，并将时间配准参数作为时间配准结果。
其中，拍摄场景变化指的是动态视觉传感器或者图像传感器采集的图像信息发生变化，例如可以是拍摄场景中存在移动的目标物体，例如还可以是拍摄场景中光强(或者光源等)发生变化，等等。
在每次拍摄场景发生变化时，确定动态视觉传感器在此次变化过程中输出的事件序列，并获取该事件序列中与每个事件数据对应的事件序号，同时确定图像传感器在此次变化过程中输出的一组图像帧(其中，一组图像帧中可以包括一个或多个图像帧)，并获取其中每个图像帧的帧号。
进而，根据在多次拍摄场景发生变化的过程中获取到的各个事件序列的事件序号信息以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，也即确定以图像采集时间为维度动态视觉传感器输出的事件数量与图像传感器的输出的图像帧数量之间的对应比例关系，如在同一个拍摄场景中动态视觉传感器输出的m个事件数据对应于图像传感器输出的一个图像帧。
可选的，对在多次拍摄场景发生变化的过程中获取到的各个事件序列的事件序号信息以及各组图像帧的帧号进行统计分析，根据统计分析结果确定动态视觉传感器和图像传感器之间的时间配准参数。
在一些实施例中，根据各个事件序列的事件序号以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，可以包括：
分别确定每个事件序列的事件序号均值；根据各个事件序列的事件序号均值以及各组图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将各个对应比例的均值作为时间配准参数。
假设，拍摄场景变化的次数为n，在第一次拍摄场景变化中获取到的一个事件序列中各个事件的事件序号分别为t10、t11、…、t1m，取t10、t11、…、t1m的均值t1avg作为该事件序列的事件序号均值，类似的，计算得到的各个事件序列的事件序号均值t1avg、t2avg、…、tnavg。
事件序号均值的变化量，指的是与连续两次拍摄场景变化分别对应的两个连续事件序列的事件序号均值的差值，即为tnavg-t(n-1)avg；帧号的变化量，指的是与连续两次拍摄场景变化分别对应的两组视频帧中最后一个图像帧的帧号的差值，假设n组视频帧中最后一个图像帧的帧号依次为n1、n2、…、nn，则连续两组图像帧的帧号的变化量为nn-n(n-1)。
值得指出的是，各组图像帧的帧号是连续的，例如在第一次拍摄场景变化中获取到的一组图像帧中各个图像帧的帧号为1、2、…、n1，则在第二次拍摄场景变化中获取到的一组图像帧的帧号从n1+1开始计数。
事件序号均值的变化量与帧号的变化量的比例，指的是与连续两次拍摄场景变化对应的事件序号均值的变化量和帧号的变化量的比值，即(tnavg-t(n-1)avg)/(nn-n(n-1))。
根据拍摄场景变化次数n，可以得到(n-1)个事件序号均值的变化量与帧号的变化量的对应比例，计算这(n-1)个对应比例的均值，并将该均值作为时间配准参数。
在一种示例性的实施方式中，采用一个闪烁的光源来实现拍摄场景的变化，该光源以脉冲形式发光，分别使用动态视觉传感器和图像传感器来录制此场景。当动态视觉传感器检测到闪烁光源时，会在光强变化瞬间(增强或减弱)输出事件序列，取事件序列中各个事件的事件序号的均值作为动态视觉传感器的时间标记。同时，图像传感器进行录像并输出各个图像帧的帧号，将最后一个图像帧的帧号与动态视觉传感器的时间标记进行关联，即可得到一个动态视觉传感器的时间标记与图像传感器的图像帧号的对应关系。随着光源再次闪烁，记录下一个动态视觉传感器的时间标记与图像传感器的图像帧号的对应关系，从而可以得到时间标记的变化量(也即事件序号均值的变化量)与图像帧号的变化量的关系，也即时间标记的变化量与图像帧号的变化量的相对比例。类似的，可以得到多个时间标记的变化量与图像帧号的变化量的相对比例，取这些相对比例的平均值，作为动态视觉传感器与图像传感器之间的时间配准参数。
在另一种可选的实施方式中，还可以分别计算事件序号均值变化量的均值，以及帧号变化量的均值，将事件序号均值变化量的均值与帧号变化量的均值的比值作为时间配准参数。
在一种可选的示例中，时间配准参数为事件序号变化量与帧号变化量的比例，其中，帧号变化量为1，也即时间配准参数指示图像传感器输出一个图像帧对应的动态视觉传感器的事件序号变化量。
E120、根据时间配准参数，对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系。
时间配准结果，指的是动态视觉传感器与图像传感器关于图像采集时间的配准详情，可选的，时间配准结果即为时间配准参数。
空间配准，指的是将动态视觉传感器和图像传感器的输出信息以图像信息为维 度进行配准。可选的，将动态视觉传感器和图像传感器在目标配准时间域内输出的图像信息在像素方面进行配准，以实现对动态视觉传感器和图像传感器进行空间配准。
基于时间配准结果，获取时间配准后的动态视觉传感器的输出数据以及图像传感器的输出数据，对这些输出数据进行空间配准即可实现对动态视觉传感器和图像传感器的空间配准。
作为一种可选的实施方式，根据时间配准结果，对动态视觉传感器和图像传感器进行空间配准，可以包括：
根据时间配准参数，获取动态视觉传感器在目标配准时间域内输出的目标事件序列，以及图像传感器在目标配准时间域内输出的至少一个目标图像帧；根据目标事件序列以及至少一个目标图像帧，对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数。
目标配准时间域，指的是与动态视觉传感器以及图像传感器对应的一个相同时间域，在目标配准时间域内动态视觉传感器和图像传感器采集的图像源是相同的。也即，动态视觉传感器在目标配准时间域内输出的事件数据(也可称之为事件流数据)，与图像传感器在目标配准时间域内输出的图像帧，在时间维度是配准的。
其中，针对动态视觉传感器而言，目标配准时间域可以根据输出事件数据的事件序号来确定；针对图像传感器而言，目标配准时间域可以根据输出图像帧的数量来确定。
例如，可以首先根据选取的图像传感器输出图像帧的帧号变化量以及时间配准参数，确定与动态视觉传感器对应的事件序号变化量，然后根据选取的图像传感器输出图像帧的帧号，以及动态视觉传感器输出事件数据的时序，获取动态视觉传感器输出的与该事件序号变化量对应的各个事件数据，即为动态视觉传感器在目标配准时间域内输出的目标事件序列。相应的，选取的图像传感器输出图像帧即为图像传感器在目标配准时间域输出的目标图像帧。
可选的，在本步骤中确定的目标配准时间域内图像传感器输出的图像帧的数量至少为一个。
根据目标事件序列以及目标图像帧，对动态视觉传感器和图像传感器进行空间配准时，可选的，根据动态视觉传感器在目标配准时间域内输出的目标事件序列构建待配准事件帧，根据图像传感器在目标配准时间域内输出的至少一个目标图像帧，并将构建的待配准事件帧与目标图像帧进行图像匹配，得到空间配准参数，以实现对动态视觉传感器和图像传感器进行空间配准。
E130、根据空间配准参数、所述事件序列和所述图像帧获取配准结果图像。
空间配准结果，指的是动态视觉传感器与图像传感器关于图像信息的配准详情，可选的，空间配准结果即为空间配准参数。
可选的，动态视觉传感器输出的数据包括事件数据以及根据事件数据构建的事件流特征帧中的至少一种。
在使用图像传感器输出的图像帧对动态视觉传感器输出的事件数据进行图像配准时，首先基于时间配准结果确定与待配准的事件数据对应的图像传感器输出的图像帧，然后基于空间配准结果，使用该图像帧对待配准的事件数据进行图像配准。重复上述流程，即可实现对动态视觉传感器输出的所有事件数据进行配准。
其中，基于时间配准结果可能确定一个图像帧对应于多个事件数据，进而可以使用这个图像帧对多个事件数据分别进行图像配准。
在使用图像传感器输出的图像帧对根据动态视觉传感器输出的事件数据构建的事件流特征帧进行图像配准时，首先基于时间累积，将目标时间段内(例如为与 图像传感器输出一个图像帧对应的时间段内)的多个事件数据构建成事件流特征帧，然后基于时间配准结果获取与该目标时间段对应的图像传感器输出的图像帧，最后基于空间配准结果，使用图像帧对事件流特征帧进行图像配准。重复上述流程，即可实现对动态视觉传感器各个目标时间段内的由多个事件数据构建成的事件流特征帧进行配准。
本公开实施例提供的技术方案，针对动态视觉传感器和图像传感器，首先进行时间配准，然后基于时间配准结果进行空间配准，进而可以基于动态视觉传感器和图像传感器之间的时间配准结果和空间配准结果，使用图像传感器输出的图像帧对由动态视觉传感器输出的数据进行图像配准，以此实现了动态视觉传感器和图像传感器之间的图像配准，配准后动态视觉传感器图像的物体感知能力更强。
实施例二
图18是本公开实施例二提供的一种图像配准方法的流程图。本实施例在上述实施例的基础上进行具体化，其中，根据所述目标事件数据以及所述目标图像帧，对所述动态视觉传感器和所述图像传感器进行空间配准，可以包括：
根据所述目标事件数据，构建至少一个目标图像帧对应的事件流特征帧；确定所述至少一个目标图像帧对应的特征描述子图像帧；对所述至少一个目标图像帧对应的事件流特征帧与特征描述子图像帧进行基于特征的图像配准，得到动态视觉传感器和图像传感器之间的空间配准参数，并经空间配准参数作为空间配准结果。
如图18所示，本实施例提供的图像配准方法，包括：
E210、对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
E220、根据时间配准参数，获取动态视觉传感器在目标配准时间域内输出的目标事件序列，以及图像传感器在所述目标配准时间域内输出的至少一个目标图像帧。
E230、根据目标事件序列，构建至少一个目标图像帧对应的待配准事件帧。
动态视觉传感器在目标配准时间域内输出的目标事件序列是多个事件数据的组合，也即在目标配准时间域内动态视觉传感器输出的事件序列。基于时间累积，将在某一个时间段范围内的事件数据收集在一起，并构建成图像帧，称之为事件流特征帧，或待配准事件帧。可选的，将与图像传感器输出一个图像帧对应的时间范围内的事件数据收集在一起，构建待配准事件帧。
当目标配准时间域内图像传感器输出的目标图像帧的数量为多个时，根据动态视觉传感器在目标配准时间域内输出的目标事件数据构建的待配准事件帧也为多个。
E240、根据至少一个目标图像帧获取特征描述子图像帧。
根据在目标配准时间域内图像传感器输出的至少一个目标图像帧，通过预设方法得到与至少一个目标图像帧对应的特征描述子图像帧。
获取在目标配准时间域内图像传感器输出的目标图像帧，对该目标图像帧进行时序特征特提取，生成与该目标图像帧对应的特征描述子图像帧。
可选的，对图像传感器按照帧采集的图像提取边缘轮廓，生成对应的特征描述子图像帧。其中，可以采用帧差法或者使用滤波器滤波的方法来提取图像边缘轮廓，本实施例对此不作具体限定。
E250、对待配准事件帧与特征描述子图像帧进行图像匹配，得到动态视觉传感器和图像传感器之间的空间配准参数，并将空间配准参数作为空间配准结果。
通过图像传感器输出的目标图像帧确定的图像边缘轮廓和动态视觉传感器采集的光强变化帧数据具有类似性，也即特征描述子图像帧与构建的事件流特征帧具 有类似性，因此，可以通过基于特征的图像配准方法确定两个图像帧的图像配准参数，作为动态视觉传感器和图像传感器之间的空间配准参数。
其中，将特征描述子图像帧作为基准图像，事件流特征帧作为待配准图像。如图19所示，基于特征的图像配准方法的核心步骤为：
1)特征检测，对基准图像和待配准图像进行关键点检测及图像特征提取，如图像边缘、轮廓等；
2)特征匹配，使用特征描述符、相似性度量等建立基准图像和待配准图像之间的相关性；
3)模型参数估计，可以基于配准时所用到的空间变换模型、配准的相似性测度准则以及空间变换矩阵，确定图像配准参数；
4)图像转换配准，基于确定的图像配准参数，使用基准图像对待配准图像进行图像配准。
值得指出的是，在目标配准时间域内图像传感器输出的图像帧的数量为多个时，可以将在目标配准时间域内获取的事件数据按照图像传感器输出图像帧的数量分为多个事件序列，并基于其中一个事件序列构建待配准事件帧。进而，使用与该事件序列对应的一个图像帧的特征描述子图像帧对该待配准事件帧进行图像配准，以确定动态视觉传感器和图像传感器之间的空间配准参数。
E260、根据空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
在一些实施例中，基于空间配准结果以及时间配准结果，使用图像传感器输出的图像帧对动态视觉传感器输出的事件数据进行图像配准，可以包括：
基于时间配准结果，确定图像传感器输出的图像帧，以及与所述图像帧对应的动态视觉传感器采集到的至少一个事件数据；基于空间配准参数，使用图像帧对至少一个事件数据进行图像配准。
其中，动态视觉传感器输出的数据包括事件数据以及根据事件数据构建的待配准事件帧中的至少一种。
首先，基于时间配准结果，确定图像传感器输出的图像帧，以及与图像帧对应的动态视觉传感器输出的事件数据。其中，图像传感器输出的图像帧与动态视觉传感器输出的事件数据是属于同一个时间域。然后，基于空间配准参数，利用目标图像帧对动态视觉传感器输出的事件数据进行图像配准。
其中，图像传感器输出图像帧的数量可以是一个或多个，当数量为多个时，依次使用每个图像帧对动态视觉传感器输出的相应事件数据进行配准即可，直至处理完图像传感器输出的所有图像帧。
在选取一个图像帧之后，首先，根据时间配准结果(例如为时间配准参数)确定与该图像帧对应的动态视觉传感器采集到的多个事件数据；其次，基于动态视觉传感器与图像传感器之间的空间配准参数，使用该图像帧对这些事件数据进行图像配准，或者，使用该图像帧对由这些事件数据构建的待配准事件帧进行图像配准。
本实施例未尽详细解释之处请参见前述实施例，在此不再赘述。
上述技术方案，实现了动态视觉传感器和图像传感器之间的图像配准，配准后动态视觉传感器图像的物体感知能力更强；同时，将基于特征的图像配准方法应用于事件流数据上，提供了一种处理事件流数据的实现方式，解决了现有大多图像处理方法及图像检测方法无法直接用于处理事件流的问题。
实施例三
图20是本公开实施例三提供的一种图像配准方法的流程图。本实施例提供了一种可选的实施方式，其中，图像传感器可以为COMS图像传感器。
如图20所示，本实施例提供的图像配准方法，包括：
E410、响应于至少两次拍摄场景变化，分别获取与每次拍摄场景变化对应的动态视觉传感器输出的一个事件序列的事件序号，以及与每次拍摄场景变化对应的CMOS图像传感器输出的一组图像帧的帧号，并分别确定每个事件序列的事件序号均值。
值得指出的是，动态视觉传感器和CMOS图像传感器的使用设备、场景配置与实际使用场景中保持一致。
E420、根据各个事件序列的事件序号均值以及各组图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将各个对应比例的均值作为动态视觉传感器和CMOS图像传感器之间的时间配准参数。
取每个事件序列的事件序号均值作为动态视觉传感器的时间标记，每组图像帧中最后一个图像帧的帧号作为CMOS图像传感器的时间标记。分别计算连续两个事件序列的事件序号均值的变化量，以及连续两组图像帧中最后一个图像帧的帧号的变化量，取二者变化量的比值，反复多次，直至变化量的比值处于设定变化区间，取这些比值的均值，作为动态视觉传感器和CMOS图像传感器之间的时间配准参数。
E430、基于时间配准参数，获取动态视觉传感器在目标配准时间域内输出的目标事件序列，以及CMOS图像传感器在目标配准时间域内输出的目标图像帧。
在确定动态视觉传感器和CMOS图像传感器之间的时间配准参数之后，基于时间配准参数进行动态视觉传感器和CMOS图像传感器之间空间配准。
值得指出的是，动态视觉传感器和CMOS图像传感器的使用设备、场景配置与实际使用场景中保持一致，也即与E410时间配准场景中保持一致。
E440、根据在目标配准时间域内动态视觉传感器输出的多个事件数据，构建事件流特征帧(待配准事件帧)。
E450、根据在目标配准时间域内CMOS图像传感器输出的图像帧，通过预设方法得到与图像帧对应的特征描述子图像帧。
可选的，通过帧差法或者滤波器滤波法得到特征描述子图像帧。
E460、对事件流特征帧与特征描述子图像帧进行基于特征的图像配准，得到动态视觉传感器和CMOS图像传感器之间的空间配准参数。
通过基于特征的图像配准方法，确定事件流特征帧与特征描述子图像帧之间的空间配准参数，作为动态视觉传感器和CMOS图像传感器之间的空间配准参数。
E470、基于空间配准结果以及时间配准结果，使用CMOS图像传感器输出的图像帧对根据动态视觉传感器输出的事件数据构建的事件流特征帧进行图像配准。
本实施例未尽详细解释之处请参见前述实施例，在此不再赘述。
在上述技术方案中，通过CMOS图像传感器输出的图像帧对动态视觉传感器进行图像配准，提升了动态视觉传感器的物体感知能力，例如颜色、边缘等，两种不同模式的传感器结合实现了高静态空间分辨率-高动态时间分辨率的双高感知，同时也减小了CMOS图像传感器对数据存储、数据算力和传输带宽的要求。
实施例四
图21是本公开实施例四提供的一种时间配准方法的流程图，可适用于对动态视觉传感器与传统图像传感器进行时间配准的情况，该方法可以由本公开实施例提供的时间配准装置来执行，该装置可采用软件和/或硬件的方式实现，并一般可集成在计算机设备中，例如可以是与动态视觉传感器以及传统图像传感器建立连接的计算机设备，该计算机设备能够接收并处理接收动态视觉传感器以及传统图像传感器的采集数据。
如图21所示，本实施例提供的图像配准方法，包括：
E510、响应于至少两次拍摄场景变化，分别获取与每次拍摄场景变化对应的动态视觉传感器输出的一个事件序列的事件序号，以及与每次拍摄场景变化对应的图像传感器输出的一组图像帧的帧号。
E520、根据各个事件序列的事件序号以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数。
作为本实施例一种可选的实施方式，E520可以包括：
分别确定每个事件序列的事件序号均值；根据各个事件序列的事件序号均值以及各组图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将各个对应比例的均值作为时间配准参数。
本实施例未尽详细解释之处请参见前述实施例，在此不再赘述。
本实施例提供的技术方案实现了不同模式的传感器之间的时间配准，适用于需要对动态视觉传感器与传统图像传感器进行时间配准的应用场景。
实施例五
图22是本公开实施例五提供的一种图像配准方法的流程图，可适用于对如何结合动态视觉传感器与传统图像传感器以实现运动目标平滑跟踪的情况，该方法可以由本公开实施例提供的图像配准装置来执行，该装置可采用软件和/或硬件的方式实现，并一般可集成在计算机设备中，例如可以是与动态视觉传感器以及传统图像传感器建立连接的计算机设备，该计算机设备能够接收并处理接收动态视觉传感器以及传统图像传感器的采集数据。
如图22所示，本实施例提供的图像配准方法，包括：
E610、对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
动态视觉传感器和图像传感器，分别指的是需要进行图像配准的一个动态视觉传感器与一个图像传感器，且二者的拍摄场景是相同的。其中，图像传感器采集图像信息后输出的是图像帧，可选的，此类图像传感器为COMS图像传感器。
时间配准，指的是将动态视觉传感器的输出信息和图像传感器的输出信息以图像采集时间为维度进行配准。可选的，将针对同一拍摄场景采集图像信息时图像传感器输出的图像帧与动态视觉传感器输出的事件数据组成的事件序列在时序方面进行配准，以实现对动态视觉传感器和图像传感器进行时间配准。
在一些实施例例中，可以将动态视觉传感器输出的事件序列(由多个事件数据组成)的事件序号信息与图像传感器输出的图像帧的帧号进行配准，以实现对动态视觉传感器和图像传感器进行时间配准。其中，事件序号即为动态视觉传感器输出事件数据的时间戳的序号。
可选的，通过确定动态视觉传感器和图像传感器之间的时间配准参数来实现对动态视觉传感器和图像传感器进行时间配准。其中，时间配准参数可以是以图像采集时间为统计维度时动态视觉传感器和图像传感器的输出信息数量之间的对应比例关系。
在些实施例中，E610可以包括：响应于至少两次拍摄场景变化，分别获取与每次拍摄场景变化对应的动态视觉传感器输出的一个事件序列的事件序号，以及与每次拍摄场景变化对应的图像传感器输出的一组图像帧的帧号；根据各个事件序列的事件序号以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，并将时间配准参数作为时间配准结果。
其中，拍摄场景变化指的是动态视觉传感器或者图像传感器采集的图像信息发生变化，例如可以是拍摄场景中存在移动的目标物体，例如还可以是拍摄场景中光 强(或者光源等)发生变化，等等。
在每次拍摄场景发生变化时，确定动态视觉传感器在此次变化过程中输出的事件序列，并获取该事件序列中与每个事件数据对应的事件序号，同时确定图像传感器在此次变化过程中输出的一组图像帧(其中，一组图像帧中可以包括一个或多个图像帧)，并获取其中每个图像帧的帧号。
进而，根据在多次拍摄场景发生变化的过程中获取到的各个事件序列的事件序号信息以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，也即确定以图像采集时间为维度动态视觉传感器输出的事件数量与图像传感器的输出的图像帧数量之间的对应比例关系，如在同一个拍摄场景中动态视觉传感器输出的m个事件数据对应于图像传感器输出的一个图像帧。
可选的，对在多次拍摄场景发生变化的过程中获取到的各个事件序列的事件序号信息以及各组图像帧的帧号进行统计分析，根据统计分析结果确定动态视觉传感器和图像传感器之间的时间配准参数。
在一些实施例中，根据各个事件序列的事件序号以及各组图像帧的帧号，确定动态视觉传感器和图像传感器之间的时间配准参数，可以包括：
分别确定每个事件序列的事件序号均值；根据各个事件序列的事件序号均值以及各组图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将各个对应比例的均值作为时间配准参数。
假设，拍摄场景变化的次数为n，在第一次拍摄场景变化中获取到的一个事件序列中各个事件的事件序号分别为t10、t11、…、t1m，取t10、t11、…、t1m的均值t1avg作为该事件序列的事件序号均值，类似的，计算得到的各个事件序列的事件序号均值t1avg、t2avg、…、tnavg。
事件序号均值的变化量，指的是与连续两次拍摄场景变化分别对应的两个连续事件序列的事件序号均值的差值，即为tnavg-t(n-1)avg；帧号的变化量，指的是与连续两次拍摄场景变化分别对应的两组视频帧中最后一个图像帧的帧号的差值，假设n组视频帧中最后一个图像帧的帧号依次为n1、n2、…、nn，则连续两组图像帧的帧号的变化量为nn-n(n-1)。
值得指出的是，各组图像帧的帧号是连续的，例如在第一次拍摄场景变化中获取到的一组图像帧中各个图像帧的帧号为1、2、…、n1，则在第二次拍摄场景变化中获取到的一组图像帧的帧号从n1+1开始计数。
事件序号均值的变化量与帧号的变化量的比例，指的是与连续两次拍摄场景变化对应的事件序号均值的变化量和帧号的变化量的比值，即(tnavg-t(n-1)avg)/(nn-n(n-1))。
根据拍摄场景变化次数n，可以得到(n-1)个事件序号均值的变化量与帧号的变化量的对应比例，计算这(n-1)个对应比例的均值，并将该均值作为时间配准参数。
在一种示例性的实施方式中，采用一个闪烁的光源来实现拍摄场景的变化，该光源以脉冲形式发光，分别使用动态视觉传感器和图像传感器来录制此场景。当动态视觉传感器检测到闪烁光源时，会在光强变化瞬间(增强或减弱)输出事件序列，取事件序列中各个事件的事件序号的均值作为动态视觉传感器的时间标记。同时，图像传感器进行录像并输出各个图像帧的帧号，将最后一个图像帧的帧号与动态视觉传感器的时间标记进行关联，即可得到一个动态视觉传感器的时间标记与图像传感器的图像帧号的对应关系。随着光源再次闪烁，记录下一个动态视觉传感器的时间标记与图像传感器的图像帧号的对应关系，从而可以得到时间标记的变化量(也即事件序号均值的变化量)与图像帧号的变化量的关系，也即时间标记的变化量与 图像帧号的变化量的相对比例。类似的，可以得到多个时间标记的变化量与图像帧号的变化量的相对比例，取这些相对比例的平均值，作为动态视觉传感器与图像传感器之间的时间配准参数。
在另一种可选的实施方式中，还可以分别计算事件序号均值变化量的均值，以及帧号变化量的均值，将事件序号均值变化量的均值与帧号变化量的均值的比值作为时间配准参数。
在一种可选的示例中，时间配准参数为事件序号变化量与帧号变化量的比例，其中，帧号变化量为1，也即时间配准参数指示图像传感器输出一个图像帧对应的动态视觉传感器的事件序号变化量。
E620、根据时间配准参数，对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系。
时间配准结果，指的是动态视觉传感器与图像传感器关于图像采集时间的配准详情，可选的，时间配准结果即为时间配准参数。
空间配准，指的是将动态视觉传感器和图像传感器的输出信息以图像信息为维度进行配准。可选的，将动态视觉传感器和图像传感器在目标配准时间域内输出的图像信息在像素方面进行配准，以实现对动态视觉传感器和图像传感器进行空间配准。
基于时间配准结果，获取时间配准后的动态视觉传感器的输出数据以及图像传感器的输出数据，对这些输出数据进行空间配准即可实现对动态视觉传感器和图像传感器的空间配准。
作为一种可选的实施方式，根据时间配准结果，对动态视觉传感器和图像传感器进行空间配准，可以包括：
根据时间配准参数，获取动态视觉传感器在目标配准时间域内输出的目标事件序列，以及图像传感器在目标配准时间域内输出的至少一个目标图像帧；根据目标事件序列以及至少一个目标图像帧，对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数。
目标配准时间域，指的是与动态视觉传感器以及图像传感器对应的一个相同时间域，在目标配准时间域内动态视觉传感器和图像传感器采集的图像源是相同的。也即，动态视觉传感器在目标配准时间域内输出的事件数据(也可称之为事件流数据)，与图像传感器在目标配准时间域内输出的图像帧，在时间维度是配准的。
其中，针对动态视觉传感器而言，目标配准时间域可以根据输出事件数据的事件序号来确定；针对图像传感器而言，目标配准时间域可以根据输出图像帧的数量来确定。
例如，可以首先根据选取的图像传感器输出图像帧的帧号变化量以及时间配准参数，确定与动态视觉传感器对应的事件序号变化量，然后根据选取的图像传感器输出图像帧的帧号，以及动态视觉传感器输出事件数据的时序，获取动态视觉传感器输出的与该事件序号变化量对应的各个事件数据，即为动态视觉传感器在目标配准时间域内输出的目标事件序列。相应的，选取的图像传感器输出图像帧即为图像传感器在目标配准时间域输出的目标图像帧。
可选的，在本步骤中确定的目标配准时间域内图像传感器输出的图像帧的数量至少为一个。
根据目标事件序列以及目标图像帧，对动态视觉传感器和图像传感器进行空间配准时，可选的，根据动态视觉传感器在目标配准时间域内输出的目标事件序列构建待配准事件帧，根据图像传感器在目标配准时间域内输出的至少一个目标图像帧， 并将构建的待配准事件帧与目标图像帧进行图像匹配，得到空间配准参数，以实现对动态视觉传感器和图像传感器进行空间配准。
E630、基于空间配准参数以及时间配准参数、事件序列和图像帧获取配准结果图像。即使用动态视觉传感器输出的事件数据对待配准图像帧进行图像配准，其中，待配准图像帧是根据图像传感器获取的图像帧中提取的至少一个运动目标生成的。
运动目标，指的是需要进行跟踪的运动目标。待配准图像帧是根据图像传感器输出的图像帧中提取的至少一个运动目标生成的，需要进行图像配准的图像帧。其中，待配准图像帧中可以包括一个或多个运动目标，运动目标的数量可以根据实际跟踪需求确定。
可选的，待配准图像帧是通过对图像传感器输出的图像帧进行图像处理后生成的，例如可以将图像传感器输出的图像帧中的背景区域进行抠除，使得待配准图像帧中仅包括运动目标以及运动目标附近的像素数据，不包括图像传感器输出的图像帧中的背景像素数据。待配准图像帧还可以是对图像传感器输出的图像帧进行截取处理，例如，对图像传感器输出的图像帧进行运动目标识别，并预测运动目标的运动趋势，根据运动目标以及运动趋势，对图像传感器输出的图像帧进行截取处理，得到的包括运动目标以及运动目标附近区域的待配准图像帧。
在一种可选的示例中，待配准图像帧中仅包括运动目标的轮廓数据以及运动目标轮廓附近的像素数据。
可选的，动态视觉传感器输出的数据包括事件数据以及根据事件数据构建的事件流特征帧中的至少一种。
在使用动态视觉传感器输出的至少一个事件数据对一个待配准图像帧进行图像配准时，首先基于时间配准结果获取与待配准图像帧对应的原始视频帧所对应的至少一个事件数据，然后基于空间配准结果，使用至少一个事件数据对待配准图像帧进行图像配准。
可选的，首先基于位置坐标对这多个事件数据进行筛选，得到与待配准图像帧对应的部分事件数据，也即与待配准图像帧中的各个运动目标对应的部分事件数据，并根据这部分事件数据对待配准图像帧进行图像配准。
在使用根据动态视觉传感器输出的事件数据构建的事件流特征帧对一个待配准图像帧进行图像配准时，可以基于时间配准结果获取与待配准图像帧对应的原始视频帧所对应的多个事件数据，并将多个事件数据构建成事件流特征帧，基于空间配准结果，使用事件流特征帧对待配准图像帧进行图像配准。
可选的，在根据时间配准结果(例如为时间配准参数)确定与待配准图像帧对应的动态视觉传感器采集到的多个事件数据之后，在这多个事件数据中筛选出与各个运动目标位置匹配的部分事件数据，并根据这部分事件数据构建事件流特征帧，基于空间配准参数利用该事件流特征帧对待配准图像帧进行图像配准。
基于图像传感器输出的图像帧实时生成对应的待配准图像帧，同时基于空间配准结果以及时间配准结果，使用动态视觉传感器输出的事件数据或者根据多个事件数据构建的事件流特征帧对各个对应的待配准图像帧进行图像配准，即可得到与待跟踪的运动目标对应的多个配准图像帧。
进一步的，还可以根据使用动态视觉传感器输出的数据对待配准图像帧进行图像配准后得到的多个配准图像帧，确定至少一个运动目标的运动轨迹信息。
在得到与待跟踪的运动目标对应的多个配准图像帧之后，通过对每一个配准图像帧中的运动目标进行检测即可实现对该运动目标的跟踪。通过多个配准图像帧可以得到待跟踪的运动目标的平滑运动轨迹，也即能够得到各个时刻待跟踪的运动目标的精准数据，从而获取到待跟踪的运动目标的实时在线精准数据。
本公开实施例提供的技术方案，对图像传感器和动态视觉传感器进行时间配准，确定时间配准结果，并根据时间配准结果对图像传感器和动态视觉传感器进行空间配准，确定空间配准结果，进而可以基于时间配准结果和空间配准结果，使用动态视觉传感器输出的数据对根据图像传感器输出的图像帧提取的运动目标生成的待配准图像帧进行图像配准。相对于现有的运动目标跟踪方法而言，上述技术方案中根据动态视觉传感器输出的数据对根据图像传感器输出的图像帧提取的运动目标生成的待配准图像帧进行图像配准，生成某一时刻运动目标的精准数据，以此提高了进行运动目标跟踪时所采用图像的精准度，进而可以在低存储低传输要求下基于配准后的图像进行运动目标跟踪识别，减小了对数据算力、传输带宽、数据存储的要求，也减少了使用目标检测跟踪目标而造成算力浪费的问题，还能够实现对运动目标运动轨迹的平滑跟踪，得到待跟踪的运动目标的实时在线精准数据。
实施例六
图23是本公开实施例六提供的一种图像配准方法的流程图。本实施例在上述实施例的基础上进行具体化，其中，根据所述目标图像帧以及所述目标事件数据对所述图像传感器和动态视觉传感器进行空间配准，确定所述空间配准结果，可以包括：
根据所述目标事件数据，构建至少一个目标图像帧对应的事件流特征帧；
确定所述至少一个目标图像帧对应的特征描述子图像帧；
对所述至少一个目标图像帧对应的事件流特征帧与所述特征描述子图像帧进行基于特征的图像配准，得到所述动态视觉传感器和所述图像传感器之间的空间配准参数，并将所述空间配准参数作为所述空间配准结果。
如图23所示，本实施例提供的图像配准方法，可以包括：
E710、对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
E720、根据所述时间配准结果，确定目标配准时间域内所述图像传感器输出的目标图像帧和所述动态视觉传感器输出的目标事件数据。
E730、根据所述时间配准参数，获取目标配准时间域内的目标事件序列。
动态视觉传感器在目标配准时间域内输出的目标事件序列是多个事件数据的组合，也即在目标配准时间域内动态视觉传感器输出的事件流数据。基于时间累积，将在某一个时间段范围内的事件数据收集在一起，并构建成图像帧，称之为事件流特征帧。可选的，将与图像传感器输出一个图像帧对应的时间范围内的事件数据收集在一起，构建事件流特征帧。
当目标配准时间域内图像传感器输出的目标图像帧的数量为多个时，根据动态视觉传感器在目标配准时间域内输出的目标事件数据构建的事件流特征帧也为多个。
E740、根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征。
根据在目标配准时间域内图像传感器输出的至少一个目标图像帧，通过预设方法得到与至少一个目标图像帧对应的待配准图像帧。
获取在目标配准时间域内图像传感器输出的目标图像帧，对该目标图像帧进行时序特征特提取，生成与该目标图像帧对应的待配准图像帧。
可选的，对图像传感器按照帧采集的图像提取边缘轮廓，生成对应的待配准图像帧。其中，可以采用帧差法或者使用滤波器滤波的方法来提取图像边缘轮廓，本实施例对此不作具体限定。
E750、根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
通过图像传感器输出的目标图像帧确定的图像边缘轮廓和动态视觉传感器采集的光强变化帧数据具有类似性，也即待配准图像帧与构建的事件流特征帧具有类似性，因此，可以通过基于特征的图像配准方法确定两个图像帧的图像配准参数，作为动态视觉传感器和图像传感器之间的空间配准参数。
其中，将事件流特征帧作为基准图像，特征描述子图像帧作为待配准图像。如图19所示，基于特征的图像配准方法的核心步骤为：
1)特征检测，对基准图像和待配准图像进行关键点检测及图像特征提取，如图像边缘、轮廓等；
2)特征匹配，使用特征描述符、相似性度量等建立基准图像和待配准图像之间的相关性；
3)模型参数估计，可以基于配准时所用到的空间变换模型、配准的相似性测度准则以及空间变换矩阵，确定图像配准参数；
4)图像转换配准，基于确定的图像配准参数，使用基准图像对待配准图像进行图像配准。
值得指出的是，在目标配准时间域内图像传感器输出的图像帧的数量为多个时，可以将在目标配准时间域内获取的事件数据按照图像传感器输出图像帧的数量分为多个事件序列，并基于其中一个事件序列构建事件流特征帧。进而，使用与该事件序列对应的一个图像帧的待配准图像帧对该事件流特征帧进行图像配准，以确定图像传感器和动态视觉传感器之间的空间配准参数。
E760、根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
在一种可选的实施方式中，基于空间配准结果以及时间配准结果，使用动态视觉传感器输出的数据对待配准图像帧进行图像配准，可以包括：基于图像采集空间配准结果，确定动态视觉传感器输出的至少一个数据，以及与至少一个数据对应的图像传感器输出的图像帧；基于空间配准参数，使用至少一个数据对图像帧进行图像配准。
其中，动态视觉传感器输出的数据包括事件数据以及根据事件数据构建的事件流特征帧中的至少一种。
在对待跟踪的运动目标进行实时跟踪时，针对根据图像传感器输出的图像帧中的运动目标对应的一个待配准图像帧，基于图像传感器和动态视觉传感器的时间配准结果，确定与该待配准图像帧对应的动态视觉传感器采集到的多个事件数据，基于图像传感器和动态视觉传感器的空间配准参数，使用这些事件数据或者由这些事件数据构建的事件流特征帧对待配准图像帧进行图像配准。
在一种可选的实施方式中，本实施例提供的方法在对待配准图像帧进行图像配准前，还包括：
构建与图像传感器输出的图像帧大小对应的初始权值矩阵；根据至少一个运动目标在图像帧中的当前分布区域以及预测分布区域，对初始权值矩阵进行赋值，得到目标赋值权值矩阵；将目标赋值权值矩阵与所述图像帧进行点乘操作，得到待配准图像帧。
其中，可以针对图像传感器输出的每个图像帧单独构建一个大小对应的初始权值矩阵，且初始权值矩阵中所有区域权值置零；也可以针对图像传感器在输出的各个图像帧构建一个大小对应且通用的初始权值矩阵，且初始权值矩阵中所有区域权值置零。
可选的，根据至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对初始权值矩阵进行赋值，得到目标赋值权值矩阵，包括：
对初始权值矩阵中与至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；对初始权值矩阵中与至少一个运动目标在图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；结合第一赋值权值矩阵和第二赋值权值矩阵，生成目标赋值权值矩阵。
以针对一个目标图像帧生成待配准图像帧为例进行解释说明，通过该方法可以生成与每个目标图像帧对应的待配准图像帧。
获取目标图像帧中的待跟踪的运动目标，根据待跟踪的运动目标在目标图像帧中的当前分布区域对初始权值矩阵中匹配区域(即第一区域)进行赋值，如赋值为1，得到第一赋值权值矩阵。例如，可以使用帧差法获取目标图像帧中的待跟踪的运动目标，再将基于差分法得到的与待跟踪的运动目标对应的区域权值置1，本实施对此不作具体限定。
预测待跟踪运动目标在目标图像帧之后的下一个目标图像帧中的分布区域(即预测分布区域)，根据待跟踪运动目标在目标图像帧中的预测分布区域对初始权值矩阵中匹配区域(即第二区域)进行赋值，如赋值为1，得到第二赋值权值矩阵。
在一示例中，首先对目标图像帧中的各个运动目标进行目标分解，通过运动空间连续性特征，提取多个运动闭合目标区域，其次通过视频图像前后帧的空间位置得到每个运动目标的运动矢量场。针对一个待跟踪运动目标，通过该待跟踪运动目标的运动矢量场，并结合该待跟踪运动目标的历史运动轨迹(如方向及速度等)，加权预测该待跟踪的运动目标中心点在下一个目标图像帧中的空间位置，结合其网格划分(不同大小的运动目标对应于不同大小的网格划分)获得到该待跟踪的运动目标在下个目标图像帧中的预测分布区域。
将第一赋值权值矩阵和第二赋值权值矩阵进行“或”运算，得到目标赋值权值矩阵。
可选的，预测待跟踪的运动目标在目标图像帧之后的下一个目标图像帧中的分布区域(即预测分布区域)之后，根据待跟踪的运动目标在目标图像帧中的预测分布区域继续对第一赋值权值矩阵中匹配区域(即第二区域)进行赋值，如赋值为1，可直接得到上述目标赋值权值矩阵。
将目标赋值权值矩阵与目标图像帧进行点乘操作，得到针对待跟踪的运动目标的配准图像帧。此时，配准图像帧中包括各个待跟踪的运动目标及其附近像素数据，或者是各个待跟踪的运动目标轮廓数据及轮廓附近像素数据，也即配准图像帧中提取了目标图像帧中待跟踪的运动目标及其细节特征。
本实施例未尽详细解释之处请参见前述实施例，在此不再赘述。
上述技术方案可以在低存储低传输要求下进行运动目标的跟踪识别，实现了对运动目标运动轨迹的平滑跟踪；同时，将基于特征的图像配准方法应用于事件流数据上，提供了一种处理事件流数据的实现方式，解决了现有大多图像处理方法及图像检测方法无法直接用于处理事件流的问题。
实施例七
图24是本公开实施例七提供的一种图像配准方法的流程图。本实施例提供了一种可选的实施方式，其中，将图像传感器为COMS图像传感器。
如图24所示，本实施例提供的图像配准方法，可以包括：
E810、对CMOS图像传感器和动态视觉传感器进行时间配准，确定CMOS图像传感器和动态视觉传感器之间的时间配准参数。
值得指出的是，CMOS图像传感器和动态视觉传感器的使用设备、场景配置以 及数据记录配置应与实际使用场景中保持一致。
E820、基于时间配准参数，获取动态视觉传感器在目标配准时间域内输出的目标事件数据，以及CMOS图像传感器在目标配准时间域内输出的目标图像帧。
在确定动态视觉传感器和CMOS图像传感器之间的时间配准参数之后，基于时间配准参数进行CMOS图像传感器和动态视觉传感器之间空间配准。
值得指出的是，CMOS图像传感器和动态视觉传感器的使用设备、场景配置以及数据记录配置应与实际使用场景中保持一致，也即与E810时间配准场景中保持一致。
E830、根据在目标配准时间域内动态视觉传感器输出的事件数据，构建事件流特征帧。
E840、根据在目标配准时间域内CMOS图像传感器输出的图像帧，通过预设方法得到与图像帧对应的特征描述子图像帧。
可选的，通过帧差法或者滤波器滤波法得到特征描述子图像帧。
E850、对特征描述子图像帧与事件流特征帧进行基于特征的图像配准，得到CMOS图像传感器和动态视觉传感器之间的空间配准参数。
E860、基于时间配准参数，确定同一个时间域内动态视觉传感器输出的事件数据，以及CMOS图像传感器输出的图像帧。
E870、构建与CMOS图像传感器输出的图像帧大小对应的初始权值矩阵。
E880、依次根据至少一个待跟踪的运动目标在CMOS图像传感器在各个图像帧中的当前分布区域以及预测分布区域，对初始权值矩阵进行赋值，得到各个目标赋值权值矩阵，并与匹配的图像帧进行点乘操作，得到各个待配准图像帧。
在此步骤中，得到的待配准图像帧是与CMOS图像传感器在目标时间域内输出目标图像帧一一对应的。
E890、依次获取一个待配准图像帧，以及与待配准图像帧对应的动态视觉传感器采集到的事件数据。
E8100、根据事件数据构建事件流特征帧，并基于空间配准参数，利用事件流特征帧对待配准图像帧进行图像配准，生成相应的配准图像帧，返回执行E890。
本实施例未尽详细解释之处请参见前述实施例，在此不再赘述。
在上述技术方案中，在配准的一定时间范围内，采用高分辨CMOS图像传感器跟踪提取运动目标及局部细节特征(且只考虑运动目标附近的像素数据)，同时采用动态视觉传感器补充运动目标的运动细节及局部行为微动特征信息，由此形成了对待跟踪的运动目标的实时在线精准地特征信息描述。
实施例八
图25是本公开实施例八提供的一种图像配准装置的结构示意图，可适用于对动态视觉传感器与传统图像传感器进行图像配准的情况，该装置可采用软件和/或硬件的方式实现，并一般可集成在计算机设备中，例如可以是与动态视觉传感器以及传统图像传感器建立连接的计算机设备，该计算机设备能够接收并处理接收动态视觉传感器以及传统图像传感器的采集数据。
如图25所示，该图像配准装置包括：传感器间时间配准模块610、传感器间空间配准模块620和图像配准模块630。其中，
传感器间时间配准模块610，设置为对动态视觉传感器和图像传感器进行时间配准；
传感器间空间配准模块620，设置为根据时间配准结果，对所述动态视觉传感器和所述图像传感器进行空间配准；
图像配准模块630，设置为基于空间配准结果以及所述时间配准结果，使用所 述图像传感器输出的图像帧对所述动态视觉传感器输出的数据进行图像配准。
本公开实施例提供的技术方案，针对动态视觉传感器和图像传感器，首先进行时间配准，然后基于时间配准结果进行空间配准，进而可以基于动态视觉传感器和图像传感器之间的时间配准结果和空间配准结果，使用图像传感器输出的图像帧对由动态视觉传感器输出的事件数据构建的事件流特征帧进行图像配准，以此实现了动态视觉传感器和图像传感器之间的图像配准，配准后动态视觉传感器图像的物体感知能力更强。
可选的，传感器间空间配准模块620包括：
空间配准数据获取单元，设置为根据时间配准结果，获取所述动态视觉传感器在目标配准时间域内输出的目标事件数据，以及所述图像传感器在所述目标配准时间域内输出的目标图像帧；
传感器间空间配准单元，设置为根据所述目标事件数据以及所述目标图像帧，对所述动态视觉传感器和所述图像传感器进行空间配准。
可选的，传感器间时间配准模块610，包括：
传感器图像采集时序信息获取单元，设置为响应于至少两次拍摄场景变化，分别获取与每次所述拍摄场景变化对应的所述动态视觉传感器输出的一个事件序列的事件序号，以及与每次所述拍摄场景变化对应的所述图像传感器输出的一组图像帧的帧号；
传感器间时间配准单元，设置为根据各个所述事件序列的事件序号以及各组所述图像帧的帧号，确定所述动态视觉传感器和所述图像传感器之间的时间配准参数，并将所述时间配准参数作为所述时间配准结果。
进一步的，传感器间时间配准单元，设置为分别确定每个所述事件序列的事件序号均值；根据各个所述事件序列的事件序号均值以及各组所述图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将所述各个对应比例的均值作为所述时间配准参数。
可选的，传感器间空间配准单元，设置为根据所述目标事件数据，构建至少一个目标图像帧对应的事件流特征帧；确定所述至少一个目标图像帧对应的特征描述子图像帧；对所述至少一个目标图像帧对应的事件流特征帧与所述特征描述子图像帧进行基于特征的图像配准，得到所述动态视觉传感器和所述图像传感器之间的空间配准参数，并将所述空间配准参数作为所述空间配准结果。
进一步的，图像配准模块630，可以设置为基于所述时间配准结果，确定所述图像传感器输出的图像帧，以及与所述图像帧对应的所述动态视觉传感器采集到的至少一个数据；基于所述空间配准参数，使用所述图像帧对所述至少一个数据进行图像配准。
可选的，所述动态视觉传感器输出的数据包括事件数据以及根据事件数据构建的事件流特征帧中的至少一种。
上述图像配准装置可执行本公开任意实施例所提供的图像配准方法，具备执行的图像配准方法相应的功能模块和有益效果。
实施例九
图26是本公开实施例九提供的一种图像配准装置的结构示意图，可适用于对动态视觉传感器与传统图像传感器进行时间配准的情况，该装置可采用软件和/或硬件的方式实现，并一般可集成在计算机设备中，例如可以是与动态视觉传感器以及传统图像传感器建立连接的计算机设备，该计算机设备能够接收并处理接收动态视觉传感器以及传统图像传感器的采集数据。
如图26所示，该图像配准装置包括：传感器图像采集时序信息获取模块710 和传感器间时间配准模块720。其中，
传感器图像采集时序信息获取模块710，设置为响应于至少两次拍摄场景变化，分别获取与每次所述拍摄场景变化对应的动态视觉传感器输出的一个事件序列的事件序号，以及与每次所述拍摄场景变化对应的图像传感器输出的一组图像帧的帧号；
传感器间时间配准模块720，设置为根据各个所述事件序列的事件序号以及各组所述图像帧的帧号，确定所述动态视觉传感器和所述图像传感器之间的时间配准参数。
本实施例提供的技术方案实现了不同模式的传感器之间的时间配准，适用于需要进行时间配准的应用场景中。
进一步的，传感器间时间配准模块720，设置为分别确定每个所述事件序列的事件序号均值；根据各个所述事件序列的事件序号均值以及各组所述图像帧的帧号，分别确定事件序号均值的变化量与帧号的变化量的各个对应比例，并将所述各个对应比例的均值作为所述时间配准参数。
上述图像配准装置可执行本公开第一方面任意实施例所提供的图像配准方法，具备执行的第一方面的图像配准方法相应的功能模块和有益效果。
实施例十
图27是本公开实施例十提供的一种计算机设备的结构示意图。如图27所示，该计算机设备包括处理器810、存储器820、输入装置830和输出装置840；计算机设备中处理器810的数量可以是一个或多个，图27中以一个处理器810为例；计算机设备中的处理器810、存储器820、输入装置830和输出装置840可以通过总线或其他方式连接，图27中以通过总线连接为例。
存储器820作为一种计算机可读存储介质，可用于存储软件程序、计算机可执行程序以及模块，如本公开实施例中第一方面所述的图像配准方法对应的程序指令/模块(例如，图15所示的图像配准装置中的时间配准模块100)，又如本公开实施例中第二方面所述的图像配准方法对应的程序指令/模块(例如，图16所示的图像配准装置中的空间配准模块200和图像配准模块300)。处理器810通过运行存储在存储器820中的软件程序、指令以及模块，从而执行计算机设备的各种功能应用以及数据处理，即实现上述图像配准方法或者上述图像配准方法。
存储器820可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据计算机设备的使用所创建的数据等。此外，存储器820可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器820可进一步包括相对于处理器810远程设置的存储器，这些远程存储器可以通过网络连接至计算机设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置830可用于接收输入的数字或字符信息，以及产生与计算机设备的用户设置以及功能控制有关的键信号输入。输出装置840可包括显示屏等显示设备。
实施例十一
本公开实施例十一还提供一种存储有计算机程序的计算机可读存储介质，计算机程序在由计算机处理器执行时用于执行一种图像配准方法，包括：
对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系；
和/或
根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像。
当然，本公开实施例所提供的存储有计算机程序的计算机可读存储介质，其计算机程序不限于如上的方法操作，还可以执行本公开任意实施例所提供的图像配准方法中的相关操作。
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本公开可借助软件及必需的通用硬件来实现，当然也可以通过硬件实现，但很多情况下前者是更佳的实施方式。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，
ROM)、随机存取存储器(Random Access Memory，RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例的方法。
值得注意的是，上述图像配准装置以及图像配准装置的实施例中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本公开的保护范围。
注意，上述仅为本公开的较佳实施例及所运用技术原理。本领域技术人员会理解，本公开不限于这里所述的特定实施例，对本领域技术人员来说能够进行各种明显的变化、重新调整和替代而不会脱离本公开的保护范围。因此，虽然通过以上实施例对本公开进行了较为详细的说明，但是本公开不仅仅限于以上实施例，在不脱离本公开构思的情况下，还可以包括更多其他等效实施例，而本公开的范围由所附的权利要求范围决定。

Claims (30)

  1. 一种图像配准方法，包括：
    对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
  2. 根据权利要求1所述的图像配准方法，其中，对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数的步骤包括：
    获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
    获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
    根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
  3. 根据权利要求2所述的图像配准方法，其中，获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记的步骤包括：
    获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；
    计算多个事件数据的时间戳的平均值，作为所述时间标记。
  4. 根据权利要求2或3所述的图像配准方法，其中，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数的步骤包括：
    确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；
    计算多个所述相对比例的平均值作为所述时间配准参数。
  5. 一种图像配准方法，包括：
    根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
    根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；
    其中，所述时间配准参数为根据权利要求1至4中任意一项所述的图像配准方法获得的。
  6. 根据权利要求5所述的图像配准方法，其中，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
    根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
    根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
  7. 根据权利要求6所述的图像配准方法，其中，根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数的步骤包括：
    根据所述目标事件序列构建待配准事件帧；
    根据所述至少一个目标图像帧获取特征描述子图像帧；
    对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
  8. 根据权利要求7所述的图像配准方法，其中，根据所述目标事件序列构建待配准事件帧的步骤包括：
    对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
  9. 根据权利要求7或8所述的图像配准方法，其中，根据所述空间配准参数、 所述事件序列和所述图像帧获取配准结果图像的步骤包括：
    根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
  10. 根据权利要求5所述的图像配准方法，其中，根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数的步骤包括：
    根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
    根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
    根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
  11. 根据权利要求10所述的图像配准方法，其中，根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧的步骤包括：
    根据多个所述图像帧确定至少一个运动目标；
    获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
  12. 根据权利要求11所述的图像配准方法，其中，获取所述至少一个运动目标的像素数据，生成所述待配准图像帧的步骤包括：
    构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；
    根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；
    将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
  13. 根据权利要求12所述的图像配准方法，其中，根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵，包括：
    对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；
    对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；
    结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
  14. 根据权利要求10至13中任意一项所述的图像配准方法，其中，根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像的步骤包括：
    根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
  15. 一种图像配准装置，包括：
    时间配准模块，用于对动态视觉传感器和图像传感器进行时间配准，得到时间配准参数，所述时间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的时间关联关系。
  16. 根据权利要求15所述的图像配准装置，其中，所述时间配准模块包括：
    事件数据处理单元，用于获取所述动态视觉传感器响应于多个配准事件采集的事件序列的多个时间标记，每一个所述配准事件对应一个所述时间标记；
    图像帧处理单元，用于获取所述图像传感器响应于多个所述配准事件采集的多个图像帧的帧号；
    时间配准单元，根据多个所述时间标记和多个所述帧号，确定所述时间配准参数。
  17. 根据权利要求16所述的图像配准装置，其中，所述事件数据处理单元用于 获取对应同一个所述配准事件的事件序列中多个事件数据的时间戳；计算多个事件数据的时间戳的平均值，作为所述时间标记。
  18. 根据权利要求16或17所述的图像配准装置，其中，所述时间配准单元用于确定任意两个相邻的所述配准事件对应的时间标记的变化量和帧号的变化量的相对比例；计算多个所述相对比例的平均值作为所述时间配准参数。
  19. 一种图像配准装置，包括：
    空间配准模块，用于根据时间配准参数对动态视觉传感器和图像传感器进行空间配准，得到空间配准参数，所述空间配准参数表征所述动态视觉传感器采集的事件序列和所述图像传感器采集的图像帧的空间关联关系；
    图像配准模块，用于根据所述空间配准参数、所述事件序列和所述图像帧获取配准结果图像；
    其中，所述时间配准参数为根据权利要求1至4中任意一项所述的图像配准方法获得的。
  20. 根据权利要求19所述的图像配准装置，其中，所述空间配准模块包括：
    第一数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列和至少一个目标图像帧；
    第一空间配准单元，用于根据所述目标事件序列和所述至少一个目标图像帧进行空间配准，得到所述空间配准参数。
  21. 根据权利要求20所述的图像配准装置，其中，所述第一数据处理单元用于根据所述目标事件序列构建待配准事件帧；根据所述至少一个目标图像帧获取特征描述子图像帧；对所述待配准事件帧与所述特征描述子图像帧进行图像匹配，得到所述空间配准参数。
  22. 根据权利要求21所述的图像配准装置，其中，所述第一数据处理单元用于对所述目标事件序列中的多个事件数据进行时间累加，得到所述待配准事件帧。
  23. 根据权利要求21或22所述的图像配准装置，其中，所述图像配准模块用于根据所述空间配准参数和所述至少一个目标图像帧，确定所述待配准事件帧中运动目标的细节信息，得到所述配准结果图像。
  24. 根据权利要求19所述的图像配准装置，其中，所述空间配准模块包括：
    第二数据处理单元，用于根据所述时间配准参数，获取目标配准时间域内的目标事件序列；
    所述第二数据处理单元还用于根据所述时间配准参数，获取所述目标配准时间域内的待配准图像帧，所述待配准图像帧表征所述图像帧中的运动目标的细节特征；
    第二空间配准单元，用于根据所述目标事件序列和所述待配准图像帧进行空间配准，得到所述空间配准参数。
  25. 根据权利要求24所述的图像配准装置，其中，所述第二数据处理单元用于根据多个所述图像帧确定至少一个运动目标；获取所述至少一个运动目标的像素数据，生成所述待配准图像帧。
  26. 根据权利要求25所述的图像配准装置，其中，第二数据处理单元用于构建与所述图像传感器采集的图像帧大小对应的初始权值矩阵；根据所述至少一个运动目标在所述图像帧中的当前分布区域以及预测分布区域，对所述初始权值矩阵进行赋值，得到目标赋值权值矩阵；将所述目标赋值权值矩阵与所述图像帧进行点乘操作，得到所述待配准图像帧。
  27. 根据权利要求26所述的图像配准装置，其中，第二数据处理单元，用于对所述初始权值矩阵中与所述至少一个运动目标在所述图像帧中的当前分布区域匹配的第一区域进行赋值，得到第一赋值权值矩阵；对所述初始权值矩阵中与所述至 少一个运动目标在所述图像帧中的预测分布区域匹配的第二区域进行赋值，得到第二赋值权值矩阵；结合所述第一赋值权值矩阵和所述第二赋值权值矩阵，生成所述目标赋值权值矩阵。
  28. 根据权利要求24至27中任意一项所述的图像配准装置，其中，所述图像配准模块用于根据所述目标事件序列和所述空间配准参数，确定所述待配准图像帧中至少一个运动目标的运动信息，得到所述配准结果图像。
  29. 一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如权利要求1-4中任一所述的方法，和/或实现如权利要求5-14中任一所述的方法。
  30. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-4中任一所述的方法，和/或实现如权利要求5-14中任一所述的方法。
PCT/CN2021/111207 2020-08-06 2021-08-06 图像配准方法及装置、计算机设备、介质 WO2022028576A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202010785636.7 2020-08-06
CN202010784874.6A CN111951312A (zh) 2020-08-06 2020-08-06 图像配准、图像采集时间配准方法、装置、设备及介质
CN202010785636.7A CN111951313B (zh) 2020-08-06 2020-08-06 图像配准方法、装置、设备及介质
CN202010784874.6 2020-08-06

Publications (1)

Publication Number Publication Date
WO2022028576A1 true WO2022028576A1 (zh) 2022-02-10

Family

ID=80117058

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/111207 WO2022028576A1 (zh) 2020-08-06 2021-08-06 图像配准方法及装置、计算机设备、介质

Country Status (1)

Country Link
WO (1) WO2022028576A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108090935A (zh) * 2017-12-19 2018-05-29 清华大学 混合相机系统及其时间标定方法及装置
CN108734739A (zh) * 2017-04-25 2018-11-02 北京三星通信技术研究有限公司 用于时间对齐标定、事件标注、数据库生成的方法及装置
CN109215063A (zh) * 2018-07-05 2019-01-15 中山大学 一种事件触发相机与三维激光雷达的配准方法
US20200134827A1 (en) * 2018-10-26 2020-04-30 Samsung Electronics Co., Ltd. Method and apparatus for image segmentation using an event sensor
CN111951312A (zh) * 2020-08-06 2020-11-17 北京灵汐科技有限公司 图像配准、图像采集时间配准方法、装置、设备及介质
CN111951313A (zh) * 2020-08-06 2020-11-17 北京灵汐科技有限公司 图像配准方法、装置、设备及介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108734739A (zh) * 2017-04-25 2018-11-02 北京三星通信技术研究有限公司 用于时间对齐标定、事件标注、数据库生成的方法及装置
CN108090935A (zh) * 2017-12-19 2018-05-29 清华大学 混合相机系统及其时间标定方法及装置
CN109215063A (zh) * 2018-07-05 2019-01-15 中山大学 一种事件触发相机与三维激光雷达的配准方法
US20200134827A1 (en) * 2018-10-26 2020-04-30 Samsung Electronics Co., Ltd. Method and apparatus for image segmentation using an event sensor
CN111951312A (zh) * 2020-08-06 2020-11-17 北京灵汐科技有限公司 图像配准、图像采集时间配准方法、装置、设备及介质
CN111951313A (zh) * 2020-08-06 2020-11-17 北京灵汐科技有限公司 图像配准方法、装置、设备及介质

Similar Documents

Publication Publication Date Title
CN111951313B (zh) 图像配准方法、装置、设备及介质
Baldwin et al. Time-ordered recent event (tore) volumes for event cameras
Zhang et al. Spiking transformers for event-based single object tracking
Sekikawa et al. Eventnet: Asynchronous recursive event processing
CN105160703B (zh) 一种使用时域视觉传感器的光流计算方法
JP2018170003A (ja) ビデオ中のイベントの検出装置、方法及び画像処理装置
US11200681B2 (en) Motion detection method and motion detection system with low computational complexity and high detection accuracy
US11501536B2 (en) Image processing method, an image processing apparatus, and a surveillance system
WO2022121130A1 (zh) 电力目标检测方法、装置、计算机设备和存储介质
CN111695681B (zh) 一种高分辨率动态视觉观测方法及装置
CN112399032A (zh) 一种基于检测器的脉冲式图像传感器的光流获取方法
WO2022028576A1 (zh) 图像配准方法及装置、计算机设备、介质
Bai et al. Accurate and efficient frame-based event representation for aer object recognition
CN105203045B (zh) 一种基于异步时域视觉传感器的产品形状完整性检测系统及检查方法
Miura et al. The examination of the image correction of the moving-object detection for low illumination video image
WO2018179119A1 (ja) 映像解析装置、映像解析方法および記録媒体
CN111951312A (zh) 图像配准、图像采集时间配准方法、装置、设备及介质
CN111127355A (zh) 一种对缺损光流图进行精细补全的方法及其应用
Duan et al. NeuroZoom: Denoising and super resolving neuromorphic events and spikes
Oliveira et al. FPGA architecture for static background subtraction in real time
CN110874589B (zh) 人脸照片获取方法与系统
JP2022054057A (ja) 情報処理装置および情報処理システム
CN110826455A (zh) 一种目标识别方法及图像处理设备
CN111860229A (zh) 异常行为智能识别方法及其装置和存储介质
JP4942197B2 (ja) テンプレート作成装置及び表情認識装置並びにその方法、プログラム及び記録媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21854524

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 24.05.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21854524

Country of ref document: EP

Kind code of ref document: A1