CN114926514A - Registration method and device of event image and RGB image - Google Patents
Registration method and device of event image and RGB image Download PDFInfo
- Publication number
- CN114926514A CN114926514A CN202210519218.2A CN202210519218A CN114926514A CN 114926514 A CN114926514 A CN 114926514A CN 202210519218 A CN202210519218 A CN 202210519218A CN 114926514 A CN114926514 A CN 114926514A
- Authority
- CN
- China
- Prior art keywords
- event
- registered
- image
- registration
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Abstract
The application discloses a registration method and a registration device of an event image and an RGB image. The registration method of the event image and the RGB image comprises the following steps: acquiring a contour image of an image to be registered; acquiring an event image of an image to be registered; acquiring each event sub-graph to be registered according to the event image; and registering each event sub-graph to be registered according to the contour image, so as to acquire the registration position information of each event sub-graph to be registered in the contour image. The application provides a registration method of an event image and an RGB image, which can realize accurate registration of a high-resolution event image and the RGB image.
Description
Technical Field
The application relates to the technical field of event image processing, in particular to a registration method and a registration device for an event image and an RGB image.
Background
Event cameras, also known as Dynamic Vision Sensors (DVS), as a new type of image sensor, perceive the world by asynchronously detecting the logarithmic intensity change of the image at each pixel and generating an event stream encoding time, pixel position and polarity of the intensity change. Compared with the traditional camera widely applied at present, the event camera has the obvious advantages of very high dynamic range, high time resolution, low delay, low power consumption, obvious motion blur reduction and the like. Therefore, the sensor provides a new direction for the development of computer vision and robots, and has revolutionary influence.
The event camera shows the superiority of the traditional RGB camera in the mature fields of feature detection and tracking, optical flow estimation, three-dimensional reconstruction and the like. In addition, in some tasks, such as target detection and identification, target segmentation, target tracking and visualization events, the frame rate is high, the motion blur problem is almost eliminated, and the event data has strong advantages. However, neither the event images nor the RGB images used in these studies were registered. In fact, due to the difference of lens, angle of view, sensor, and placement, the geometric information of the images captured by the two cameras will be different, which means that the distortion of the two images will be different, but since the fusion of the data of the two cameras needs to be implemented in future research or practical application, the performance of these researches may be affected by whether the events and the RGB images are registered or not.
For the fusion of the event camera and the RGB camera data, a large number of event camera data sets have been released at present, which cover a plurality of fields including driving scenes, optical flows, target tracking, and the like, and show a wide prospect for environmental perception under high-speed movement due to the advantages of no motion blur, low time delay, and the like.
In the prior art, there is a device that can simultaneously acquire an event image and a grayscale image, but the hardware is limited, and two images that are simultaneously acquired and accurately registered cannot achieve high resolution, and such an image only contains a grayscale image and cannot have color information as rich as an RGB image, so that other methods are required to achieve accurate registration of the event image and the RGB image with high resolution.
There is also a method of acquiring RGB images and event images by placing an RGB camera and an event camera adjacent to each other, but since the two cameras do not overlap, there is a problem that the images do not overlap.
Accordingly, a solution is desired to solve or at least mitigate the above-mentioned deficiencies of the prior art.
Disclosure of Invention
The present invention is directed to a method for registering an event image and an RGB image to solve at least one of the above problems.
In one aspect of the present invention, a method for registering an event image and an RGB image is provided, and the method for registering the event image and the RGB image includes:
acquiring a contour image of an image to be registered;
acquiring an event image of an image to be registered;
acquiring each event sub-graph to be registered according to the event image;
and registering each event sub-image to be registered according to the contour image, so as to acquire the registration position information of each event sub-image to be registered in the contour image.
Optionally, the obtaining each event sub-graph to be registered according to the event image includes:
segmenting the event image to obtain a plurality of event sub-images;
acquiring the event number of each event subgraph;
and acquiring an event sub-graph with the event number more than a preset threshold value as an event sub-graph to be registered.
Optionally, registering each event sub-graph to be registered according to the contour image, so as to acquire registration position information of each event sub-graph to be registered in the contour image, includes:
and performing the following operations for each event subgraph to be registered:
respectively superposing the event subgraphs to be registered to different positions of the outline image so as to form at least two superposed images, wherein the position where the event subgraphs to be registered are superposed to the outline image for the first time is an initial position;
respectively calculating the contrast of each superposed image;
and acquiring the position information of the event subgraph to be registered in the superposed image with the minimum contrast as the registration position information.
Optionally, after one or more event subgraphs to be registered acquire registration position information, registering each event subgraph to be registered according to the contour image, so as to acquire registration position information of each event subgraph to be registered in the contour image, further includes:
before the event subgraphs to be registered are respectively superposed at different positions of the contour image, the following operations are further performed for one or more of each event subgraph to be registered except the acquired registration position information:
and acquiring an initial superposition position through a parameter transmission method according to the registration position information acquired by one of the sub-graphs of the event to be registered, which has acquired the registration position information.
Optionally, each event subgraph to be registered includes an event region and a blank region;
the event subgraph to be registered, which has acquired the registration position information, is called a registered event subgraph;
the parameter transferring method comprises the following steps:
acquiring a pre-estimated position to be registered of a preset pixel point to be registered in an event region of an event subgraph to be registered in the outline image;
acquiring the registered position of a registered preset pixel point in the event region of each registered event subgraph in the contour image;
acquiring a registered event subgraph corresponding to a registered estimated position closest to the estimated position to be registered;
acquiring the position of the origin of the registered event subgraph as the initial position of the origin of the event subgraph to be registered;
and acquiring the initial superposition position of the event subgraph to be registered according to the initial position of the origin of the event subgraph to be registered.
Optionally, if there are more than two registered event subgraphs corresponding to the registered predicted position closest to the predicted position to be registered, the parameter passing method further includes:
acquiring initial positions of the original points of the registered event subgraphs closest to the estimated position to be registered;
shifting the initial position of each origin in one or more directions by taking a preset unit as a distance;
acquiring the position of the origin in the canvas of which the first contour diagram is moved out in each direction in the direction as a position to be combined;
combining the positions to be combined to form a new origin position;
and taking the new origin position as the initial position of the origin of the event subgraph to be registered.
Optionally, after the event sub-images to be registered are registered according to the contour image, so as to obtain registration position information of each event sub-image to be registered in the contour image, the registration method of the event image and the RGB image further includes:
smoothing the registration position information of each event sub-image to be registered in the contour image so as to obtain the mean value of all the optimal registration positions;
sequencing the optimal registration positions according to the difference between the optimal registration positions and the average value from small to large;
acquiring event subgraphs to be registered corresponding to the first sequenced optimal registration position as optimal event subgraphs to be registered;
and adjusting the positions of other event subgraphs to be registered on the basis of the optimal event subgraph to be registered, thereby ensuring that the event subgraphs to be registered are not overlapped or missing.
Optionally, after the optimal event subgraph to be registered is used as a basis and the positions of other event subgraphs to be registered are adjusted, so as to ensure that there is no overlap or deletion between the event subgraphs to be registered, the registration method of the event image and the RGB image further includes:
judging whether event subgraphs to be registered with events lost exist in the event subgraphs to be registered, if so, judging whether event subgraphs to be registered with events lost exist in the event subgraphs to be registered
And performing event compensation on the event subgraph to be registered with event loss by adopting an optical flow estimation method, so as to compensate the event loss in the event subgraph to be registered.
Optionally, the acquiring a contour image of the image to be registered includes:
acquiring an RGB image;
and (5) carrying out edge detection on the RGB image by adopting a Canny operator to obtain a contour image of the image to be registered.
The present application also provides an RGB image-based event image registration apparatus, including:
the system comprises a contour image acquisition module, a registration module and a registration module, wherein the contour image acquisition module is used for acquiring a contour image of an image to be registered;
the event image acquisition module is used for acquiring an event image of an image to be registered;
the event sub-graph to be registered acquisition module is used for acquiring each event sub-graph to be registered according to the event image;
and the registration module is used for registering each event sub-graph to be registered according to the contour image so as to acquire registration position information of each event sub-graph to be registered in the contour image.
Advantageous effects
The application provides a registration method of an event image and an RGB image, which can realize accurate registration of a high-resolution event image and the RGB image. Specifically, the method and the device can analyze the tiny regions independently by segmenting the event subgraph, and because the distortion condition and the distortion of each region are different, the registration can be performed by segmenting the single regions, and the registration can be more accurate. And the comparison is carried out by adopting a contrast mode, so that the calculation amount can be saved as much as possible, and the response speed is higher. The initial position is searched by adopting a parameter transmission method, so that the registration speed can be further improved on the premise of ensuring the registration accuracy.
Drawings
FIG. 1 is a schematic flowchart of a registration method of an event image and an RGB image according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a system implementing the method for registering the event image and the RGB image shown in FIG. 1 according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a contour image of a registration method of an event image and an RGB image according to an embodiment of the present application;
FIG. 4 is a canvas schematic diagram of a registration method of an event image and an RGB image according to an embodiment of the application;
FIG. 5 is a schematic illustration of a registration process according to an embodiment of the present application;
FIG. 6 is a diagram illustrating a parameter passing method according to an embodiment of the present application;
fig. 7 is another schematic diagram of a parameter passing method according to an embodiment of the present application.
Detailed Description
In order to make the implementation objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are a subset of the embodiments in the present application and not all embodiments in the present application. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application. Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart illustrating a registration method of an event image and an RGB image according to an embodiment of the present application.
Fig. 2 is a schematic diagram of a system for implementing the registration method between the event image and the RGB image shown in fig. 1 according to an embodiment of the present application.
The registration method of the event image and the RGB image as shown in fig. 1 includes:
step 1: acquiring a contour image of an image to be registered;
and 2, step: acquiring an event image of an image to be registered;
and 3, step 3: acquiring each event subgraph to be registered according to the event images;
and 4, step 4: and registering each event sub-image to be registered according to the contour image, so as to acquire the registration position information of each event sub-image to be registered in the contour image.
The application provides a registration method of an event image and an RGB image, which can realize accurate registration of a high-resolution event image and the RGB image.
In this embodiment, acquiring each event sub-graph to be registered according to the event image includes:
segmenting the event image to obtain a plurality of event sub-images;
acquiring the event number of each event subgraph;
and acquiring an event sub-graph with the event number more than a preset threshold value as an event sub-graph to be registered.
In this embodiment, registering each event sub-graph to be registered according to the contour image, so as to acquire registration position information of each event sub-graph to be registered in the contour image, includes:
and performing the following operations for each event subgraph to be registered:
respectively superposing the event subgraph to be registered to different positions of the outline image so as to form at least two superposed images, wherein the position where the event subgraph to be registered is superposed to the outline image for the first time is an initial position;
respectively calculating the contrast of each superposed image;
and acquiring the position information of the event subgraph to be registered in the superposed image with the minimum contrast as the registration position information.
In this embodiment, after one or more event subgraphs to be registered acquire registration position information, registering each event subgraph to be registered according to the contour image, so as to acquire registration position information of each event subgraph to be registered in the contour image, further includes:
before the event subgraphs to be registered are respectively superposed at different positions of the contour image, the following operations are further performed for one or more of each event subgraph to be registered except the acquired registration position information:
and acquiring an initial superposition position by a parameter transmission method according to the registration position information acquired by one of the sub-images of the event to be registered, which have acquired the registration position information.
In this embodiment, each event sub-graph to be registered includes an event region and a blank region;
the event subgraph to be registered, which has acquired the registration position information, is called a registered event subgraph;
the parameter transferring method comprises the following steps:
acquiring a pre-estimated position to be registered of a preset pixel point to be registered in the event area of the event subgraph to be registered in the outline image;
acquiring registered positions of registered preset pixel points in the event regions of the registered event sub-graphs in the contour image;
acquiring a registered event subgraph corresponding to a registered estimated position closest to the position to be registered;
acquiring the position of the origin of the registered event subgraph as the initial position of the origin of the event subgraph to be registered;
and acquiring the initial superposition position of the event subgraph to be registered according to the initial position of the origin of the event subgraph to be registered.
In this embodiment, if there are more than two registered event sub-graphs corresponding to the registered estimated positions closest to the estimated position to be registered, the parameter delivery method further includes:
acquiring initial positions of the original points of the registered event subgraphs closest to the estimated position to be registered;
shifting the initial position of each origin in one or more directions by taking a preset unit as a distance;
acquiring the position of the origin in the canvas of the first moved outline drawing in each direction in the direction as a position to be combined;
combining the positions to be combined to form a new origin position;
and taking the new origin position as the initial position of the origin of the event subgraph to be registered.
In this embodiment, after the registering is performed on each event sub-graph to be registered according to the contour image, so as to obtain the registration position information of each event sub-graph to be registered in the contour image, the registration method of the event image and the RGB image further includes:
smoothing the registration position information of each event sub-image to be registered in the contour image so as to obtain the mean value of all the optimal registration positions;
sequencing the optimal registration positions according to the difference between the optimal registration positions and the average value from small to large;
acquiring event subgraphs to be registered corresponding to the first sequenced optimal registration position as optimal event subgraphs to be registered;
and adjusting the positions of other event subgraphs to be registered on the basis of the optimal event subgraph to be registered, thereby ensuring that the event subgraphs to be registered are not overlapped or missing.
In this embodiment, after the optimal event sub-graph to be registered is used as a basis and the positions of other event sub-graphs to be registered are adjusted, so as to ensure that there is no overlap or lack between the event sub-graphs to be registered, the registration method of the event image and the RGB image further includes:
judging whether event subgraphs to be registered with events lost exist in each event subgraph to be registered, if so, judging whether the event subgraphs to be registered with events lost exist in each event subgraph to be registered
And performing event compensation on the event subgraph to be registered with event loss by adopting an optical flow estimation method, so as to compensate the event loss in the event subgraph to be registered.
In this embodiment, the acquiring the contour image of the image to be registered includes:
acquiring an RGB image;
and (5) carrying out edge detection on the RGB image by adopting a Canny operator to obtain a contour image of the image to be registered.
The present application is further described in detail below by way of examples, and it should be understood that the examples are not to be construed as limiting the present application in any way.
The registration method of the event image and the RGB image comprises the following steps:
step 1: acquiring a contour image of an image to be registered, wherein in the embodiment, the RGB image resolution is 1440 × 1080, and performing edge detection on the RGB image by using a Canny operator to obtain a contour map (see fig. 3);
acquiring an event image of an image to be registered, wherein in the embodiment, an output data stream of the event camera is in a data format of (x, y, p, t), where (x, y) is spatial information of an event, p represents polarity information of the event, and t is asynchronous time information triggered by the event. By characterizing the event stream as an image, namely: and selecting a fixed time interval delta t, and overlapping the event data streams to obtain an event image with the resolution of 640 multiplied by 480. Scaling the event image in a specific scale to realize that the size of the frame of the object in the event image is the same as that of the object in the RGB image (1440 multiplied by 1080);
acquiring each event sub-graph to be registered according to the event image, in this embodiment, performing region segmentation on the event image, dividing the event image into 4 × 4 event sub-graphs, and performing image pixel-level registration based on the event sub-graphs and the contour image, it can be understood that the number of segmentation of a specific region can be set according to needs, for example, 6 × 6 and the like;
and registering each event sub-graph to be registered according to the contour image so as to acquire the registration position information of each event sub-graph to be registered in the contour image.
In this embodiment, registering each event sub-graph to be registered according to the contour image, so as to acquire registration position information of each event sub-graph to be registered in the contour image includes:
and performing the following operations for each event subgraph to be registered:
respectively superposing the event subgraph to be registered to different positions of the outline image so as to form at least two superposed images, wherein the position where the event subgraph to be registered is superposed to the outline image for the first time is an initial position;
respectively calculating the contrast of each superposed image;
and acquiring the position information of the event subgraph to be registered in the superposed image with the minimum contrast as the registration position information.
In the embodiment, each event sub-graph to be registered is respectively superposed into the outline image, the contrast of the superposed image is calculated, and the optimal registration position of the sub-graph is searched by minimizing the contrast of the superposed image.
The definition of image contrast is as follows:
C＝∑ δ δ(i,j) 2 P δ (i,j)
δ(i,j)＝|i-j|
where i, j is the gray scale value of the adjacent pixel, δ (i, j) is the gray scale difference value of the adjacent pixel, P δ (i, j) represents a distribution probability that the gray scale difference between adjacent pixels is δ. The contrast represents the size of the image gray difference, and the method adopts the contrast as an evaluation index of image registration.
In this embodiment, acquiring each event sub-graph to be registered according to the event image includes:
segmenting the event image to obtain a plurality of event sub-images;
acquiring the event number of each event subgraph;
and acquiring an event sub-graph with the event number more than a preset threshold value as an event sub-graph to be registered.
In order to accelerate the registration process, the number of events in each event sub-graph to be registered is calculated, if the number of events in the sub-graph is small, the sub-graph is considered to have little influence on the whole registration process, and an optimal registration position is given to the event sub-graph in a default mode.
Wherein m is a sequence number of subgraphs, N m The number of events in the m subgraph belongs to a threshold value, and when the number of subgraphs with the sequence of m is less than the threshold value, Mask m And 0, giving the event subgraph a default optimal registration position, and not performing subsequent registration operation to accelerate the registration process.
It is to be understood that in other embodiments, registration may be performed for each event sub-graph to be registered.
In this embodiment, after one or more event subgraphs to be registered acquire registration position information, registering each event subgraph to be registered according to a contour image, so as to acquire registration position information of each event subgraph to be registered in the contour image, further includes:
before the event subgraphs to be registered are respectively superposed at different positions of the contour image, the following operations are further performed for one or more of each event subgraph to be registered except the acquired registration position information:
and acquiring an initial superposition position by a parameter transmission method according to the registration position information acquired by one of the sub-images of the event to be registered, which have acquired the registration position information.
In the embodiment, a set of parameter transmission method is designed for the registration process, so that the registration efficiency of the event image and the RGB image is improved. Specifically, each event subgraph to be registered comprises an event area and a blank area;
the event subgraph to be registered, which has acquired the registration position information, is called a registered event subgraph;
the parameter transferring method comprises the following steps:
acquiring a pre-estimated position to be registered of a preset pixel point to be registered in an event region of an event subgraph to be registered in the outline image;
acquiring registered positions of registered preset pixel points in the event regions of the registered event sub-graphs in the contour image;
acquiring a registered event subgraph corresponding to a registered estimated position closest to the position to be registered;
acquiring the position of the origin of the registered event subgraph as the initial position of the origin of the event subgraph to be registered;
and acquiring an initial superposition position of the event subgraph to be registered according to the initial position of the origin of the event subgraph to be registered.
By adopting the method, except that the event areas of one or more registered event subgraphs need to calculate all superposed contrasts, other event subgraphs to be registered can directly lock an initial superposed position, and then the superposed events are superposed on the left and right of the initial superposed position so as to acquire the registered position information.
For example, assume there are 9 event sub-graphs to be registered A, B, C, D, E, F, G, H, I, where event sub-graph a to be registered is already registered;
in this embodiment, the preset pixel points to be registered in the event region of the event sub-graph to be registered may be all central points of the event sub-graph to be registered, for example, one event image is 1440 × 1080, that is, the canvas is 1440 × 1080, if the event image is divided into 9 event sub-graphs, each event sub-graph is also 9 times, but the canvas is 1440 × 1080, that is, each event sub-graph is still placed on the canvas of 1440 × 1080, the event sub-graphs are located at corresponding positions of the canvas, where most of the canvas is a blank region, and only the event sub-graph is located in the event region (see fig. 4). For example, suppose event sub-graph a, then that is, the B, C, D, E, F, G, H, I regions are all blank regions, and only a is the event image.
The method includes the steps of obtaining a pre-determined position to be registered of a pre-determined pixel point to be registered in an event area of an event sub-graph to be registered in the outline image, and obtaining the pre-determined position to be registered of B if the event sub-graph to be registered is B.
In this embodiment, the position to be registered of the preset pixel to be registered in the contour image is the midpoint position, for example, what we want to find is the accurate registration position of the event sub-image to be registered in the contour image, and when the position is the accurate registration position, the position where the midpoint position should be located is the estimated position to be registered, for example, if the accurate registration position of the preset pixel to be registered in the event sub-image to be registered in the contour image should be (121 ), then (121 ) should be the estimated position to be registered.
In this embodiment, the registered positions of the registered preset pixel points in the event region of each registered event sub-graph in the contour image are obtained, in this embodiment, each registered preset pixel point needs to correspond to each other, for example, assuming that one registered preset pixel point is a midpoint of the event sub-graph, other registered preset pixel points should also be midpoints of the respective event sub-graphs.
In this embodiment, assuming that a B event subgraph is registered, since the a subgraph is already registered and other subgraphs are not yet registered, the origin of the a subgraph is used as the initial position of the origin of the event subgraph to be registered;
and acquiring the initial superposition position of the event subgraph to be registered according to the initial position of the origin of the event subgraph to be registered.
In a graph, as long as the position of a point in the graph is obtained, the positions of other pixel points can be known, so that the initial superposition position of the event sub-graph to be registered is obtained.
In this way, on one hand, an initial stacking position can be quickly assigned to other event subgraphs to be registered, and on the other hand, the stacking of all positions of each event subgraph to be registered can be prevented.
For example, assuming that a canvas has 100 pixels, if the contrast calculation is to be performed, theoretically, each pixel of 100 pixels needs to be superimposed once, for example, the origin is superimposed on the first pixel to obtain the primary contrast, and the origin is superimposed on the second pixel to obtain the primary contrast.
When the method is adopted to give the initial superposition position, the initial superposition position is theoretically a position closer to the optimal superposition position, and at the moment, the optimal registration position information can be found only by providing a deviation range, for example, a mode of deviating 10 pixel points in each direction.
If more than two registered event subgraphs corresponding to the registered estimated positions closest to the position to be registered are simultaneously provided, the parameter transmission method further comprises the following steps:
acquiring initial positions of the original points of the registered event subgraphs closest to the estimated position to be registered;
shifting the initial position of each origin in one or more directions by taking a preset unit as a distance;
acquiring the position of the origin in the canvas of the first moved outline drawing in each direction in the direction as a position to be combined;
combining the positions to be combined to form a new origin position;
and taking the new origin position as the initial position of the origin of the event subgraph to be registered.
Since the origin is taken as the initial position, and the origin is theoretically the closest point to the coordinates (0,0) of the canvas, when the point first moved out of the canvas is necessarily the closest point to the coordinates (0,0) of the canvas, and therefore, the point shifts the initial position of each origin in one or more directions by taking the preset unit as the distance, the origin in the canvas from which the outline diagram is first moved out is the point which should be closest to the point 0 in the direction, and therefore, the new origin position formed by combining the points should be the coordinate point closest to the real origin in practice. In terms of distance, assuming that a is a registered event sub-graph and B is a registered event sub-graph, the origin of a is actually located at the (1,1) position of the canvas after registration (theoretically, the origin should be located at (0,0), but is not necessarily exactly at the origin of the canvas due to actual deviation), and the origin of B is actually located at the (2,2) position of the canvas after registration, and the initial position of each origin is shifted in one or more directions by taking the preset unit as distance, the origin of a is found to be moved first, and thus the origin of a is taken as the new origin position.
In another embodiment, another parameter transfer method is adopted for the registration process, so that the registration efficiency of the event image and the RGB image is improved. In the registration process, the event sub-graph is divided into a left part and a right part (it can be understood that the division into several parts can be set by itself according to needs, for example, the division into 3 parts, 4 parts, and the like), registration of the left part of the event image is performed first, the event sub-graph to be registered is registered with a specific region, the size of the region is determined by the order of the event sub-graphs to be registered, and the image superimposition position with the minimum contrast of each region in the region is calculated as the optimal registration position, respectively, in the method, the specific value distribution is 16,8,8,16,18,14, and the specific registration flow is shown in fig. 5:
and sequentially carrying out region registration on the subgraph according to the transfer sequence, obtaining the optimal registration position based on a contrast minimization method, and carrying out parameter transfer on the optimal registration position of the subgraph by a self-designed parameter transfer method. The specific parameter delivery rules are as follows:
when all event subgraphs are used as event subgraphs to be registered:
wherein, E 1 ，E 2 Respectively, represent the regions shown in fig. 6.
When an event sub-graph with the number of events more than a preset threshold is acquired as an event sub-graph to be registered (as shown in fig. 7):
in fig. 7a, 2 and 3 indicate that the sub-graph does not perform the registration process due to too few events, in fig. 7b, 3 indicates that the sub-graph does not perform the registration process due to too few events, in fig. 7c, 3 indicates that the sub-graph does not perform the registration process due to too few events, and in fig. 7d, 2 indicates that the sub-graph does not perform the registration process due to too few events. The arrow direction indicates the transfer direction of the optimal registration position.
In this embodiment, after registering each event sub-image to be registered according to the contour image, so as to obtain registration position information of each event sub-image to be registered in the contour image, the registration method of the event image and the RGB image further includes:
smoothing the registration position information of each event sub-image to be registered in the contour image so as to obtain the mean value of all the optimal registration positions;
sequencing the optimal registration positions according to the difference between the optimal registration positions and the average value from small to large;
acquiring event subgraphs to be registered corresponding to the first ordered optimal registration position as optimal event subgraphs to be registered;
and adjusting the positions of other event subgraphs to be registered on the basis of the optimal event subgraph to be registered, thereby ensuring that the event subgraphs to be registered are not overlapped or missing.
In the registration process, because the sub-images are not connected with each other, the optimal registration position is only searched according to the contrast, and the pixel overlapping or missing of the superposed image is easily caused. In order to solve the problem, after the optimal registration positions of all the subgraphs are obtained, all the registration positions are subjected to smoothing processing, and the mean value and the variance of all the optimal registration positions are taken. And sequencing the event subgraphs according to the difference between the optimal registration position and the mean value from small to large, ensuring that the subgraph position with small difference value is the same as the original optimal position, mapping other subgraphs through affine transformation, and compensating the influence caused by pixel overlapping or missing after registration while ensuring the registration accuracy.
In this embodiment, after adjusting the positions of other event subgraphs to be registered based on the optimal event subgraph to be registered, so as to ensure that there is no overlap or lack between the event subgraphs to be registered, the registration method of the event image and the RGB image further includes:
judging whether event subgraphs to be registered with events lost exist in each event subgraph to be registered, if so, judging whether the event subgraphs to be registered with events lost exist in each event subgraph to be registered
And performing event compensation on the event subgraph to be registered with event loss by adopting an optical flow estimation method, so as to compensate the event loss in the event subgraph to be registered.
Due to the fact that the frame of the registered image is limited, the event image is easy to cause event missing in the sub-image superposition process, the number of events is reduced, and the registration effect is poor. Based on the imaging principle of the event camera, the method of optical flow estimation is adopted to make up for the missing event. And evaluating brightness change by adopting an optical flow estimation method through all events in the graph, estimating the motion direction, and triggering positive and negative events through a brightness change threshold so as to compensate for missing events.
The application also provides a registration device of the event image and the RGB image, the registration device of the event image and the RGB image comprises a contour image acquisition module, an event subgraph acquisition module to be registered and a registration module, wherein the contour image acquisition module is used for acquiring a contour image of the image to be registered; the event image acquisition module is used for acquiring an event image of an image to be registered; the event subgraph to be registered acquiring module is used for acquiring each event subgraph to be registered according to the event image; and the registration module is used for registering each event sub-image to be registered according to the contour image so as to acquire registration position information of each event sub-image to be registered in the contour image.
It will be appreciated that the above description of the method applies equally to the description of the apparatus.
The application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the registration method of the event image and the RGB image as above when executing the computer program.
The present application also provides a computer-readable storage medium storing a computer program which, when executed by a processor, is capable of implementing the registration method of an event image and an RGB image as above.
Fig. 2 is an exemplary block diagram of an electronic device capable of implementing a registration method of an event image and an RGB image provided according to an embodiment of the present application.
As shown in fig. 2, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processing unit 503, the memory 504 and the output interface 505 are connected to each other through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and further connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on computer-executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.
That is, the electronic device shown in fig. 2 may also be implemented to include: a memory storing computer executable instructions; and one or more processors which, when executing the computer executable instructions, may implement the registration method of the event image and the RGB image described in connection with fig. 1.
In one embodiment, the electronic device shown in FIG. 2 may be implemented to include: a memory 504 configured to store executable program code; one or more processors 503 configured to execute the executable program code stored in the memory 504 to perform the registration method of the event image and the RGB image in the above embodiments.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media include both non-transitory and non-transitory, removable and non-removable media that implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Furthermore, it will be obvious that the term "comprising" does not exclude other elements or steps. A plurality of units, modules or devices recited in the device claims may also be implemented by one unit or overall device by software or hardware.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks identified in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The Processor referred to in this embodiment may be a Central Processing Unit (CPU), and may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the apparatus/terminal device by executing or performing the computer programs and/or modules stored in the memory, as well as invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
In this embodiment, the module/unit integrated with the apparatus/terminal device may be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as a separate product. Based on such understanding, all or part of the flow in the method according to the embodiments of the present invention may also be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of the embodiments of the method. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunications signal, software distribution medium, etc. It should be noted that the computer readable medium may contain content that is appropriately increased or decreased as required by legislation and patent practice in the jurisdiction. Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
Furthermore, it will be obvious that the term "comprising" does not exclude other elements or steps. A plurality of units, modules or devices recited in the device claims may also be implemented by one unit or overall device by software or hardware.
Although the invention has been described in detail hereinabove with respect to a general description and specific embodiments thereof, it will be apparent to those skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, it is intended that all such modifications and alterations be included within the scope of this invention as defined in the appended claims.
Claims (10)
1. A registration method of an event image and an RGB image is characterized in that the registration method of the event image and the RGB image comprises the following steps:
acquiring a contour image of an image to be registered;
acquiring an event image of an image to be registered;
obtaining each event sub-graph to be registered according to the event image;
and registering each event sub-image to be registered according to the contour image, so as to acquire the registration position information of each event sub-image to be registered in the contour image.
2. The registration method for event images and RGB images according to claim 1, wherein the obtaining each event sub-graph to be registered according to the event image comprises:
segmenting the event image to obtain a plurality of event sub-images;
acquiring the event number of each event subgraph;
and acquiring an event sub-graph with the event number more than a preset threshold value as an event sub-graph to be registered.
3. The registration method of an event image and an RGB image according to claim 2, wherein registering each event sub-graph to be registered according to the contour image, so as to obtain the registration position information of each event sub-graph to be registered in the contour image includes:
and performing the following operations for each event subgraph to be registered:
respectively superposing the event subgraph to be registered to different positions of the outline image so as to form at least two superposed images, wherein the position where the event subgraph to be registered is superposed to the outline image for the first time is an initial position;
respectively calculating the contrast of each superposed image;
and acquiring the position information of the event subgraph to be registered in the superposed image with the minimum contrast as the registration position information.
4. The method for registering an event image and an RGB image according to claim 3, wherein after one or more event sub-images to be registered acquire the registration position information, the registering is performed for each event sub-image to be registered according to the contour image, so as to acquire the registration position information of each event sub-image to be registered in the contour image, further comprising:
before the event subgraphs to be registered are respectively superposed to different positions of the outline image, the following operations are further carried out for one or more of the other event subgraphs to be registered except the registration position information acquisition:
and acquiring an initial superposition position by a parameter transmission method according to the registration position information acquired by one of the sub-images of the event to be registered, which have acquired the registration position information.
5. The registration method of an event image and an RGB image as claimed in claim 4, wherein each event sub-graph to be registered includes an event area and a blank area;
the event subgraph to be registered, which has acquired the registration position information, is called a registered event subgraph;
the parameter transferring method comprises the following steps:
acquiring a pre-estimated position to be registered of a preset pixel point to be registered in an event region of an event subgraph to be registered in the outline image;
acquiring the registered position of a registered preset pixel point in the event region of each registered event subgraph in the contour image;
acquiring a registered event subgraph corresponding to a registered estimated position closest to the estimated position to be registered;
acquiring the position of the origin of the registered event subgraph as the initial position of the origin of the event subgraph to be registered;
and acquiring the initial superposition position of the event subgraph to be registered according to the initial position of the origin of the event subgraph to be registered.
6. The method as claimed in claim 5, wherein if there are two or more registered event sub-graphs corresponding to the registered estimated positions closest to the estimated position to be registered, the parameter passing method further comprises:
acquiring initial positions of the original points of the registered event subgraphs closest to the estimated position to be registered;
shifting the initial position of each origin in one or more directions by taking a preset unit as a distance;
acquiring the position of the origin in the canvas of which the first contour diagram is moved out in each direction in the direction as a position to be combined;
combining the positions to be combined to form a new origin position;
and taking the new origin position as the initial position of the origin of the event subgraph to be registered.
7. The registration method of an event image and an RGB image according to any one of claims 1 to 6, wherein after the registration is performed on each event sub-graph to be registered according to the contour image, so as to obtain the registration position information of each event sub-graph to be registered in the contour image, the registration method of the event image and the RGB image further comprises:
smoothing the registration position information of each event sub-image to be registered in the contour image so as to obtain the mean value of all the optimal registration positions;
sequencing the optimal registration positions according to the difference between the optimal registration positions and the average value from small to large;
acquiring event subgraphs to be registered corresponding to the first sequenced optimal registration position as optimal event subgraphs to be registered;
and adjusting the positions of other event subgraphs to be registered on the basis of the optimal event subgraph to be registered, thereby ensuring that the event subgraphs to be registered are not overlapped or missing.
8. The method for registering an event image and an RGB image according to claim 7, wherein after the event sub-graph to be registered is used as a base, the positions of other event sub-graphs to be registered are adjusted, so as to ensure that there is no overlap or lack between the event sub-graphs to be registered, the method for registering an event image and an RGB image further comprises:
judging whether event subgraphs to be registered with events lost exist in the event subgraphs to be registered, if so, judging whether event subgraphs to be registered with events lost exist in the event subgraphs to be registered
And performing event compensation on the event subgraph to be registered with event loss by adopting an optical flow estimation method, so as to compensate the event loss in the event subgraph to be registered.
9. The RGB image-based registration method for event images and RGB images according to claim 8, wherein the obtaining the contour image of the image to be registered includes:
acquiring an RGB image;
and (5) carrying out edge detection on the RGB image by adopting a Canny operator to obtain a contour image of the image to be registered.
10. An apparatus for registering an event image and an RGB image, comprising:
the system comprises a contour image acquisition module, a registration module and a registration module, wherein the contour image acquisition module is used for acquiring a contour image of an image to be registered;
the event image acquisition module is used for acquiring an event image of an image to be registered;
the event to be registered subgraph acquisition module is used for acquiring each event to be registered subgraph according to the event image;
and the registration module is used for registering each event sub-image to be registered according to the contour image so as to acquire registration position information of each event sub-image to be registered in the contour image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210519218.2A CN114926514B (en) | 2022-05-13 | 2022-05-13 | Registration method and device of event image and RGB image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210519218.2A CN114926514B (en) | 2022-05-13 | 2022-05-13 | Registration method and device of event image and RGB image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114926514A true CN114926514A (en) | 2022-08-19 |
| CN114926514B CN114926514B (en) | 2023-02-03 |
Family
ID=82807831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210519218.2A Active CN114926514B (en) | 2022-05-13 | 2022-05-13 | Registration method and device of event image and RGB image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114926514B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116152051A (en) * | 2023-02-27 | 2023-05-23 | 上海福柯斯智能科技有限公司 | Method and device for blocking registration image silhouette of X-ray image |
| CN117671472A (en) * | 2024-01-31 | 2024-03-08 | 吉林大学 | Underwater multi-target group identification method based on dynamic visual sensor |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160217198A1 (en) * | 2015-01-26 | 2016-07-28 | Samsung Electronics Co., Ltd. | User management method and apparatus |
| CN108182670A (en) * | 2018-01-15 | 2018-06-19 | 清华大学 | A kind of resolution enhancement methods and system of event image |
| CN110400361A (en) * | 2019-07-30 | 2019-11-01 | 上海联影医疗科技有限公司 | The method, apparatus and computer equipment of subset division and image reconstruction |
| CN111696143A (en) * | 2020-06-16 | 2020-09-22 | 清华大学 | Event data registration method and system |
-
2022
- 2022-05-13 CN CN202210519218.2A patent/CN114926514B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160217198A1 (en) * | 2015-01-26 | 2016-07-28 | Samsung Electronics Co., Ltd. | User management method and apparatus |
| CN108182670A (en) * | 2018-01-15 | 2018-06-19 | 清华大学 | A kind of resolution enhancement methods and system of event image |
| CN110400361A (en) * | 2019-07-30 | 2019-11-01 | 上海联影医疗科技有限公司 | The method, apparatus and computer equipment of subset division and image reconstruction |
| CN111696143A (en) * | 2020-06-16 | 2020-09-22 | 清华大学 | Event data registration method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116152051A (en) * | 2023-02-27 | 2023-05-23 | 上海福柯斯智能科技有限公司 | Method and device for blocking registration image silhouette of X-ray image |
| CN116152051B (en) * | 2023-02-27 | 2023-12-01 | 上海福柯斯智能科技有限公司 | Method and device for blocking registration image silhouette of X-ray image |
| CN117671472A (en) * | 2024-01-31 | 2024-03-08 | 吉林大学 | Underwater multi-target group identification method based on dynamic visual sensor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114926514B (en) | 2023-02-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110827200B (en) | Image super-resolution reconstruction method, image super-resolution reconstruction device and mobile terminal | |
| CN114926514B (en) | Registration method and device of event image and RGB image | |
| CN103325112B (en) | Moving target method for quick in dynamic scene | |
| US8290212B2 (en) | Super-resolving moving vehicles in an unregistered set of video frames | |
| Ding et al. | Spatio-temporal recurrent networks for event-based optical flow estimation | |
| US20060050788A1 (en) | Method and device for computer-aided motion estimation | |
| CN112435223B (en) | Target detection method, device and storage medium | |
| CN112288628A (en) | Aerial image splicing acceleration method and system based on optical flow tracking and frame extraction mapping | |
| CN113029128A (en) | Visual navigation method and related device, mobile terminal and storage medium | |
| CN115619826A (en) | Dynamic SLAM method based on reprojection error and depth estimation | |
| CN110689014B (en) | Method and device for detecting region of interest, electronic equipment and readable storage medium | |
| WO2017094140A1 (en) | Object detection device and object detection method | |
| CN112101205A (en) | Training method and device based on multi-task network | |
| Alletto et al. | Self-supervised optical flow estimation by projective bootstrap | |
| WO2014172875A1 (en) | Moving object detection | |
| US9392146B2 (en) | Apparatus and method for extracting object | |
| CN110322479B (en) | Dual-core KCF target tracking method based on space-time significance | |
| CN115546027B (en) | Image suture line determination method, device and storage medium | |
| TW201516965A (en) | Method of detecting multiple moving objects | |
| CN108389177B (en) | Vehicle bumper damage detection method and traffic safety early warning method | |
| WO2022206679A1 (en) | Image processing method and apparatus, computer device and storage medium | |
| CN116524026A (en) | Dynamic vision SLAM method based on frequency domain and semantics | |
| CN113723432B (en) | Intelligent identification and positioning tracking method and system based on deep learning | |
| Al-Harasis et al. | On the design and implementation of a dual fisheye camera-based surveillance vision system | |
| Li | Super-resolution for synthetic zooming |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |