CN110322477B

CN110322477B - Feature point observation window setting method, tracking method, device, equipment and medium

Info

Publication number: CN110322477B
Application number: CN201910494926.3A
Authority: CN
Inventors: 罗汉杰
Original assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd
Current assignee: Guangzhou Shiyuan Electronics Thecnology Co Ltd
Priority date: 2019-06-10
Filing date: 2019-06-10
Publication date: 2022-01-04
Anticipated expiration: 2039-06-10
Also published as: CN110322477A

Abstract

The invention relates to a method, a tracking method, a device, equipment and a medium for setting a characteristic point observation window based on an image pyramid, wherein the method comprises the steps of determining the initial size of the observation window of a current layer image, acquiring the tracking convergence state obtained by tracking the characteristic point in a previous layer image, acquiring a preset size adjusting step length, and setting the size of the observation window of the current layer image by using the initial size and the size adjusting step length based on the tracking convergence state, the scheme can reasonably set the size of the observation window of the current layer image by combining the tracking convergence state of the characteristic point tracked by the previous layer image, realizes the dynamic setting of the observation window of each layer image, enables the size of the observation window to be flexibly changed, can improve the flexibility of tracking the characteristic point based on the flexibly changed observation window, and dynamically sets the observation window according to the tracking convergence state, the method is also beneficial to reducing the tracking time of the characteristic points while ensuring the accurate tracking of the characteristic points.

Description

Feature point observation window setting method, tracking method, device, equipment and medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for setting a feature point observation window based on an image pyramid, a method and an apparatus for tracking a feature point based on an image pyramid, a computer device, and a computer-readable storage medium.

Background

In a computer vision system, feature texture matching belongs to an important image processing task, and the feature texture matching refers to extracting some feature textures, called feature points, from multiple images and then matching the feature points.

In the conventional technology, the moving speed and direction of corresponding feature points are usually calculated by detecting the conversion rate of the pixel point intensity under an observation window with a fixed size along with time, so that the feature points are tracked. However, in order to accurately find the position of the matching point, the technique usually employs a large observation window to track the feature point, and as the observation window increases, the amount of computation increases greatly, and the efficiency of tracking the feature point becomes low, so that the flexibility of tracking the feature point is low, and it is difficult to achieve both accuracy and efficiency of tracking the feature point.

Disclosure of Invention

Based on this, it is necessary to provide a feature point observation window setting method based on an image pyramid, a feature point observation window setting device based on an image pyramid, a feature point tracking method based on an image pyramid, a feature point tracking device based on an image pyramid, a computer device, and a computer-readable storage medium, for a technical problem that the flexibility of tracking feature points in the conventional technology is low.

A method for setting a characteristic point observation window based on an image pyramid comprises the following steps:

determining the initial size of a viewing window of a current layer image;

acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image;

acquiring a preset size adjusting step length;

setting the size of the observation window using the initial size and a size adjustment step size based on the tracking convergence state.

A feature point tracking method based on an image pyramid comprises the following steps:

respectively establishing an image pyramid for the original image and the target image; the image pyramid comprises a multi-layered image;

setting the sizes of observation windows of each layer of original images and each layer of target images based on the above method for setting the characteristic point observation windows based on the image pyramid;

tracking the characteristic points in each layer of image by using observation windows with corresponding sizes; and the images of all layers comprise original images of all layers and target images of all layers.

An image pyramid-based feature point observation window setting device, comprising:

the initial size determining module is used for determining the initial size of a viewing window of the current layer image;

the convergence state acquisition module is used for acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image;

the adjusting step length obtaining module is used for obtaining a preset size adjusting step length;

and the window size setting module is used for setting the size of the observation window by using the initial size and the size adjusting step length based on the tracking convergence state.

An image pyramid-based feature point tracking apparatus, comprising:

the establishing module is used for respectively establishing an image pyramid for the original image and the target image; the image pyramid comprises a multi-layered image;

the setting module is used for setting the sizes of the observation windows of each layer of original images and each layer of target images based on the characteristic point observation window setting method based on the image pyramid;

the tracking module is used for tracking the characteristic points in each layer of image by using the observation windows with corresponding sizes; and the images of all layers comprise original images of all layers and target images of all layers.

A computer device comprising a processor and a memory, the memory storing a computer program that when executed by the processor performs the steps of:

determining the initial size of a viewing window of a current layer image; acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image; acquiring a preset size adjusting step length; setting the size of the observation window using the initial size and a size adjustment step size based on the tracking convergence state.

respectively establishing an image pyramid for the original image and the target image; the image pyramid comprises a multi-layered image; setting the sizes of observation windows of each layer of original images and each layer of target images based on the above method for setting the characteristic point observation windows based on the image pyramid; tracking the characteristic points in each layer of image by using observation windows with corresponding sizes; and the images of all layers comprise original images of all layers and target images of all layers.

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

The above setting method, tracking method, apparatus, computer device and storage medium for feature point observation window based on image pyramid, determining the initial size of observation window of current layer image, obtaining tracking convergence state obtained by tracking feature point in previous layer image, obtaining preset size adjustment step length, then setting the size of observation window of current layer image by using initial size and size adjustment step length based on the tracking convergence state, the scheme can combine the tracking convergence state of previous layer image to reasonably set the size of observation window of current layer image, thereby realizing dynamic setting of observation window of each layer image of image pyramid, making the size of observation window flexibly change, based on flexibly changed observation window capable of improving flexibility of tracking feature point, and the observation window is dynamically set according to the tracking convergence state, so that the characteristic point tracking time is reduced while the accurate tracking of the characteristic point is ensured.

Drawings

FIG. 1 is a diagram of an application scenario of a feature point observation window setting method based on an image pyramid in an embodiment;

FIG. 2 is a flowchart illustrating a method for setting a feature point observation window based on an image pyramid in one embodiment;

FIG. 3 is a schematic diagram of an image pyramid in one embodiment;

FIG. 4 is a schematic view of a viewing window in one embodiment;

FIG. 5 is a schematic illustration of an optical flow vector in one embodiment;

FIG. 6 is a flowchart illustrating an exemplary method for feature point tracking based on an image pyramid;

FIG. 7 is a comparison of the effects of tracking feature points in one embodiment;

FIG. 8 is a graph comparing the effects of tracking feature points in one embodiment;

FIG. 9 is a block diagram of an exemplary image pyramid-based feature point observation window setting apparatus;

FIG. 10 is a block diagram of an embodiment of an image pyramid-based feature point tracking apparatus;

FIG. 11 is a diagram illustrating an internal structure of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

It should be noted that the term "first \ second" referred to in the embodiments of the present invention only distinguishes similar objects, and does not represent a specific ordering for the objects, and it should be understood that "first \ second" may exchange a specific order or sequence when allowed. It should be understood that "first \ second" distinct objects may be interchanged under appropriate circumstances such that embodiments of the invention described herein may be practiced in sequences other than those illustrated or described herein.

The method for setting the characteristic point observation window based on the image pyramid provided by the invention can be applied to an application scene shown in fig. 1, fig. 1 is an application scene diagram of the method for setting the characteristic point observation window based on the image pyramid in one embodiment, the application scene includes a computer device 100, the computer device 100 can be a terminal or a server with image processing capability, wherein the terminal can include various personal computers, notebook computers, tablet computers and the like, and the server can be implemented by an independent server or a server cluster consisting of a plurality of servers.

The computer device 100 can track and match feature points in a plurality of images, generally speaking, when tracking the feature points, the feature points can be tracked by calculating the moving speed and direction of the feature points through the time change rate of the gray scale of the feature points of the images under the observation window with a fixed size, and the traditional technology adopts the observation window with the fixed size to track the feature points, as shown in fig. 1, the computer device 100 can track the feature points by using an image pyramid, wherein, the feature points K are the corner points of the target 200 on the images, when tracking the feature points K by using the image pyramid, the size of the observation window W1 adopted in the image I1 of the current layer and the size of the observation window W2 adopted in the image I2 of the previous layer are the same, and in order to track the feature points in the images of the layers with higher accuracy, the conventional technique generally sets the size of the observation window to be large, but as the size of the observation window increases, the amount of computation of the computer apparatus 100 in tracking the feature points increases greatly.

The invention provides a method for setting a characteristic point observation window of an image pyramid, which comprises the steps that computer equipment 100 determines the initial size of an observation window of a current layer image, acquires a preset size adjusting step length and acquires a tracking convergence state obtained by tracking characteristic points in a previous layer image, finally, the computer equipment 100 sets the size of the observation window of the current layer image by utilizing the initial size and the size adjusting step length based on the tracking convergence state, reasonably sets the size of the observation window of the current layer image by combining the tracking convergence state of the characteristic points tracked by the previous layer image, realizes the dynamic setting of the observation window of each layer image of the image pyramid, enables the size of the observation window to be flexibly changed, can improve the flexibility of tracking the characteristic points based on the flexibly changed observation window, dynamically sets the observation window according to the tracking convergence state, the method is also beneficial to reducing the tracking time of the characteristic points while ensuring the accurate tracking of the characteristic points.

In an embodiment, an image pyramid-based feature point observation window setting method is provided, referring to fig. 2, fig. 2 is a flowchart illustrating an image pyramid-based feature point observation window setting method in an embodiment, which may be implemented based on the computer device 100 shown in fig. 1, where the image pyramid-based feature point observation window setting method may include the following steps:

step S101, determining an initial size of a viewing window of a current layer image.

In tracking the feature points of the image, the feature points may be tracked based on the image pyramid. As shown in fig. 3, fig. 3 is a schematic diagram of an image pyramid in an embodiment, and the image pyramiding generally includes two steps, namely, firstly, performing low-pass filtering on an image for smoothing, and then performing 1/2 sampling processing on pixel points of the image in horizontal and vertical directions, so as to obtain a series of scaled-down images. Let L denote the hierarchy of the image pyramid, and as shown in fig. 3, when L is 0, it is the original image, and when moving to the upper layer of the image pyramid, the size and resolution of the image are reduced, and the accompanying details are reduced. When tracking the feature points based on the image pyramid, generally, the target feature points are tracked from the top layer (L ═ 3), a rough tracking result is obtained first, then the tracking result is used as the initial point of the next layer for tracking, and iteration is continued until the L ═ 0 layer is reached, so that the target feature points are used as an analysis strategy from rough to fine.

In this step, the current layer image is an image corresponding to the current hierarchy in which the feature points are being tracked in the image pyramid, and for example, if the feature points are being tracked in the hierarchy of L ═ 2, the current layer image is an image corresponding to the hierarchy of L ═ 2. When the feature points of the image are tracked by using the image pyramid, each level of the image pyramid includes two images, one is an original image, and the other is a target image. The original image is an image where the feature points are located, the target image is an image where the matching points are located, the matching points are points matched with the feature points, and the feature points of the original image are tracked, namely, the positions of the matching points are found in the target image according to the related information of the feature points of the original image. In one embodiment, the original image and the target image may be images of two adjacent frames in the video image. When the feature points of the image are tracked by using the image pyramid, the image pyramid can be respectively established for the original image and the target image, and the image of each level comprises the images corresponding to the original image and the target image.

In this step, the current layer image may include a current layer original image and a current layer target image, and when the current layer image tracks the feature points, an initial size of an observation window of the current layer image is determined first. For the initial size of the observation window, the initial size may be a preset fixed value, and each layer of image uses the same initial size, or each layer of image uses different initial sizes, such as the observation window with size a for the top layer image, the observation window with size B for the next layer of the top layer image, and so on. For the shape of the observation window, for the convenience of calculation, a square observation window can be adopted to track the characteristic point, for the square observation window, the size of the size can be measured by the length of the side length, the longer the side length is, the larger the size of the observation window is, in the step, the observation window with the same size is adopted in the current layer original image and the current layer target image to track the characteristic point, and the main function of the observation window in the characteristic point tracking process is to calculate the moving speed and the moving direction of the characteristic point by detecting the information such as the change rate of the intensity of the characteristic point in the observation window along with the time, so that the characteristic point is tracked.

Specifically, the following description is made with reference to fig. 4, where fig. 4 is a schematic diagram of an observation window in an embodiment, fig. 4 shows an original image 310 and a target image 320, the original image 310 has a feature point K1, the feature point K1 is a corner point of a target object 31 on the original image 310, the target image 320 has a target object 32, the target object 32 is corresponding to the target object 31 and can be regarded as a same object in two adjacent video images moving from the position of the target object 31 to the position of the target object 32, and the target image 320 has a feature point K2, the feature point K2 is matched with the feature point K1, that is, the feature point K2 is a matching point, and the feature point K1 of the original image 310 is tracked, that is, the position of the feature point K2 is found in the target image 320. The observation window W may be used to track the feature point K1 in the original image 310 and the target image 320 and find the position of the feature point K2, and specifically, the position offset a of the feature point K1 in the target image 320 may be calculated based on the observation window W by a feature point tracking method such as an optical flow method, and then the position of the feature point K2 in the original image 310 may be tracked to the position of the feature point K2 in the target image 320 according to the position offset a and the position of the feature point K1.

And step S102, acquiring a tracking convergence state obtained by tracking the characteristic point in the previous layer image.

In this step, the previous-layer image is the previous-layer image of the current-layer image, and in the case where the current-layer image is an image corresponding to L ═ 2 level, as shown in fig. 3, the previous-layer image is the top-layer image (i.e., an image corresponding to L ═ 3). The method mainly comprises the steps of obtaining a tracking convergence state obtained by tracking the feature points in the previous layer of image, obtaining a corresponding tracking result when tracking the feature points in each layer of image of the image pyramid, wherein the tracking convergence state is whether the feature points are converged when searching the feature points in the previous layer of image, namely whether similar textures corresponding to the observation windows of the feature points can be found in the previous layer of image. In this step, the tracking convergence state may include convergence and non-convergence, where the tracking convergence state is that convergence usually can find the position of the matching point within a certain time or iteration number, and non-convergence usually cannot find the position of the matching point within a set time or iteration number, or find the matching point within a set time or iteration number, but the position of the matching point is offset from the position of the feature point too much.

Step S103, a preset size adjustment step is obtained.

In this step, the size adjustment step is used to adjust the size of the observation window of the current layer image, and the observation window of the current layer image may be increased or decreased by using the size adjustment step, where the size adjustment step may be a preset fixed value, that is, the same size adjustment step is used for each layer image to adjust the size of the observation window.

In one embodiment, before obtaining a preset size adjustment step, the number of pyramid layers of the image pyramid is determined, then the maximum size and the minimum size of the observation window are obtained, and the size adjustment step is set together with the maximum size, the minimum size and the number of pyramid layers.

Specifically, let L_mRepresenting a given number of pyramid layers, the maximum size of the viewing window being W_maxMinimum dimension is W_minThe size adjustment step can then be expressed as:

the present embodiment employs a fixed step size to adjust the size of the observation window, which makes the computer device 100 more efficient in tracking the feature points.

For setting the maximum size and the minimum size of the observation window, the pyramid layer number is L_mFor example, the image pyramid has the resolution of the top image and the bottom image

So if the size of the viewing window is W and the width and height are W and h, respectively, then in the top layer image the viewing window comprises a substantial range of image content

And

when the observation window is large enough to make the observation window larger than the movement range of the characteristic point, let w be h be 21, L_mThe observation window can maximally cover the range 168 × 168 of the image, that is, if the displacement of the feature point is less than 84 pixels (half of the maximum range), the observation window can be observed. Therefore, when the maximum size of the observation window is set, the maximum size can be determined based on the average displacement length of the feature point, which is set as

It is necessary to set a suitable pyramid level (typically 3) and maximum size W of the viewing window_maxSo that

The minimum size of the observation window should be small enough to minimize the computation time, but if the observation window is too small, it is easily interfered by noise, and generally, the minimum size W of the observation window can be set_minSet to 9 x 9.

And step S104, setting the size of the observation window by using the initial size and the size adjusting step length based on the tracking convergence state.

The step is mainly to set the size of the observation window of the current layer image through the initial size and the size adjusting step length according to the tracking convergence state of the previous layer image.

Specifically, if the feature point is not converged as a result of convergence of the previous image, a larger size may be set for the observation window by using the initial size and the resizing step, where the larger size is relative to the observation window of the previous image, that is, the size of the observation window of the previous image is assumed to be W₁Then a size larger than W can be set for the current layer image using the initial size and the size adjustment step size₁When the previous image pair does not converge on the feature point tracking, the observation window of (2) is increased to increase the probability of finding a matching point by increasing the observation window. Correspondingly, if the tracking convergence result obtained by tracking the feature point of the previous layer image is convergence, the initial size and the size adjusting step length can be used for setting a size smaller than W for the current layer image₁The observation window of (2) reduces the computation amount of tracking the feature point by the current layer image, so that the size of the observation window of the current layer image can be flexibly adjusted based on the tracking convergence state of the feature point tracked by the previous layer image. After the size of the observation window of the current-layer image is set, in one embodiment, the feature points may be tracked in the current-layer image by using the observation window based on a feature point tracking method such as an optical flow method. It should be noted that, if the current layer image is the top layer image, since the top layer image does not have the previous layer image, for the top layer image, a preset observation window with the maximum size may be used to track the feature points, so as to ensure that the positions of the matching points can be found in the top layer image, and then the size of the observation window of the corresponding layer image is adjusted in the lower layer image to balance the efficiency and accuracy of tracking the feature points.

The method for setting the characteristic point observation window based on the image pyramid determines the initial size of the observation window of the current layer image, acquires the tracking convergence state obtained by tracking the characteristic point in the previous layer image, acquires the preset size adjusting step length, and then sets the size of the observation window of the current layer image by using the initial size and the size adjusting step length based on the tracking convergence state, and the scheme can reasonably set the size of the observation window of the current layer image by combining the tracking convergence state of the previous layer image for tracking the characteristic point, thereby realizing the dynamic setting of the observation window of each layer image of the image pyramid, enabling the size of the observation window to be flexibly changed, improving the flexibility of tracking the characteristic point based on the flexibly changed observation window, and dynamically setting the observation window according to the tracking convergence state, the method is also beneficial to reducing the tracking time of the characteristic points while ensuring the accurate tracking of the characteristic points.

The characteristic points can be tracked in each layer of image of the image pyramid by adopting an optical flow method, the traditional KLT optical flow method is based on the assumption that the gray level is unchanged, namely the same point in the space, the gray level in different images is fixed, and under the assumption, the moving speed and the moving direction of the corresponding characteristic points are obtained by detecting the change rate of the pixel point intensity in an observation window with a set size along with the time, so that the tracking of the characteristic points is realized.

The feature points in the previous image may be tracked by an optical flow method, and in an embodiment, the tracking convergence state obtained by tracking the feature points in the previous image by the previous image may be a tracking convergence state obtained by iteratively tracking the feature points in the previous image by the optical flow method.

The optical flow method can be used for carrying out multiple iterative operations when the feature points in the previous layer image are tracked, and the result obtained by each iterative operation is the optical flow vector p of the iteration^kAnd k represents the iteration times, an optical flow vector is obtained by each iteration operation, and the sum of the optical flow vectors obtained by the iteration operations represents the position offset of the matching point and the feature point. The optical flow vectors are described with reference to FIG. 5, where FIG. 5 is a schematic diagram of the optical flow vectors in one embodimentSetting up in the upper layer image 400, using optical flow method to carry out 3 times of iterative tracking to the feature point, the optical flow vector obtained by each iterative tracking is p¹、p²And p³The sum of the three optical flow vectors indicates the amount of positional displacement between the matching point obtained from the previous layer image 400 and the feature point, and the matching point obtained from the previous layer image 400 can be regarded as x 4.

Further, in an embodiment, the step of obtaining a tracking convergence state obtained by tracking the feature point in the previous image may specifically include: determining iteration times for performing iterative tracking on the feature points in the previous layer of image; acquiring an optical flow vector of the previous layer image under the iteration times; and acquiring a tracking convergence state according to the optical flow vector.

Before setting the size of the observation window of the current layer image, a tracking convergence state of the previous layer image for tracking the feature point needs to be acquired, in this embodiment, when the feature point is tracked by using an optical flow method based on the previous layer image, the iteration number of iteratively tracking the feature point in the previous layer image is determined, where the iteration number is generally 30, and then an optical flow vector of the previous layer image at the iteration number may be acquired, where the iteration number refers to the maximum iteration number, and if the iteration number is set to 30, the optical flow vector obtained when the iteration number is 30 is acquired, and finally the tracking convergence state is acquired according to the optical flow vector. Because the optical flow vector has a certain size, and the optical flow vector corresponding to the maximum iteration number can reflect whether the feature points of the previous layer image are converged (namely whether the feature points can be found) under the specified iteration number when the feature points of the previous layer image are tracked, the tracking convergence state of the previous layer image can be accurately judged according to the optical flow vector corresponding to the maximum iteration number, if the feature points are not converged, the size of the observation window needs to be increased, so that the position of the matching point can be accurately found in the current layer image, and otherwise, the size of the observation window can be reduced to accelerate the operation speed.

In an embodiment, the step of acquiring the tracking convergence state according to the optical flow vector may specifically include: and acquiring a modulus of the optical flow vector under the iteration times, and then judging the tracking convergence state of the feature point tracked by the previous layer image according to the modulus of the optical flow vector.

In this embodiment, the optical flow vector p^kThe optical flow vector of the feature point under the k iteration number is represented, namely the position offset of the matching point found under the corresponding iteration number and the feature point, | p^kI denotes the optical flow vector p^kI.e. the length of the position offset, in the normal iterative process, | p with each iteration proceeding^kThe value of | p will become smaller until the correct matching position is reached (i.e. the correct matching point is found), i | | p^kThe value of | becomes 0, but becomes 0 is | p in an ideal state^kThe value that can be achieved, in practical cases, | | p^kI will generally not be 0. Therefore, in order to accurately determine the tracking convergence state of the previous layer image for tracking the feature point, the optical flow vector | | p may be set^kIn a set threshold (e.g. 1 e)^-4) Comparing, when | | p^kIf the value of | is smaller than the threshold, the tracking convergence state may be considered as convergence, and if the maximum number of iterations K is reached, the optical flow vector p^KIf the tracking state is still not less than the threshold, the tracking state is judged to be non-convergence, and the tracking convergence state of the image feature points of the previous layer can be quickly and effectively obtained by adopting the mode.

After the tracking convergence state of the previous layer image to the feature point is obtained, the size of the observation window of the current layer image can be adjusted according to the two conditions that the tracking convergence state is convergence or non-convergence.

In one embodiment, the step of setting the size of the observation window using the initial size and the size adjustment step size based on the tracking convergence state may include: and if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of the observation window.

In this embodiment, the size of the observation window of the current layer image is obtained by subtracting the initial size and the size adjustment step length, and the size of the observation window of the current layer image can be reduced when the tracking convergence state of the previous layer image to the feature point is convergence.

In another embodiment, the step of setting the size of the observation window using the initial size and the size adjustment step size based on the tracking convergence state may include: and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window.

In this embodiment, the initial size and the size adjustment step are summed to obtain the size of the observation window of the current layer image, and when the tracking convergence state of the previous layer image on the feature point is not converged, the probability of finding the matching point in the current layer image can be increased by increasing the size of the observation window.

Further, in the process of summing the initial size and the size adjustment step size to obtain the size of the observation window, if the size of the observation window obtained through the summing is smaller than the maximum size, the size of the observation window is set as the maximum size, and the observation window with the maximum size is used for tracking the feature point in the current layer image. This is mainly to consider that even if the size of the observation window is increased according to the step length, it is difficult to make the tracking of the feature point converge only once, sometimes it is necessary to increase the window many times to make the tracking converge, and too much adjustment will reduce the tracking efficiency, so the size of the observation window of the current layer image can be adjusted to the maximum size once to track the feature point, and the efficiency will be higher by adopting this method, and the feature point can be tracked more accurately on the current layer image.

In an embodiment, an image pyramid-based feature point tracking method is further provided, referring to fig. 6, fig. 6 is a schematic flowchart of the image pyramid-based feature point tracking method in an embodiment, where the method may include the following steps:

step S401, respectively creating an image pyramid for the original image and the target image.

In this step, before tracking the feature points, an image pyramid may be established for an original image and a target image, where the original image is an image where the feature points are located, the target image is an image where the matching points are located, the matching points are points matched with the feature points, the image pyramid includes multiple layers of images, and each layer of images of the image pyramid may include each layer of the original image and each layer of the target image. Taking an image pyramid with 3 levels as an example, the 2 nd-level image may include a 2 nd-level original image and a 2 nd-level target image, and tracking the feature points in the 2 nd-level image means finding the positions of matching points matching the feature points in the 2 nd-level target image based on the related information of the feature points on the 2 nd-level original image.

Step S402, setting the sizes of the observation windows of each layer of original image and each layer of target image based on the feature point observation window setting method based on the image pyramid as described in any one of the above embodiments.

In the step, the sizes of the observation windows of each layer of the original image and each layer of the target image in the image pyramid are set by using the characteristic point observation window setting method based on the image pyramid.

In one embodiment, if the current layer image is the top layer image, the sizes of the observation windows of the current layer original image and the current layer target image are both set to be the maximum size, that is, the feature points are tracked on the top layer image by the observation window with the maximum size, which is beneficial to accurately locating the feature points on the top layer image.

And if the current layer image is not the top layer image, setting the size of the observation window of the previous layer image as the initial size of the observation window of the current layer image, namely, adopting the size of the observation window adopted by the previous layer image when tracking the characteristic point as the initial size of the observation window of the current layer image, and adjusting the size of the observation window of the current layer image based on the initial size. Then, a preset size adjustment step length is obtained, and a tracking convergence state obtained by tracking the feature point in the previous layer image is obtained.

If the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of an observation window of the current layer image; and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window of the current layer image.

Step S403, tracking the feature points in each layer of image by using observation windows with corresponding sizes; each layer of image comprises each layer of original image and each layer of target image.

In the step, the characteristic points are tracked in each layer of image of the image pyramid by using the observation windows with corresponding sizes, so that the sizes of the observation windows adopted by each layer of image of the image pyramid can be flexibly adjusted based on the tracking convergence state of the previous layer of image for tracking the characteristic points, and the tracking efficiency and accuracy are balanced.

In an embodiment, the step of tracking the feature points in each layer image by using the observation windows with corresponding sizes specifically includes:

in the current layer original image, a first gray value and a first gray gradient value of the feature point are calculated by using the observation window with the corresponding size, and a Hessian matrix of the feature point is calculated according to the first gray gradient value.

In the step, given the original image I and the position x of a feature point on the original image I, the position x' of a matching point is found in the target image J. Suppose the position of the initial matching point on the target image J is x'_initThe minimum size of the observation window is W_minMaximum dimension of W_max. An image pyramid { I } can be established for the original image I and the target image J respectively^L}_L＝0...LmAnd { J^L}_L＝0...LmThe image pyramid may comprise a plurality of layers, L_mRepresenting a given number of pyramid levels, typically 3, the viewing window adjustment step size is calculated

Let W { (u, v) |0 < u < W,0 < v < h }, where W is the width of the observation window, h is the height of the observation window, and u and v represent the coordinates of the pixel points within the observation window.

In the current layer original image, the specific steps of calculating the first gray value and the first gray gradient value of the feature point, and calculating the hessian matrix of the feature point according to the first gray gradient value may include:

firstly, obtaining an original image I of a current layer^LFirst gray value I of the characteristic point^L(x) Then based on the first gray value I^L(x) Calculating the characteristic point of the original image I in the current layer^LGradient matrix of up-horizontal coordinate direction X and vertical coordinate direction Y

And

the gradient matrix in both the abscissa direction and the ordinate direction may be used as the first grayscale gradient value of the feature point.

Then, based on the observation window W, the original image I of the current layer is calculated^LAt position x^L＝[p_x p_y]^THessian matrix H (p)_xp_y)：

Wherein (u, v) represents the coordinates of the pixel points in the observation window W, H (p)_x p_y) Is a 2 x 2 matrix, which represents the original image I at the current layer^LIn x^L＝[p_x p_y]^TThe image gray scale second derivative of position.

Then, in the current layer target image, calculating a second gray value of the initial matching point by using an observation window with a corresponding size; the initial matching points are pixel points preset on a target image of the current layer; performing difference on the first gray value and the second gray value to obtain a gray deviation value; and calculating an optical flow vector of the current layer image according to the gray level deviation value, the first gray level gradient value and the Hessian matrix.

Specifically, the above steps may adopt an optical flow method, and calculate an optical flow vector of the current layer image in an iterative manner. Wherein a variable K can be set from 1 to K for controlling the number of iterations of the following steps, typically set to 30, iterating the following operations:

target image J at current layer^LUsing the observation window W to calculate a second gray scale value of an initial matching point, which is preset in the target image J on the current layer^LThe initial matching point is at the current layer target image J^LIs shown as

Wherein,

and

to a preset matching offset position

Wherein

Can be preset to [ 00 ]]^TTo do so

And

iterating the parameters for the position in the k-1 st iteration process, wherein the iteration parameters are initialized to gamma⁰＝[0 0]^T. Based on the observation window W, a second gray value of the initial matching point can be calculated as J^L(q′_x+u,q′_y+ v), a gray offset value of I may then be calculated^L(p_x+u,p_y+v)-J^L(q′_x+u,q′_y+ v) of formula (I)^L(p_x+u,p_y+ v) represents the first gray value. An optical flow vector p of the current layer image may then be calculated^k：

Wherein the optical flow vector p^kThe tracking offset of the feature point in this iteration is shown, and the optical flow vector p^kGray scale deviation information and gray scale gradient information are included.

Finally, acquiring the position of the matching point on the current layer target image based on the optical flow vector and the position of the initial matching point on the current layer target image; and the matching points are pixel points matched with the characteristic points on the target image of the current layer.

In the above step, the optical flow vector p is obtained^kThereafter, the position iteration parameter gamma is updated^k＝γ^k-1-p^kIf p | |^kIf is less than a given threshold (e.g., 1 e)^-4) Then the iteration is said to have converged and the iteration process can be exited if p^kIf | is greater than a given threshold, the iteration continues. If the iteration number K is larger than K, it indicates that there is no convergence (i.e. no feature point is found) under the specified iteration number, and at this time, the size W of the observation window may be increased to W + step, and if the increased observation window is not larger than W_maxThen tracking is performed with the enlarged viewing window. If converged at a specified number of iterations, the optical flow vector p may be based on^kObtaining the final position iteration parameter gamma^k＝γ^k-1-p^kAnd it can be determined that the final matching offset in the current layer image (e.g., the L-th pyramid) is: d^L＝γ^kThen, based on the final matching offset and in combination with the position of the initial matching point on the current-layer target image, the positions of the matching points on the current-layer target image can be added. And the position of the matching point on the current layer target image can be used for initializing the matching offset position of the next layer pyramid: g^L-1＝2(g^L+d^L) Then setting L as L-1, and re-depositing gold on next layerAnd searching the position of the matching point in the character tower image. Finally, in the L ═ 0 layer image, the final position of the matching point of the feature point x in the target image J can be expressed as x ═ x'_init+g⁰+d⁰。

The effect of the above-mentioned feature point tracking method based on the image pyramid can refer to fig. 7 and 8. As shown in fig. 7, fig. 7 is an effect comparison diagram of tracking feature points in an embodiment, an original image 701 and a target image 702 correspond to an effect diagram of tracking feature points by a conventional optical flow method, the original image 703 and the target image 704 correspond to an effect diagram of a feature point tracking method according to an embodiment of the present invention, a midpoint of a rectangular square in the diagram represents positions of the feature points and matching points, and a size of the rectangle is a size of a final observation window.

As shown in fig. 8, fig. 8 is a comparison diagram of another effect of tracking feature points in an embodiment, where the effect diagram shows a time cost result of performing feature point matching on a test set by using a conventional optical flow method (raw optical flow method) and a feature point tracking method (variable window strategy optical flow method) provided in an embodiment of the present invention, and an ordinate represents the time cost, and it can be seen that the feature point tracking method provided in the embodiment of the present invention has a significant advantage in terms of computational efficiency.

It can be seen that, the feature point tracking method based on the image pyramid provided in the embodiment of the present invention can reasonably set the size of the observation window of the current layer of image in combination with the tracking convergence state of the previous layer of image for tracking the feature point, thereby implementing dynamic setting of the observation window of each layer of image of the image pyramid, enabling the size of the observation window to be flexibly changed, improving the flexibility of tracking the feature point based on the observation window that is flexibly changed, dynamically setting the observation window according to the tracking convergence state, and also facilitating to reduce the tracking time of the feature point while ensuring accurate tracking of the feature point.

In an embodiment, an image pyramid-based feature point observation window setting apparatus is provided, and referring to fig. 9, fig. 9 is a block diagram of a structure of an image pyramid-based feature point observation window setting apparatus in an embodiment, where the apparatus may include:

an initial size determining module 101, configured to determine an initial size of a viewing window of a current layer image;

a convergence state obtaining module 102, configured to obtain a tracking convergence state obtained by tracking the feature point in the previous layer image;

an adjustment step length obtaining module 103, configured to obtain a preset size adjustment step length;

and a window size setting module 104, configured to set a size of the observation window using the initial size and the size adjustment step size based on the tracking convergence state.

In one embodiment, the tracking convergence state is a tracking convergence state obtained by iteratively tracking the feature points in the upper layer image by using an optical flow method; the convergence status obtaining module 102 is further configured to: determining iteration times for performing iterative tracking on the feature points in the previous layer of image; acquiring an optical flow vector of the previous layer image under the iteration times; and acquiring a tracking convergence state according to the optical flow vector.

In one embodiment, the convergence status acquisition module 102 is further configured to: acquiring a modulus of an optical flow vector; if the modulus of the optical flow vector is greater than or equal to the threshold value, judging that the tracking convergence state is not convergence; and if the modulus of the optical flow vector is smaller than the threshold value, judging the tracking convergence state as convergence.

In one embodiment, the window size setting module 104 is further configured to: and if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of the observation window.

In one embodiment, the window size setting module 104 is further configured to: and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window.

In one embodiment, the window size setting module 104 is further configured to: and if the size of the observation window obtained by the summation processing is smaller than the maximum size, setting the size of the observation window as the maximum size.

In an embodiment, the apparatus for setting a feature point observation window based on an image pyramid may further include: an adjustment step setting unit configured to: determining pyramid layer number of an image pyramid; acquiring the maximum size and the minimum size of an observation window; and setting a size adjusting step length according to the maximum size, the minimum size and the pyramid layer number.

In an embodiment, the apparatus for setting a feature point observation window based on an image pyramid may further include: and the characteristic point tracking unit is used for tracking the characteristic points in the current layer image by utilizing the observation window.

The image pyramid-based feature point observation window setting device and the image pyramid-based feature point observation window setting method of the present invention correspond to each other one by one, and specific limitations on the image pyramid-based feature point observation window setting device can be referred to the above limitations on the image pyramid-based feature point observation window setting method, and technical features and advantages thereof explained in the above embodiment of the image pyramid-based feature point observation window setting method are all applicable to the embodiment of the image pyramid-based feature point observation window setting device, and are not described herein again. All or part of the modules in the image pyramid-based feature point observation window setting device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In an embodiment, there is further provided an image pyramid-based feature point tracking apparatus, and referring to fig. 10, fig. 10 is a block diagram of a structure of the image pyramid-based feature point tracking apparatus in an embodiment, where the apparatus may include:

an establishing module 401, configured to respectively establish an image pyramid for an original image and a target image; the image pyramid comprises a multi-layer image;

a setting module 402, configured to set sizes of observation windows of each layer of original images and each layer of target images based on the feature point observation window setting method based on the image pyramid as described in any one of the above embodiments;

a tracking module 403, configured to track feature points in each layer of image by using observation windows with corresponding sizes; each layer of image comprises each layer of original image and each layer of target image.

In one embodiment, the setup module 402 is further configured to: if the current layer image is the top layer image, setting the size of an observation window of the current layer original image and the size of an observation window of the current layer target image as the maximum size; if the current layer image is not the top layer image, setting the size of the observation window of the previous layer image as the initial size of the observation window of the current layer image; acquiring a preset size adjusting step length; acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image; if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of an observation window of the current layer image; and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window of the current layer image.

In one embodiment, the tracking module 403 is further configured to: in the current layer original image, calculating a first gray value and a first gray gradient value of the feature point by using an observation window with a corresponding size, and calculating a Hessian matrix of the feature point according to the first gray gradient value; calculating a second gray value of the initial matching point by using an observation window with a corresponding size in the current layer target image; the initial matching points are pixel points preset on a target image of the current layer; performing difference on the first gray value and the second gray value to obtain a gray deviation value; calculating an optical flow vector of the current layer image according to the gray level deviation value, the first gray level gradient value and the Hessian matrix; acquiring the position of the matching point on the current layer target image based on the optical flow vector and the position of the initial matching point on the current layer target image; and the matching points are pixel points matched with the characteristic points on the target image of the current layer.

The image pyramid-based feature point tracking device and the image pyramid-based feature point tracking method of the present invention are in one-to-one correspondence, and for specific limitations of the image pyramid-based feature point tracking device, reference may be made to the above limitations on the image pyramid-based feature point tracking method, and the technical features and the advantageous effects thereof set forth in the above embodiments of the image pyramid-based feature point tracking method are applicable to the embodiments of the image pyramid-based feature point tracking device, and are not described herein again. All or part of the modules in the image pyramid-based feature point tracking device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, and the computer device may be a terminal or a server with image processing capability, and its internal structure diagram may be as shown in fig. 11, and fig. 11 is an internal structure diagram of the computer device in one embodiment. The computer device includes a processor and a memory connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The computer program is executed by a processor to realize a feature point observation window setting method based on an image pyramid and a feature point tracking method based on the image pyramid.

Those skilled in the art will appreciate that the architecture shown in fig. 11 is merely a block diagram of some of the structures associated with the inventive arrangements and is not intended to limit the computing devices to which the inventive arrangements may be applied, as a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, which includes a processor and a memory, the memory stores a computer program, and the processor implements the image pyramid based feature point observation window setting method and the image pyramid based feature point tracking method as described in any one of the above embodiments when executing the computer program.

In one embodiment, there is provided a computer device comprising a processor and a memory, the memory storing a computer program which when executed by the processor performs the steps of:

determining the initial size of a viewing window of a current layer image; acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image; acquiring a preset size adjusting step length; the size of the observation window is set using the initial size and the size adjustment step size based on the tracking convergence state.

In one embodiment, the processor, when executing the computer program, further performs the steps of:

determining iteration times for performing iterative tracking on the feature points in the previous layer of image; acquiring an optical flow vector of the previous layer image under the iteration times; acquiring a tracking convergence state according to the optical flow vector; the tracking convergence state is obtained by performing iterative tracking on the feature points in the upper-layer image by using an optical flow method.

acquiring a modulus of an optical flow vector; if the modulus of the optical flow vector is greater than or equal to the threshold value, judging that the tracking convergence state is not convergence; and if the modulus of the optical flow vector is smaller than the threshold value, judging the tracking convergence state as convergence.

and if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of the observation window.

and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window.

and if the size of the observation window obtained by the summation processing is smaller than the maximum size, setting the size of the observation window as the maximum size.

determining pyramid layer number of an image pyramid; acquiring the maximum size and the minimum size of an observation window; and setting a size adjusting step length according to the maximum size, the minimum size and the pyramid layer number.

and tracking the characteristic points in the current layer image by using the observation window.

respectively establishing an image pyramid for the original image and the target image; the image pyramid comprises a multi-layer image; setting the sizes of observation windows of each layer of original images and each layer of target images based on the feature point observation window setting method based on the image pyramid as described in any one of the above embodiments; tracking the characteristic points in each layer of image by using observation windows with corresponding sizes; each layer of image comprises each layer of original image and each layer of target image.

if the current layer image is the top layer image, setting the size of an observation window of the current layer original image and the size of an observation window of the current layer target image as the maximum size; if the current layer image is not the top layer image, setting the size of the observation window of the previous layer image as the initial size of the observation window of the current layer image; acquiring a preset size adjusting step length; acquiring a tracking convergence state obtained by tracking the feature points in the previous layer of image; if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of an observation window of the current layer image; and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window of the current layer image.

in the current layer original image, calculating a first gray value and a first gray gradient value of the feature point by using an observation window with a corresponding size, and calculating a Hessian matrix of the feature point according to the first gray gradient value; calculating a second gray value of the initial matching point by using an observation window with a corresponding size in the current layer target image; the initial matching points are pixel points preset on a target image of the current layer; performing difference on the first gray value and the second gray value to obtain a gray deviation value; calculating an optical flow vector of the current layer image according to the gray level deviation value, the first gray level gradient value and the Hessian matrix; acquiring the position of the matching point on the current layer target image based on the optical flow vector and the position of the initial matching point on the current layer target image; and the matching points are pixel points matched with the characteristic points on the target image of the current layer.

According to the computer equipment, the observation windows of the images of all layers of the image pyramid are dynamically set through the computer program running on the processor, so that the size of the observation windows can be flexibly changed, the flexibility of tracking the feature points can be improved based on the observation windows which are flexibly changed, the observation windows are dynamically set according to the tracking convergence state, and the characteristic point tracking time is reduced while accurate tracking of the feature points is ensured.

It will be understood by those skilled in the art that all or part of the processes for implementing the image pyramid-based feature point observation window setting method and the image pyramid-based feature point tracking method according to any of the above embodiments may be implemented by instructing associated hardware through a computer program, where the computer program may be stored in a non-volatile computer-readable storage medium, and when executed, the computer program may include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Accordingly, in an embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the program when executed by a processor implements the image pyramid based feature point observation window setting method and the image pyramid based feature point tracking method as described in any one of the above embodiments.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

In one embodiment, the computer program when executed by the processor further performs the steps of:

According to the computer-readable storage medium, the observation windows of the images of all layers of the image pyramid are dynamically set through the stored computer program, so that the sizes of the observation windows can be flexibly changed, the flexibility of tracking the feature points can be improved based on the observation windows which are flexibly changed, the observation windows are dynamically set according to the tracking convergence state, and the characteristic point tracking time is reduced while the accurate tracking of the feature points is ensured.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for setting a characteristic point observation window based on an image pyramid is characterized by comprising the following steps:

if the current layer image is a non-top layer image, taking the size of the observation window of the previous layer image as the initial size of the observation window of the current layer image;

acquiring a preset size adjusting step length;

setting the size of the observation window by using the initial size and the size adjustment step length based on the tracking convergence state;

wherein the setting the size of the observation window using the initial size and the size adjustment step size based on the tracking convergence state comprises:

when the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of the observation window;

when the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window;

further comprising:

and if the current layer image is the top layer image, setting the size of the observation window of the current layer image as the maximum size.

2. The method according to claim 1, wherein the tracking convergence state is a tracking convergence state obtained by iteratively tracking the feature point in the previous layer image by an optical flow method;

the step of obtaining a tracking convergence state obtained by tracking the feature point in the previous layer image includes:

determining the iteration number for performing iterative tracking on the feature points in the previous layer image;

acquiring an optical flow vector of the previous layer image under the iteration times;

and acquiring the tracking convergence state according to the optical flow vector.

3. The method of claim 2, wherein the step of obtaining the tracking convergence status according to the optical flow vector comprises:

obtaining a modulus of the optical flow vector;

if the modulus of the optical flow vector is larger than or equal to a threshold value, judging that the tracking convergence state is not convergence;

and if the modulus of the optical flow vector is smaller than the threshold value, judging the tracking convergence state as convergence.

4. The method of claim 1, wherein the step of summing the initial size and the size adjustment step size to obtain the size of the observation window comprises:

5. The method of claim 1, further comprising, prior to the step of obtaining the preset size adjustment step size:

determining pyramid layer number of an image pyramid;

acquiring the maximum size and the minimum size of the observation window;

and setting the size adjusting step length according to the maximum size, the minimum size and the pyramid layer number.

6. The method according to any one of claims 1 to 5, further comprising, after the step of setting the size of the observation window with the initial size and a size adjustment step size based on the tracking convergence state:

and tracking the characteristic points in the current layer image by utilizing the observation window.

7. A feature point tracking method based on an image pyramid is characterized by comprising the following steps:

setting the sizes of observation windows of each layer of original images and each layer of target images based on the image pyramid-based feature point observation window setting method according to any one of claims 1 to 6;

8. The method according to claim 7, wherein the step of setting the size of the observation window for each layer of the original image and each layer of the target image based on the image pyramid-based feature point observation window setting method according to any one of claims 1 to 6 comprises:

if the current layer image is the top layer image, setting the size of an observation window of the current layer original image and the size of an observation window of the current layer target image as the maximum size;

if the current layer image is not the top layer image, setting the size of the observation window of the previous layer image as the initial size of the observation window of the current layer image; acquiring a preset size adjusting step length; acquiring a tracking convergence state obtained by tracking the feature points in the previous layer image; if the tracking convergence state is convergence, performing difference processing on the initial size and the size adjusting step length to obtain the size of an observation window of the current layer image; and if the tracking convergence state is non-convergence, summing the initial size and the size adjusting step length to obtain the size of the observation window of the current layer image.

9. The method of claim 7, wherein the step of tracking the feature points in each layer image using a correspondingly sized viewing window comprises:

in the current layer original image, calculating a first gray value and a first gray gradient value of the feature point by using the observation window with the corresponding size, and calculating a Hessian matrix of the feature point according to the first gray gradient value;

in the current layer target image, calculating a second gray value of the initial matching point by using the observation window with the corresponding size; the initial matching points are pixel points preset on the target image of the current layer;

performing difference on the first gray value and the second gray value to obtain a gray deviation value;

calculating an optical flow vector of the current layer image according to the gray level deviation value, the first gray level gradient value and the Hessian matrix;

acquiring the position of a matching point on the current layer target image based on the optical flow vector and the position of the initial matching point on the current layer target image; and the matching points are pixel points matched with the characteristic points on the current layer target image.

10. An image pyramid-based feature point observation window setting device, comprising:

an initial size determining module, configured to, if a current layer image is a non-top layer image, take a size of an observation window of a previous layer image as an initial size of the observation window of the current layer image;

a window size setting module for setting the size of the observation window by using the initial size and the size adjustment step length based on the tracking convergence state;

and when the current layer image is the top layer image, the size of the observation window of the current layer image is the maximum size.

11. An image pyramid-based feature point tracking device, comprising:

a setting module, configured to set sizes of observation windows of each layer of the original image and each layer of the target image based on the image pyramid-based feature point observation window setting method according to any one of claims 1 to 6;

12. A computer device comprising a processor and a memory, said memory storing a computer program, characterized in that said processor, when executing said computer program, implements the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.