CN117291875A - Lens offset detection method, device, computer equipment, chip and medium - Google Patents

Abstract

The embodiment of the specification provides a lens offset detection method, a lens offset detection device, computer equipment, a chip and a medium. The offset detection method comprises the following steps: determining a specified extraction quantity, a reference key point set of reference frame images and a current frame image acquired by a lens; the reference key point set is obtained by extracting reference key points from a reference frame image based on a trained time invariant feature detector; extracting detection key points from the current frame image based on the specified extraction quantity by a time-invariant feature detector to obtain a detection key point set; the quantity of the detection key points in the detection key point set is equal to the specified extraction quantity; performing feature matching on the detection key point set and the reference key point set to obtain a matching result; if it is determined that the shot is not shifted based on the matching result, the current frame image is used as the reference frame image, and the above-described shift detection step for the shot is repeatedly performed, so that the accuracy of shift detection can be improved.

Description

Lens offset detection method, device, computer equipment, chip and medium

Technical Field

The embodiment of the present specification relates to the technical field of image processing, and in particular, to a method, a device, a computer device, a chip, and a medium for detecting offset of a lens.

Background

Video monitoring is widely applied to scenes such as urban roads, airports, stations, super business, workshops and the like, video monitoring is required to play roles, video capturing visual angles of cameras are required to be guaranteed to be correct, and reliable video output can be guaranteed due to the fact that the cameras are in unmanned inspection conditions for a long time, particularly outdoor cameras, the lenses can deviate under the influence of weather conditions such as storm, snow and the like, the system is required to timely detect the lens deviation and send an alarm to remind operation and maintenance personnel to carry out timely maintenance.

In the related art, whether the lens is shifted is generally detected by a difference method, a histogram matching method and a linear feature matching method, however, in the related art, the shift detection of the lens is difficult to reduce the influence of environmental change, the shift of the lens is easily caused by misjudgment, and the detection error is large.

Therefore, it is desirable to provide a lens shift detection method to improve the accuracy of shift detection.

Disclosure of Invention

In view of this, various embodiments of the present disclosure are directed to providing a lens shift detection method, apparatus, computer device, chip, and medium to improve the accuracy of shift detection.

The embodiment of the specification provides a method for detecting offset of a lens, wherein the lens corresponds to a reference frame image; the offset detection method comprises the following steps: determining a specified extraction quantity, a reference key point set of the reference frame image and a current frame image acquired by the lens; the reference key point set is obtained by extracting reference key points from the reference frame image based on a trained time invariant feature detector; extracting detection key points from the current frame image based on the specified extraction quantity by the time invariant feature detector to obtain a detection key point set; the number of the detection key points in the detection key point set is equal to the specified extraction number; performing feature matching on the detection key point set and the reference key point set to obtain a matching result; and if the lens is judged not to deviate based on the matching result, taking the current frame image as the reference frame image, and repeatedly executing the deviation detection step for the lens.

Further, the performing feature matching on the detection key point set and the reference key point set to obtain a matching result includes: determining a Euclidean distance set between any detection key point and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far according to the distance; determining a reference key point corresponding to the first Euclidean distance as a matching point of any detection key point under the condition that the ratio between the first Euclidean distance and the second Euclidean distance is smaller than a first set threshold value; wherein the value range of the first set threshold is 0.3-0.7; determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the matching number.

Further, the offset detection method further includes: judging that the lens is deviated under the condition that the matching quantity is smaller than or equal to a second set threshold value; wherein the value range of the second set threshold is 8-20.

Further, the offset detection method further includes: average calculation is carried out based on the first Euclidean distance of each matching point, and average offset distances are obtained; wherein the matching result further comprises an average offset distance.

Further, the offset detection method further includes: judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; wherein the value range of the third set threshold is 6-12.

Further, the offset detection method further includes: judging that the lens is not shifted under the condition that the matching number is larger than a second set threshold value and the average offset distance is smaller than a third set threshold value; wherein the value range of the second set threshold is 8-20, and the value range of the third set threshold is 6-12.

Further, the determining manner of the reference key point set includes: acquiring an initial image of a current frame acquired by the lens; performing scaling operation on the current frame initial image to obtain the current frame image with a specified size; if the current frame image is the first frame, taking the current frame image as the reference frame image; and extracting the reference key points of the reference frame image based on the time invariant feature detector to obtain the reference key point set.

Further, the determining method of the specified extraction quantity includes: determining the specified extraction quantity in response to a setting operation of the specified extraction quantity; or, acquiring a test frame image acquired by the lens, classifying the test frame image to obtain a target class of the environment monitored by the lens, and setting the appointed extraction quantity according to a preset extraction quantity range corresponding to the target class; wherein the preset extraction number range relates to a scene intensive situation.

The embodiment of the specification provides a shift detection device of a lens, wherein the lens corresponds to a reference frame image; the offset detection device includes: the acquisition module is used for determining the appointed extraction quantity, the reference key point set of the reference frame image and the current frame image acquired by the lens; the reference key point set is obtained by extracting reference key points from the reference frame image based on a trained time invariant feature detector; the extraction module is used for extracting detection key points from the current frame image based on the specified extraction quantity through the time-invariant feature detector to obtain a detection key point set; the number of the detection key points in the detection key point set is equal to the specified extraction number; the matching module is used for carrying out feature matching on the detection key point set and the reference key point set to obtain a matching result; and the judging module is used for repeatedly executing the offset detection step for the lens by taking the current frame image as the reference frame image if the lens is judged not to be offset based on the matching result.

Further, the matching module includes: the first matching module is used for determining a Euclidean distance set between any detection key point and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far according to the distance; the second matching module is used for determining a reference key point corresponding to the first Euclidean distance as a matching point of any detection key point under the condition that the ratio between the first Euclidean distance and the second Euclidean distance is smaller than a first set threshold value; wherein the value range of the first set threshold is 0.3-0.7; the third matching module is used for determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the matching number.

Further, the determining module is further configured to: judging that the lens is deviated under the condition that the matching quantity is smaller than or equal to a second set threshold value; wherein the value range of the second set threshold is 8-20.

Further, the determining module is further configured to: average calculation is carried out based on the first Euclidean distance of each matching point, and average offset distances are obtained; wherein the matching result further comprises an average offset distance.

Further, the determining module is further configured to: judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; wherein the value range of the third set threshold is 6-12.

Further, the determining module is further configured to: judging that the lens is not shifted under the condition that the matching number is larger than a second set threshold value and the average offset distance is smaller than a third set threshold value; wherein the value range of the second set threshold is 8-20, and the value range of the third set threshold is 6-12.

The embodiment of the present disclosure provides a chip, including a storage unit and a processing unit, where the storage unit stores a computer program, and the processing unit executes the computer program to implement the offset detection method according to any one of the embodiments.

The embodiment of the present specification provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the method for detecting offset of a lens according to any of the embodiments described above when executing the computer program.

The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method for detecting offset of a lens according to any of the above embodiments.

The method comprises the steps of determining a specified extraction quantity, a reference key point set of a reference frame image and a current frame image acquired by a lens; the reference key point set is obtained by extracting reference key points of the reference frame image based on the time invariant feature detector; extracting the detection key points with the specified extraction quantity from the current frame image based on the specified extraction quantity by a time-invariant feature detector to obtain a detection key point set; performing feature matching on the detection key point set and the reference key point set to obtain a matching result; if it is determined that the shot is not shifted based on the matching result, the current frame image is used as the reference frame image, and the above-described shift detection step for the shot is repeatedly performed, so that the accuracy of shift detection can be improved.

Drawings

Fig. 1 is a flowchart illustrating a method for detecting offset of a lens according to an embodiment of the present disclosure.

Fig. 2 is a flow chart of a feature matching method according to an embodiment of the present disclosure.

Fig. 3 is a flow chart of a method for determining a reference key point set according to an embodiment of the present disclosure.

Fig. 4 is a flowchart illustrating a method for detecting offset of a lens according to an embodiment of the present disclosure.

Fig. 5 is a schematic diagram of a lens shift detection device according to an embodiment of the present disclosure.

Fig. 6 is a schematic diagram of a computer device according to an embodiment of the present disclosure.

Detailed Description

In order to make the technical solution of the present specification better understood by those skilled in the art, the technical solution of the present specification embodiment will be clearly and completely described below with reference to the accompanying drawings in the embodiment of the present specification, and it is apparent that the described embodiment is only a part of the embodiment of the present specification, but not all the embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

Video monitoring is widely applied to scenes such as urban roads, airports, stations, business superships, workshops and the like, and video monitoring can acquire video images through fixed capturing view angles of cameras to perform video output so as to play a role. The camera is in the condition of unmanned inspection for a long time, particularly the outdoor camera, under the influence of weather conditions such as storm, snow, etc., the lens of the camera can deviate, and in order to ensure the reliability of video output, the lens deviation needs to be detected in time and maintained, thereby ensuring that the capturing view angle of the camera is correct.

In the related art, whether the lens is shifted is generally detected by adopting a difference method, a histogram matching method or a linear feature matching method, however, the difference method, the histogram matching method or the linear feature matching method is sensitive to illumination in the environment, the influence of brightness change is difficult to reduce, the shifting condition of misjudging the lens is easy to cause, and the detection error is large.

Therefore, for the outdoor cameras which are located in public places such as urban roads, airports and stations or are greatly influenced by weather, seasons and time factors, it is necessary to provide a method for detecting the offset of the lens, which can acquire the current frame image acquired by the lens, extract the detection key points from the current frame image based on the time-invariant feature detector, match the reference key points obtained by extracting the reference frame image based on the time-invariant feature detector with the detection key points, determine whether the lens is offset according to the matching result, and if not, take the current frame image as the reference frame image and continuously acquire the frame image acquired by the lens after a specified time interval so as to continuously detect the offset of the lens, thereby improving the accuracy of the offset detection of the lens.

The time invariant feature DEtector (A Temporally Invariant Learned DEtector, EILDE) is capable of efficiently detecting repeatable keypoints in the event of severe lighting changes due to weather, season, time, etc. The key points refer to information such as positions, directions, scales and the like of feature points in an image, the feature points can be some remarkable or special points in the image, such as corner points, edges, blocks, contour points and the like, and for example, bright points in darker areas, dark points in lighter areas and the like, and repeatable means that the same features can be found in different images.

Referring to fig. 1, fig. 1 is a schematic flow chart of a lens shift detection method according to the present embodiment, and the present embodiment provides method operation steps as the schematic flow chart, but may include more or less operation steps based on conventional or non-creative labor. The order of steps recited in the embodiments is merely one implementation of a plurality of step execution orders and does not represent a unique execution order. In actual system or server product execution, the methods illustrated in the embodiments may be performed sequentially or in parallel (e.g., in parallel processors or in the context of multi-threaded processing). As shown in fig. 1 in particular, the offset detection method may include the following steps.

Step S110: determining a specified extraction quantity, a reference key point set of reference frame images and a current frame image acquired by a lens; the reference key point set is obtained by extracting the reference key points of the reference frame image based on the trained time-invariant feature detector.

In some cases, the offset condition of the lens may be detected by performing image matching of the reference frame image of the lens and the current frame image acquired by the lens. For example, whether the lens is shifted or not may be determined according to the matching between the reference key point and the detection key point by performing feature matching on the plurality of reference key points of the reference frame image and the plurality of detection key points of the current frame image.

In the present embodiment, the shot may correspond to a reference frame image, and the reference key point set of the reference frame image may be extracted based on the time-invariant feature detector. The reference frame image may be a first frame image acquired by a lens, or may be any frame of the camera within a preset time period after the camera is started, for example, may be any frame within 0.1 seconds after the camera is started, or may be any frame within 1 second after the camera is started. Illustratively, the reference frame image may be of a specified size, for example, the reference frame image may be 360×240 in size.

In the present embodiment, the specified extraction number may be used for determining the extraction number of the keypoints at the time of performing the keypoint extraction by the time-invariant feature detector. Specifically, the specified extraction number may be used to determine the extraction number of the detection key points when the detection key points of the current frame image are extracted based on the time-invariant feature detector. For example, different specified extraction numbers may be determined for different scenes, or alternatively, the specified extraction numbers may be determined for the number or density of buildings, facility equipment, etc. in the scene in which the lens is located. As one example, for outdoor open scenes, a specified number of extractions may be determined to range from 60-100; for a more densely populated scene of buildings, facility equipment, personnel, etc., the specified number of extractions may be determined to range from 40-60, for example, the specified number of extractions may be 50. It should be noted that the above range of the specified extraction amount is only an exemplary range, and the specified extraction amount may be determined according to specific practical situations, and is not limited herein.

In this embodiment, in determining the current frame image acquired by the shot, it may be that: and obtaining a current frame initial image acquired by the lens, and performing scaling operation on the current frame initial image to obtain a current frame image with a specified size. Illustratively, the size of the current frame image may be 360×240.

Step S120: extracting detection key points from the current frame image based on the specified extraction quantity by a time-invariant feature detector to obtain a detection key point set; wherein the number of the detection key points in the detection key point set is equal to the specified extraction number.

In this embodiment, after determining the specified extraction number, the detection key points of the specified extraction number may be extracted from the current frame image by the trained time-invariant feature detector to form a detection key point set. Specifically, a regression response of the current frame image can be determined through a trained time invariant feature detector, and a current score map of the current frame image is obtained; and dividing the current frame image into a plurality of current frame image blocks, for example, dividing the current frame image into 3×3 current frame image blocks, and then performing local maximum search on the plurality of current frame image blocks by using a Non-maximum suppression algorithm (Non-Maximum Suppression, NMS), so as to obtain a detection key point set of the current frame image.

Step S130: and performing feature matching on the detection key point set and the reference key point set to obtain a matching result.

Specifically, a nearest neighbor search method can be adopted to perform feature matching on the detection key point set and the reference key point set, so that a matching result is obtained. For example, a knn match method of a fast nearest neighbor search packet (Fast Library for Approximate Nearest Neighbors, FLANN) may be used to perform homography matching on the set of detection key points and the set of reference key points, the Euclidean distance between the set of detection key points and the set of reference key points is determined by the knn match method, and the matching points are selected according to the Euclidean distance, so that the matching points matched with the set of reference key points in the set of reference key points may be determined, and a matching result may be obtained, so that whether the lens is shifted may be determined based on the matching result.

Step S140: if it is determined that the shot is not shifted based on the matching result, the current frame image is used as the reference frame image, and the shift detection step for the shot is repeatedly performed.

Specifically, if it is determined that the shot is not shifted based on the matching result, the current frame image may be used as a reference frame image, an updated reference frame image may be obtained, and a subsequent frame image acquired by the shot may be acquired according to a specified time interval, so that the shift condition of the shot may be continuously detected by performing image matching on the updated reference frame image and the subsequent frame image. The interval between the designations may be significantly smaller than the interval between the morning, noon or evening of a day and the intervals at the time corresponding to different weather, that is, the brightness change degree in the designated interval is smaller than the brightness change degree in the morning, noon or evening of a day and the brightness change degree caused by weather change, so that the influence of brightness change caused by time change or weather change on the offset detection of the lens can be effectively reduced by performing image matching on the subsequent frame image acquired after the designated interval with the reference frame image updated based on the current frame image, and the accuracy of the offset detection is improved.

In the above embodiment, the accuracy of the matching result is improved by determining the reference key point set extracted from the reference frame image by the time-invariant feature detector, acquiring the current frame image acquired by the lens, extracting the detection key point set from the current frame image by the time-invariant feature detector, and performing feature matching on the detection key point set and the reference key point set to obtain a matching result, so that the accuracy of the matching result is improved by using the robustness of the time-invariant feature detector to the brightness variation.

In some embodiments, referring to fig. 2, feature matching is performed on the set of detection keypoints and the set of reference keypoints to obtain a matching result, which may include the following steps.

Step S210: determining a Euclidean distance set between any detection key point in the detection key point set and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far.

Specifically, for any detection key point in the detection key point set, determining the euclidean distance between the detection key point and each reference key point in the reference key point set to obtain the euclidean distance set of the detection key point, and performing ascending arrangement on the euclidean distance set of the detection key point, so that the first euclidean distance and the second euclidean distance which are arranged from near to far and located in the first two positions can be obtained.

Euclidean distance, also known as Euclidean distance, is a common distance metric used to measure the absolute or true distance between two points in a multidimensional space.

Step S220: under the condition that the ratio of the first Euclidean distance to the second Euclidean distance is smaller than a first set threshold value, determining a reference key point corresponding to the first Euclidean distance as a matching point of the detection key point; wherein, the value range of the first set threshold value can be 0.3-0.7.

For example, the first set threshold may be 0.7, and if the ratio of the first euclidean distance to the second euclidean distance is less than 0.7, the reference keypoint corresponding to the first euclidean distance may be determined as the matching point of the detection keypoint. Otherwise, the detection key point has no matching point.

Step S230: determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the number of matches.

Specifically, the matching points of each detection key point in the detection key point set can be detected and judged to obtain the matching number of the matching points in the reference key point set, so that whether the lens is deviated or not can be determined according to the matching number of the matching points.

In some embodiments, the offset detection method may further include: under the condition that the matching number is smaller than or equal to a second set threshold value, judging that the lens is shifted; wherein the value range of the second set threshold is 8-20.

For example, the second set threshold may be 10, and in the case where the number of matches of the matching points is less than 10, it is determined that the lens is shifted. It should be noted that, the range of the second set threshold is only an exemplary range, and the second set threshold may be determined according to specific practical situations, which is not limited herein.

In some embodiments, the offset detection method may further include: and carrying out average calculation based on the first Euclidean distance of each matching point to obtain an average offset distance. Wherein the matching result further comprises an average offset distance.

In this embodiment, in order to improve accuracy of lens offset determination, when the number of matching points is greater than a second set threshold, average calculation may be performed based on the first euclidean distance of each matching point to obtain an average offset distance, so that whether the lens is offset may be determined according to the average offset distance of the matching points.

In some embodiments, the offset detection method may further include: and when the average offset distance is greater than or equal to a third set threshold value, determining that the lens is offset. The value range of the third set threshold value can be 6-12.

For example, the third set threshold may be 8, and in the case where the average shift distance is greater than or equal to 8, it is determined that the lens shifts. It should be noted that, the range of the third set threshold is only an exemplary range, and the third set threshold may be determined according to specific practical situations, which is not limited herein.

In some embodiments, the warning information may be output in a case where it is determined that the lens is shifted. Specifically, for example, text, symbols or picture alarm information may be displayed through a terminal device, or voice alarm information may be broadcasted through a terminal device, or alarm information may be displayed or broadcasted through a remote terminal.

In some embodiments, the offset detection method may further include: and under the condition that the matching number is larger than the second set threshold value and the average offset distance is smaller than the third set threshold value, judging that the lens is not offset.

For example, the second set threshold may be 10, the third set threshold may be 8, and it may be determined that the lens is not shifted in a case where the number of matches is greater than 8 and the average shift distance of the matching points is less than 8.

In some embodiments, referring to fig. 3, the determination of the set of reference keypoints may include the following steps.

Step S310: and acquiring an initial image of the current frame acquired by the lens.

Step S320: and performing scaling operation on the initial image of the current frame to obtain the image of the current frame with the specified size.

Step S330: and if the current frame image is the first frame, taking the current frame image as a reference frame image.

Step S340: and extracting the reference key points of the reference frame image based on the time invariant feature detector to obtain a reference key point set.

In some embodiments, specifying the manner in which the number of extractions is determined may include: in response to a setting operation of the specified extraction number, the specified extraction number is determined.

In the present embodiment, a setting interface may be provided by which the specified extraction amount is determined in response to a setting operation by the user.

In some embodiments, specifying the manner in which the number of extractions is determined may include: the method comprises the steps of obtaining test frame images acquired by a lens, classifying the test frame images to obtain target categories of environments monitored by the lens, and setting specified extraction quantity according to a preset extraction quantity range corresponding to the target categories. Wherein the preset extraction number range may relate to a dense situation of a building, facility equipment, personnel, etc. of the scene.

In some embodiments, the time-invariant feature detector may be trained to obtain a trained time-invariant feature detector that is effective for repeatable key point detection and extraction under severe luminance variations.

In particular, a training data set may be acquired, which may include a plurality of sample frame images at different brightness variations of the same scene or the same capture perspective. For example, the temporal change may result in a change in illumination and thus a change in brightness, and the training data set may be sample frame images captured at different times of the day, such as early morning, midday, or evening, of the same scene, or may be sample frame images captured at different seasons of the same scene.

Specifically, for each sample frame image, repeatable keypoints of the sample frame image can be detected and extracted by a scale invariant feature transform algorithm (Scale invariant feature transform, SIFT) to obtain sample keypoints of the sample frame image. For each sample frame image, for example, the repeatable keypoints of the sample frame image may be detected and extracted based on a predetermined specified extraction number, resulting in a specified extraction number of sample keypoints. After obtaining the specified number of sample keypoints of each sample frame image in the training dataset, each sample frame image may be segmented to obtain a plurality of image blocks, with the image block at the sample keypoint location being a positive sample and the image block at the sample keypoint location being a negative sample. And carrying out regression training on the regressor of the time-invariant feature detector according to the positive sample, the negative sample and the sample key points so that the regressor can generate a higher score at the positive sample and a lower score at the negative sample to obtain a score map of the sample frame image. In addition, the same or similar scores can be generated at the same position, so that when the brightness changes, the regressor can return consistent scores, and interference of brightness changes is reduced.

In the above embodiment, the regression training is performed on the time-invariant feature detector, so that a score map can be obtained based on the regressive device, and the non-maximum suppression algorithm is used to perform local maximum search on a plurality of image blocks, thereby extracting the key points of the image.

The embodiment of the present disclosure provides a method for detecting a shift of a lens, referring to fig. 4, the method may include the following steps.

Step S401: and acquiring an initial image of the current frame acquired by the lens.

Step S402: and performing scaling operation on the initial image of the current frame to obtain the image of the current frame with the specified size.

Illustratively, the specified size may be 320×240.

Step S403: if the current frame image is the first frame, the current frame image is used as the reference frame image, and the process goes to step 404. Otherwise, go to step 405.

Step S404: extracting reference key points from the reference frame image based on the specified extraction quantity by a time-invariant feature detector to obtain a reference key point set; go to step S406. Wherein the number of reference keypoints in the reference keypoint set is equal to the specified extraction number.

Step S405: extracting detection key points from the current frame image based on the specified extraction quantity by a time-invariant feature detector to obtain a detection key point set; wherein the number of the detected key points in the detected key point set is equal to the specified extraction number, and the process goes to step S406.

Step S406: and performing feature matching on the detection key point set and the reference key point set to obtain a matching result.

Specifically, a cannMatch method of FLANN may be used to perform feature matching on the detection key point set and the reference key point set to obtain matching. By way of example, euclidean distance between the detection key point set and the reference key point set can be determined through a knn match method, and the matching points are selected according to the euclidean distance, so that the matching points matched with the reference key point set in the reference key point set can be determined, a matching result can be obtained, and whether the lens is deviated or not can be determined based on the matching result.

Specifically, the matching result may include the number of matches of the matching points and the average offset distance.

Step S407: under the condition that the matching number is smaller than or equal to a second set threshold value, judging that the lens is shifted; otherwise, go to step S408. Wherein the value range of the second set threshold is 8-20. For example, the second set threshold may be 10.

Specifically, when the lens is judged to be offset, alarm information is output, so that operation and maintenance personnel can maintain the lens based on the alarm information.

Step S408: judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; otherwise, the process goes to step S409. The value range of the third set threshold value can be 6-12. For example, the third set threshold may be 8.

Step S409: and judging that the shot is not shifted, taking the current frame image as a reference frame image, and repeatedly executing the shift detection step for the shot.

Specifically, the current frame image is taken as a reference frame image, that is, a set of detection key points of the current frame image is taken as a set of reference key points.

Referring to fig. 5, a lens may correspond to a reference frame image, and the offset detection device may include an obtaining module 510, an extracting module 520, a matching module 530, and a determining module 540.

An obtaining module 510, configured to determine a specified extraction number, a reference key point set of a reference frame image, and a current frame image acquired by a lens; the reference key point set is obtained by extracting reference key points from a reference frame image based on a trained time invariant feature detector;

an extracting module 520, configured to extract, by using a time invariant feature detector, detection key points from the current frame image based on a specified extraction number, to obtain a detection key point set; the quantity of the detection key points in the detection key point set is equal to the specified extraction quantity;

the matching module 530 is configured to perform feature matching on the detected key point set and the reference key point set to obtain a matching result;

The determining module 540 is configured to repeatedly execute the above-mentioned offset detection step for the shot with the current frame image as the reference frame image if it is determined that the shot is not offset based on the matching result.

In some implementations, the matching module 530 may include: the first matching module is used for determining a Euclidean distance set between any detection key point in the detection key point set and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far according to the distance; the second matching module is used for determining a reference key point corresponding to the first Euclidean distance as a matching point of the detection key point under the condition that the ratio between the first Euclidean distance and the second Euclidean distance is smaller than a first set threshold value; wherein the value range of the first set threshold is 0.3-0.7; the third matching module is used for determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the number of matches.

In some implementations, the decision module 540 is further to: under the condition that the matching number is smaller than or equal to a second set threshold value, judging that the lens is shifted; wherein the value range of the second set threshold is 8-20.

In some implementations, the decision module 540 is further to: average calculation is carried out based on the first Euclidean distance of each matching point, and average offset distances are obtained; wherein the matching result further comprises an average offset distance.

In some implementations, the decision module 540 is further to: judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; wherein the value range of the third set threshold is 6-12.

In some implementations, the decision module 540 is further to: under the condition that the matching number is larger than a second set threshold value and the average offset distance is smaller than a third set threshold value, judging that the lens is not offset; wherein the value range of the second set threshold is 8-20, and the value range of the third set threshold is 6-12.

The specific functions and effects achieved by the offset detection device may be explained with reference to other embodiments of the present specification, and will not be described herein. The various modules in the offset detection apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in hardware or independent of a processor in the computer device, or can be stored in a memory in the computer device in a software mode, so that the processor can call and execute the operations corresponding to the modules.

Referring to fig. 6, in some embodiments, a computer apparatus may be provided, including a memory and a processor, where the memory stores a computer program, and the processor implements the method for detecting offset of a lens in the above embodiments when executing the computer program.

The present disclosure further provides a chip, including a storage unit and a processing unit, where the storage unit stores a computer program, and the processing unit implements the steps of the offset detection method in any of the above embodiments when executing the computer program.

The present specification also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a computer, causes the computer to execute the shift detection method of the lens in any of the above embodiments.

The present description also provides a computer program product containing instructions which, when executed by a computer, cause the computer to perform the method for detecting offset of a lens in any of the above embodiments.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, and a communication interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program, when executed by a processor, implements a method of detecting an offset of a lens.

It will be appreciated that the specific examples herein are intended only to assist those skilled in the art in better understanding the embodiments of the present disclosure and are not intended to limit the scope of the present invention.

It should be understood that, in various embodiments of the present disclosure, the sequence number of each process does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

It will be appreciated that the various embodiments described in this specification may be implemented either alone or in combination, and are not limited in this regard.

Unless defined otherwise, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this specification belongs. The terminology used in the description is for the purpose of describing particular embodiments only and is not intended to limit the scope of the description. The term "and/or" as used in this specification includes any and all combinations of one or more of the associated listed items. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It will be appreciated that the processor of the embodiments of the present description may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be implemented by integrated logic circuits of hardware in a processor or instructions in software form. The processor may be a general purpose processor, a Digital signal processor (Digital SignalProcessor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

It will be appreciated that the memory in the embodiments of this specification may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), or a flash memory, among others. The volatile memory may be Random Access Memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present specification.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system, apparatus and unit may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, each unit may exist alone physically, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present specification may be essentially or portions contributing to the prior art or portions of the technical solutions may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present specification. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random-access memory (RAM), a magnetic disk, or an optical disk, etc.

The foregoing is merely specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope disclosed in the present disclosure, and should be covered by the scope of the present disclosure. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (17)

1. A method for detecting offset of a lens is characterized in that the lens corresponds to a reference frame image; the offset detection method comprises the following steps:

determining a specified extraction quantity, a reference key point set of the reference frame image and a current frame image acquired by the lens; the reference key point set is obtained by extracting reference key points from the reference frame image based on a trained time invariant feature detector;

extracting detection key points from the current frame image based on the specified extraction quantity by the time invariant feature detector to obtain a detection key point set; the number of the detection key points in the detection key point set is equal to the specified extraction number;

performing feature matching on the detection key point set and the reference key point set to obtain a matching result;

and if the lens is judged not to deviate based on the matching result, taking the current frame image as the reference frame image, and repeatedly executing the deviation detection step for the lens.

2. The method for detecting offset according to claim 1, wherein the performing feature matching on the set of detection keypoints and the set of reference keypoints to obtain a matching result includes:

Determining a Euclidean distance set between any detection key point and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far according to the distance;

determining a reference key point corresponding to the first Euclidean distance as a matching point of any detection key point under the condition that the ratio between the first Euclidean distance and the second Euclidean distance is smaller than a first set threshold value; wherein the value range of the first set threshold is 0.3-0.7;

determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the matching number.

3. The offset detection method according to claim 2, characterized in that the offset detection method further comprises:

judging that the lens is deviated under the condition that the matching quantity is smaller than or equal to a second set threshold value; wherein the value range of the second set threshold is 8-20.

4. The offset detection method according to claim 2, characterized in that the offset detection method further comprises:

Average calculation is carried out based on the first Euclidean distance of each matching point, and average offset distances are obtained; wherein the matching result further comprises an average offset distance.

5. The offset detection method according to claim 4, characterized in that the offset detection method further comprises:

judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; wherein the value range of the third set threshold is 6-12.

6. The offset detection method according to claim 4, characterized in that the offset detection method further comprises:

judging that the lens is not shifted under the condition that the matching number is larger than a second set threshold value and the average offset distance is smaller than a third set threshold value; wherein the value range of the second set threshold is 8-20, and the value range of the third set threshold is 6-12.

7. The offset detection method according to claim 1, wherein the determining manner of the reference key point set includes:

acquiring an initial image of a current frame acquired by the lens;

performing scaling operation on the current frame initial image to obtain the current frame image with a specified size;

If the current frame image is the first frame, taking the current frame image as the reference frame image;

and extracting the reference key points of the reference frame image based on the time invariant feature detector to obtain the reference key point set.

8. The offset detection method according to claim 7, wherein the determination of the specified extraction number includes:

determining the specified extraction quantity in response to a setting operation of the specified extraction quantity;

or, acquiring a test frame image acquired by the lens, classifying the test frame image to obtain a target class of the environment monitored by the lens, and setting the appointed extraction quantity according to a preset extraction quantity range corresponding to the target class; wherein the preset extraction number range relates to a scene intensive situation.

9. A lens shift detection device, wherein the lens corresponds to a reference frame image; the offset detection device includes:

the acquisition module is used for determining the appointed extraction quantity, the reference key point set of the reference frame image and the current frame image acquired by the lens; the reference key point set is obtained by extracting reference key points from the reference frame image based on a trained time invariant feature detector;

The extraction module is used for extracting detection key points from the current frame image based on the specified extraction quantity through the time-invariant feature detector to obtain a detection key point set; the number of the detection key points in the detection key point set is equal to the specified extraction number;

the matching module is used for carrying out feature matching on the detection key point set and the reference key point set to obtain a matching result;

and the judging module is used for repeatedly executing the offset detection step for the lens by taking the current frame image as the reference frame image if the lens is judged not to be offset based on the matching result.

10. The offset detection apparatus of claim 9, wherein the matching module comprises:

the first matching module is used for determining a Euclidean distance set between any detection key point and any reference key point in the reference key point set aiming at any detection key point in the detection key point set; the Euclidean distance set comprises a first Euclidean distance and a second Euclidean distance which are arranged at the first two positions from the near to the far according to the distance;

The second matching module is used for determining a reference key point corresponding to the first Euclidean distance as a matching point of any detection key point under the condition that the ratio between the first Euclidean distance and the second Euclidean distance is smaller than a first set threshold value; wherein the value range of the first set threshold is 0.3-0.7;

the third matching module is used for determining the matching quantity of the matching points in the reference key point set; wherein the matching result includes the matching number.

11. The offset detection apparatus of claim 10, wherein the determination module is further configured to: judging that the lens is deviated under the condition that the matching quantity is smaller than or equal to a second set threshold value; wherein the value range of the second set threshold is 8-20.

12. The offset detection apparatus of claim 10, wherein the determination module is further configured to: average calculation is carried out based on the first Euclidean distance of each matching point, and average offset distances are obtained; wherein the matching result further comprises an average offset distance.

13. The offset detection apparatus of claim 12, wherein the determination module is further configured to: judging that the lens is shifted under the condition that the average shift distance is larger than or equal to a third set threshold value; wherein the value range of the third set threshold is 6-12.

14. The offset detection apparatus of claim 12, wherein the determination module is further configured to: judging that the lens is not shifted under the condition that the matching number is larger than a second set threshold value and the average offset distance is smaller than a third set threshold value; wherein the value range of the second set threshold is 8-20, and the value range of the third set threshold is 6-12.

15. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the offset detection method of any one of claims 1 to 8 when executing the computer program.

16. A chip comprising a memory unit and a processing unit, the memory unit storing a computer program, characterized in that the processing unit implements the steps of the offset detection method according to any one of claims 1 to 8 when the computer program is executed.

17. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the offset detection method of any of claims 1 to 8.