WO2016062159A1 - Image matching method and platform for testing of mobile phone applications - Google Patents

Abstract

Disclosed is an image matching method, comprising: performing global template matching of a template image in a source image, and controlling the template image to slide in the source image so as to find the best matching region; calculating feature points and feature vectors of the template image and the source image; calculating, according to the feature points and the feature vectors, calculating a visual similarity between the optimal matching region and the template image; if the visual similarity is zero, determining that the optimal matching region does not match with the template image; if the visual similarity is not zero, obtaining a pair of matching feature points of the template image and the source image, and calculating location coordinates of the best matching image according to the pair of matching feature points. Also disclosed is a platform for testing of mobile phone applications on the basis of image matching. The present invention reduces the complexity of the matching algorithm, thereby improving the efficiency of image matching and the efficiency of the testing of mobile phone applications on the basis of image matching.

Description

Image matching method and mobile application test platform Technical field

The invention relates to the technical field of mobile phone testing, and in particular to an image matching method mobile phone application testing platform.

Background technique

Image matching refers to identifying the same-named point between two or more images by a certain matching algorithm. For example, in the two-dimensional image matching, the correlation coefficient of the same size window in the target area and the search area is compared, and the correlation coefficient in the search area is taken. The maximum corresponding window center point is the same name point.

In order to realize the image-based automatic test function, the first solution is to accurately identify the same or similar image from the test image (such as the image captured from the mobile game) in the case of a given image, that is, the image matching problem. . After the identified image positions are transmitted, they are transmitted to various terminals for automatic testing. For example, according to the identified image position, the analog click is transmitted to the mobile phone to implement the simulation operation of the mobile game.

The prior art mainly provides two relatively common image matching methods: gray-based image matching method and feature-based image matching method.

Among them, the gray-based image matching method regards the image as a two-dimensional signal, and uses statistical correlation methods (such as correlation function, covariance function, difference square sum, etc.) to find correlation matching between signals. The most classic gray matching method is normalized gray matching. The basic principle is to pixelize the gray matrix of a real-time image window with a certain size and all possible window grayscale arrays of the reference image. A similarity measure method for matching methods for search comparison. The gray-based image matching method has the advantages of simple and direct calculation, but it also has obvious defects, that is, it has no rotation invariance and no scale invariance, and requires the template image and the target image to have the same size and direction. .

The feature-based image matching method refers to extracting features of two or more images (points, lines, faces, etc.), parameterizing the features, and then using the described parameters to perform matching. Algorithm. Firstly, the image is preprocessed to extract its high-level features, and then the matching correspondence between the two images is established. The commonly used feature primitives are point features, edge features and region features. The feature-based image matching method can overcome the shortcomings of using image gray information to match. Moreover, the feature point extraction process can reduce the influence of noise, and has good adaptability to gray level changes, image deformation and occlusion. However, it also has some shortcomings: 1) the real-time performance is not high, the calculation of feature points is time-consuming; 2) for some images, there may be few feature points extracted; 3) the feature points with smooth edges cannot accurately extract feature points.

The efficiency of image matching has a direct impact on the test effect of the application (APP). Especially in the test of mobile phone game testing, such as the start of mobile games or the image of the offensive button on the phone with different resolutions, the display image of the image will not change. Therefore, by correctly and quickly using image matching technology to recognize these button images, the simulated clicks on these buttons in the game can be completed, and the corresponding game operations can be automatically completed.

Summary of the invention

The technical problem to be solved by the present invention is to provide an image matching method and a mobile phone application testing method based on image matching, which improves the accuracy and flexibility of image matching, reduces the complexity of the matching algorithm, and thereby improves image matching and image-based optimization. Match the efficiency of mobile app testing.

To solve the above technical problem, in one aspect, an embodiment of the present invention provides a method for image matching, including:

Performing global template matching on the template image in the source image, and controlling the template image to slide in the source image to find a best matching area;

Calculating a feature point and a feature vector of the template image and the source image;

Calculating a visual similarity between the best matching area and the template image according to the feature point and the feature vector;

If the visual similarity is zero, determining that the best matching area does not match the template image;

If the visual similarity is not zero, obtaining a feature matching point pair between the template image and the source image;

According to the feature matching point pair, the positioning coordinates of the best matching image are calculated.

Further, the template image is globally matched in the source image, and the template image is controlled to slide in the source image to find a best matching area, specifically:

Obtaining a height and a width of the template image and the source image respectively;

If the height of the template image is greater than the height of the source image, or the width of the template image is greater than the width of the source image, determining that there is no matching area in the source image;

If the height of the template image is less than or equal to the height of the source image, and the width of the template image is less than or equal to the width of the source image, then:

And sliding the template image in the source image by a unit length, calculating a standard correlation coefficient between the template image and the source image one by one, and obtaining a standard correlation coefficient matrix;

Finding a maximum coefficient value in the standard correlation coefficient matrix, and a coordinate position corresponding to the maximum coefficient value;

And determining a position of the best matching area according to a coordinate position corresponding to the maximum coefficient value and a height and a width of the template image.

Preferably, the coordinate position corresponding to the maximum coefficient value is (m, n), the height of the template image is h1, and the width is w1;

Then, the position of the best matching area is: a rectangular area on the source image whose coordinate position (m, n) is the upper left corner, the length is h1, and the height is w1.

Further, the calculating the feature points and feature vectors of the template image and the source image, specifically including:

Searching for image positions of all scales on the image to be detected, and detecting extreme points that are invariant to scale and rotation by a Gaussian differential function;

Determining the position and scale of the feature points by establishing a fitting model according to the degree of stability of the extreme points;

Assigning one or more directions to the position of each feature point based on the gradient direction of the image local;

Measuring a local gradient of the image on a selected scale within a neighborhood around each feature point, transforming the gradient into a feature vector representing local shape deformation and illumination variation;

When the image to be detected is the template image, the feature point is a SIFT feature point of the template image; the feature vector is a SIFT feature vector of the template image; and when the image to be detected is the In the case of the source image, the feature point is a SIFT feature point of the source image; the feature vector is a SIFT feature vector of the source image.

Further, calculating a visual similarity between the best matching area and the template image according to the feature point and the feature vector, specifically:

Calculating a length of a SIFT feature point of the template image and a length of a SIFT feature point of the best matching area;

If the length of the SIFT feature point of the template image is zero, or the length of the SIFT feature point of the best matching area is zero, determining that the visual similarity between the best matching area and the template image is zero ;

If the length of the SIFT feature point of the template image is not zero, and the length of the SIFT feature point of the best matching area is not zero, then the feature of the template image and the best matching area is calculated The number of matching point pairs; the quotient of dividing the number of the feature matching point pairs by the length of the SIFT feature point of the template image as the visual similarity.

Preferably, if the visual similarity is not zero, obtaining a feature matching point pair between the template image and the source image, specifically:

Calculating a minimum Euclidean distance and a sub-Euclidean distance of the SIFT feature vector of the template image and the SIFT feature vector of the best matching region;

And when the quotient of the minimum Euclidean distance divided by the second small Euclidean distance is less than the first threshold, the template image and the feature point of the source image are used as the feature matching point pair, and the feature is matched The number of matching point pairs is superimposed.

Further, when the number of the feature matching point pairs is higher than the minimum matching number, the calculating the positioning coordinates of the best matching image according to the feature matching point pair includes:

Using a single mapping function to find a single mapping matrix corresponding to the pair of feature matching points;

Calculating, according to the single mapping matrix, a plurality of coordinate points of the best matching region of the template image on the source image by using a perspective transformation function of the vector array;

The center point coordinates of the best matching area are calculated, and the center point coordinates are used as the positioning coordinates of the best matching image.

In an implementation manner, the calculating, according to the single mapping matrix, a plurality of coordinate points of the best matching area of the template image on the source image by using a perspective transformation function of the vector array, specifically including :

Obtaining, according to the feature matching point pair, coordinates of the SIFT feature points on the template image and coordinates of the SIFT feature points on the source image that are matched one by one;

The coordinates of the pair of matching points are randomly selected, and a mapping is performed between the template image and the source image to obtain a first equation:

And obtaining corresponding mapping coefficients, forming the mapping coefficients into a coefficient matrix H, and obtaining a second equation:

Wherein N≥4; [x' _i , y' _i ] is the coordinate of the SIFT feature point on the source image; [x _i , y _i ] is the coordinate of the SIFT feature point on the template image; H is Mapping from a SIFT feature point on the template image to a coefficient matrix of SIFT feature points on the source image;

Calculating, by using the coefficient matrix, the SIFT feature points on the template image to real-time coordinates on the source image;

When the distance between the coordinates of the SIFT feature point on the source image and the real-time coordinate is less than the second threshold, the coefficient matrix H is updated by the first equation and the second equation until the coefficient matrix H does not change, and the coefficient matrix H that does not change is used as the single mapping matrix;

According to the single mapping matrix and the first equation, the coordinates (x', y') of the N matching points of the template image in the best matching region are calculated one by one by the following third equation:

The center point coordinates of the coordinates of the N matching points are used as the positioning coordinates of the best matching image.

Further, when the number of the feature matching point pairs is lower than the minimum matching number and greater than the specified magnification coefficient, wherein the specified magnification coefficient is smaller than the minimum matching number;

Then, according to the feature matching point, the positioning coordinates of the best matching image are calculated, specifically:

Performing SIFT strong matching on the template image includes: acquiring coordinates of SIFT feature points on the template image according to the feature matching point pairs, and one-to-one matching SIFT feature points on the source image coordinate;

The coordinates of the SIFT feature points on the source image are averaged, and the obtained mean coordinate values are used as the positioning coordinates of the best matching image.

Further, when the number of the feature matching point pairs is smaller than the specified rate coefficient, wherein the specified magnification coefficient is smaller than the minimum matching number;

And then selecting a neighboring area of the feature point in the best matching area to perform partial template matching with the template image, including:

Calculating a local visual similarity between the neighboring region of the feature point and the template image;

If the local visual similarity is higher than the third threshold, determining that the matching is successful, and calculating the positioning coordinates of the best matching image according to the coordinates obtained by the partial template matching;

If the local visual similarity is lower than the third threshold, global multi-scale template matching is performed on the template image and the source image.

Further, if the local visual similarity is lower than the third threshold, performing global multi-scale template matching on the template image and the source image, specifically:

Establishing a list of scales; the list of scales includes a plurality of scale factors;

And scaling the template image according to the scale factor in the scale list;

Performing global template matching on the template image after scaling, recording matching values and matching regions obtained by each matching, and forming a best matching set;

After calculating the global template matching of all the scales, the area corresponding to the largest matching value in the best matching set is taken as the best matching image, and the central coordinate value of the best matching image is calculated as the best Match the positioning coordinates of the image.

On the other hand, the embodiment of the present invention further provides a mobile phone application testing platform, where the mobile phone application testing platform includes a test script of the mobile phone application to be tested and an image resource required for testing, and further includes:

a test resource downloading unit, configured to download a test script of the mobile phone application to be tested and the image resource to the tested mobile phone;

a screenshot unit for taking a screenshot and uploading a test image of the mobile phone application to be tested displayed on the screen of the tested mobile phone;

An image matching unit, configured to perform image matching on the corresponding image resource as the template image by using the image matching method according to any one of the above items, and find the location of the best matching image of the test image. Coordinates; and,

And a test unit, configured to start testing the test code associated with the test image according to the positioning coordinates of the best matching image searched by the image matching unit, and feed back the positioning coordinate and the test result data to the tested mobile phone.

Further, the mobile phone application test platform is provided with a plurality of common interfaces, and a corresponding driver layer is disposed on the mobile phone application test platform for the universal interface.

Preferably, the mobile phone application to be tested is a mobile game application; and the mobile application test platform is a mobile game test platform.

Further, the mobile phone application testing platform further includes a testing center; the testing unit is further configured to transmit test data result data to the testing center;

The test result data includes the model information of the mobile phone to be tested, the screenshot generated by the testing process, the CPU information, the memory information, the power consumption information, and the network card traffic information.

The image matching method provided by the embodiment of the invention firstly uses the template matching method to perform template matching on the template image in the source image, and preferably uses the SIFT (Scale-Invariant Feature Transform) feature matching algorithm to determine the template. Image with best matching area Similarity, finally, according to the feature matching point pair, the positioning coordinates of the best matching image are calculated, and the grayscale-based template matching method and the SIFT feature matching method can be combined to enhance the strength and avoid shortness, and also have grayscale-based images. The calculation method of the matching method is simple, and the advantages of the rotation invariance and the scale invariance of the feature-based image matching method are directly improved, thereby improving the accuracy and flexibility of image matching. When the image matching method provided by the invention is applied to the mobile phone application test, the target image can be quickly and accurately identified, thereby improving the efficiency of the mobile phone application test.

DRAWINGS

1 is a flow chart showing the steps of one embodiment of a method of image matching provided by the present invention.

2 is a schematic diagram of global template matching in a source image provided by the present invention.

FIG. 3 is a flow chart showing the steps of calculating a feature point and a feature vector of a template image and a source image provided by the present invention.

4 is a flow chart showing the steps of an embodiment of a mobile phone application test platform provided by the present invention.

FIG. 5 is a schematic structural diagram of a mobile phone application test platform provided by the present invention for mobile phone application testing.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings.

Referring to FIG. 1, a flow chart of steps of an embodiment of a method for image matching provided by the present invention is shown.

In this embodiment, the method for image matching includes the following steps:

Step S101: Perform global template matching on the template image T in the source image S, and control the template image T to slide in the source image S to find a best matching area.

As shown in FIG. 2, it is a schematic diagram of the global template matching in the source image provided by the present invention. The source image S includes images of a plurality of controls or buttons, which are image 1 to image 6, respectively. The control template image T is slid from the upper left corner of the source image S to find the target image 4 in the source image S, and the template image T is calculated every time the matching window (the size of the template image T) is slid once The similarity between the image regions corresponding to the window.

In a specific implementation, the step S101 is specifically:

a. respectively acquiring the height and width of the template image T and the source image S;

If the height of the template image T is greater than the height of the source image S, or the width of the template image T is greater than the width of the source image S, it is determined that there is no matching area in the source image S;

c. If the height of the template image T is less than or equal to the height of the source image S, and the width of the template image T is less than or equal to the width of the source image S, then:

C1. The template image T is slid in the source image S by unit length, and the standard correlation coefficient of the template image T and the source image S is calculated one by one to obtain a standard correlation coefficient matrix A;

C2. Find a maximum coefficient value in the standard correlation coefficient matrix A, and a coordinate position corresponding to the maximum coefficient value;

C3. determining a position of the best matching region according to a coordinate position corresponding to the maximum coefficient value and a height h1 and a width w1 of the template image T.

In an achievable manner, the coordinate position corresponding to the maximum coefficient value is (m, n), the height of the template image is h1, and the width is w1; then the position of the best matching region is: A rectangular area on the source image having a coordinate position (m, n) as an upper left corner, a length of h1, and a height of w1. Specifically, an internal function template_match() can be designed to implement the above-mentioned step S101, and the pseudo code implementation process is as follows:

In the process of performing template matching, since the size and/or direction of the target image 4 in the template image T and the source image S may be inconsistent, the best matching area obtained by the step search is not necessarily a valid match ( That is, the best matching area may not be the target image 4), and further processing analysis of the source image S is required.

Step S102: Calculate feature points and feature vectors of the template image T and the source image S. In this embodiment, the feature points and feature vectors of the template image T and the source image S are preferably calculated by using a SIFT (Scale-Invariant Feature Transform) feature matching algorithm. The SIFT feature matching algorithm is a computer vision algorithm used to detect and describe local features in images, mainly by separately finding the Interest Points or Corner Points and their related scales. The descriptor of the orientation obtains the feature, finds the extreme point in the scale space, and extracts its position, scale, and rotation invariant, and then performs feature point matching of the two images. The essence of SIFT algorithm is to find feature points in different scale spaces and calculate the direction of feature points. The feature points found are some points that are very prominent and will not change due to factors such as illumination, affine transformation and noise. Such as corner points, edge points, bright areas of dark areas and dark spots of bright areas, etc., so SIFT features remain invariant to rotation, scale scaling, brightness changes, and also maintain a certain degree of viewing angle change, affine transformation and noise. stability.

After the feature points and feature vectors of the template image T and the source image S are obtained by the above steps, the visual similarities of the two can be further compared by step S103.

Step S103: Calculate a visual similarity between the best matching area and the template image T according to the feature point and the feature vector; and determine whether the visual similarity is zero, if the visual similarity is 0, step S104 is performed; if the visual similarity is not zero, step S105 is performed;

Step S104: It is determined that the best matching area does not match the template image T.

Step S105: Obtain a feature matching point pair of the template image T and the source image S, and perform step S106.

Step S106: Calculate the positioning coordinates of the best matching image according to the feature matching point pair.

Referring to FIG. 3, it is a flow chart of steps for calculating a feature point and a feature vector of a template image and a source image provided by the present invention.

In the specific implementation, the step S102 can be specifically implemented by the following steps, including:

Step S201: Scale space extreme value detection. Image positions of all scales are searched on the image to be detected, and extreme points that are invariant to scale and rotation (also known as potential points of interest for scale and rotation) are detected by a Gaussian differential function.

Step S202: Feature point location. According to the degree of stability of the extreme points, the position and scale of the feature points are determined by establishing a fitting model.

Step S203: The feature point direction is determined. One or more directions are assigned to the position of each feature point based on the gradient direction of the image local.

Step S204: feature point feature description. Within the neighborhood around each feature point, the gradient of the image local is measured at a selected scale, and the gradient is transformed into a feature vector representing local shape deformation and illumination variation.

Specifically, in the step S201 to the step S204, when the image to be detected is the template image T, the feature point is a SIFT feature point of the template image T; the feature vector is the template a SIFT feature vector of the image T; when the image to be detected is the source image S, the feature point is a SIFT feature point of the source image S; the feature vector is a SIFT feature vector of the source image S .

Further, in an achievable manner, the step S103 can be implemented by the following steps, specifically:

Step S301: Calculate the length len(keypoint1) of the SIFT feature point of the template image T and the length len(keypoint2) of the SIFT feature point of the best matching area. It is determined whether the visual similarity between the best matching region and the template image T is zero according to the length of the SIFT feature point of the template image T and the length of the SIFT feature point of the best matching region.

If the length of the SIFT feature point of the template image T is zero, or the length of the SIFT feature point of the best matching area is zero, step S302 is performed; if the length of the SIFT feature point of the template image T is not Zero, and the length of the SIFT feature point of the best matching area is not zero, then execution Step S303.

Step S302: determining that the visual similarity between the best matching area and the template image T is zero.

Step S303: Calculate the number of feature matching point pairs of the template image T and the best matching area, Good_Match; divide the number of the feature matching point pairs Good_Match by the length of the SIFT feature point of the template image T The quotient of (keypoint1) is taken as the visual similarity, that is, the value of visual similarity = Good_Match/len (keypoint1).

In the embodiment, the visual similarity obtained by the above step S103 is a "global visual similarity" obtained by global template matching of the template image T in the entire source image S by step S101, and the purpose is to implement the source image. The coarse filtering eliminates the source images (test pictures) that do not necessarily have matching areas, improving the efficiency of the image matching process.

In a specific implementation, if the visual similarity is not zero, in the step S105, the process of matching the feature pairs of the template image T and the source image S is obtained, which specifically includes:

Calculating a minimum Euclidean distance min_E and a second small Euclidean distance next_E of the SIFT feature vector of the template image T and the SIFT feature vector of the best matching region; dividing the minimum Euclidean distance min_E by the second small When the quotient of the Euclidean distance nextmin_E is smaller than the first threshold, the template image T and the feature point of the source image S are used as the feature matching point pair, and the number of the feature matching point pairs Good_Match is superimposed. For example, assuming that the first threshold TH1 is 0.75, when the minimum Euclidean distance min_E of the SIFT feature vector of the template image T and the SIFT feature vector of the best matching region is smaller than the second small Euclidean distance nextmin_E and the first threshold TH1 When the product of min_E<0.75*nextmin_E, the number of feature matching point pairs Good_Match is superimposed: Good_Match=Good_Match+1.

The above steps S103 to S105 can be implemented by constructing a function feature_similarity(), and the pseudo code can be expressed as:

In a specific implementation, the calculated SIFT feature point descriptor is its corresponding feature vector. The constructor cv2.SIFT.detectAndCompute() calculates the SIFT feature points of the template image T and the source image S and their SIFT feature point descriptors (ie, feature vectors):

Secondly, cv2.FlannBasedMatcher() is used to perform feature point matching, and then SIFT feature matching point pairs are calculated according to the nearest neighbor distance divided by the next nearest neighbor distance below a certain threshold (ie, the first threshold TH1). Wherein, "distance" refers to the Euclidean distance between one SIFT feature vector in the template image T and one SIFT feature vector in the source image S:

Keep the SIFT feature matching point pairs and record them as Good_Match.

After obtaining the number Good_Match of the feature matching point pairs of the template image T and the source image S, different strategies are selected according to the size of the feature matching point pair Good_Match to achieve positioning of the optimal matching image.

Further, when the visual similarity between the best matching area and the template image T obtained after the global template matching is not zero, the embodiment provides a more detailed implementation manner for finding the best matching area.

In a specific implementation, the minimum number of matches (MIN_MATCH_COUNT) may be set to define the size of the number of feature matching point pairs Good_Match. Different calculation strategies are selected by comparing the number of feature matching point pairs Good_Match with the minimum number of matches.

In one aspect, when the number of feature matching point pairs Good_Match is higher than the minimum matching number (MIN_MATCH_COUNT), the calculating the positioning coordinates of the best matching image according to the feature matching point pair includes: using a single mapping (homography) The so-called homography function finds a single mapping matrix (Homography Matrix) corresponding to the pair of feature matching points. Further, according to the single mapping matrix, a plurality of coordinate points of the best matching region of the template image T on the source image S are calculated by using a perspective transformation function of the vector array; and a center point of the best matching region is calculated. Coordinates, the center point coordinates are used as positioning coordinates of the best matching image.

Specifically, it is assumed that the minimum matching number MIN_MATCH_COUNT is 5, and if the number of feature matching point pairs Good_Match is higher than 5, the matching region is found by using the homography mapping, and the cv2.findHomography() function is constructed, and the matching key points are used to find the corresponding single. Map the matrix, and then use the cv2.perspectiveTransfrom() function to map the point group, and obtain the template image T to match the four coordinate points of the mapping area on the source image S, and then use the obtained coordinate points to calculate the coordinates of the center point of the matching area. The positioning function; conversely, if the number of feature matching point pairs Good_Match is lower than 5, further judgment is needed.

In this embodiment, in an implementable manner, the optimal matching region of the template image T on the source image S is calculated according to the single mapping matrix by using a perspective transformation function of a vector array. Multiple coordinate points, including the following steps:

Step S401: Acquire a SIFT feature on the template image T according to the feature matching point pair The coordinates of the point and the coordinates of the SIFT feature points on the source image S that match one by one.

Step S402: Randomly filter out the coordinates of the pair of matching points of N, and perform mapping between the template image T and the source image S to obtain the first equation:

And obtaining corresponding mapping coefficients, forming the mapping coefficients into a coefficient matrix H, and obtaining a second equation:

Wherein N≥4; [x' _i , y' _i ] is the coordinate of the SIFT feature point on the source image S; [x _i , y _i ] is the coordinate of the SIFT feature point on the template image T; H is a coefficient matrix mapped from SIFT feature points on the template image T to SIFT feature points on the source image S, where h ₁₁ to h ₃₃ are respective elements of the coefficient matrix H.

Step S403: Calculate real-time coordinates of the SIFT feature points on the template image T onto the source image S by using the coefficient matrix H.

Step S404: when the distance between the coordinates of the SIFT feature point on the source image S and the real-time coordinate is less than the second threshold TH2, the first equation (1) and the second equation (2) are used to The coefficient matrix H is updated until the coefficient matrix H no longer changes, and the coefficient matrix H that does not change is used as the single mapping matrix.

Step S405: Calculate, according to the single mapping matrix and the first equation (1), the coordinates of the N matching points of the template image T in the best matching region (x' by the following third equation (3). , y'):

Step S406: The center point coordinates of the coordinates of the N matching points are used as the positioning coordinates of the best matching image.

On the other hand, when the number of feature matching point pairs Good_Match is lower than the minimum number of matches When the MIN_MATCH_COUNT is greater than the specified magnification factor ratio_num (for example, the coefficient ratio_num may preferably be 0.1 times the number of SIFT feature points of the template image T), the step S106 is specifically: performing SIFT strong matching on the template image T, including the following step:

Step 61: Acquire, according to the feature matching point pair, coordinates of SIFT feature points on the template image T and coordinates of SIFT feature points on the source image S that are matched one by one;

Step 62: Perform averaging processing on the coordinates of the SIFT feature points on the source image S, and obtain the obtained mean coordinate values as the positioning coordinates of the best matching image. The specified rate coefficient ratio_num is smaller than the minimum number of matches MIN_MATCH_COUNT.

The purpose of performing strong matching in the step S106 is to prevent missing pairs of images that can be matched. In the specific implementation, some template images T can only extract a few SIFT feature points, but in fact these template images T just match the source image S, and the traditional SIFT algorithm cannot find the template images T with only a few feature points. Go to the matching area. After the strong matching is performed in the embodiment of the present invention, the defect of the traditional SIFT feature extraction method can be overcome, and the image matching capability can be improved.

Further, when the number of feature matching point pairs Good_Match is smaller than the specified magnification coefficient ratio_num, or the number of feature matching point pairs Good_Match is less than zero, that is, the number of feature matching point pairs that can be extracted is very small, (where The step S106 includes: selecting a neighboring area of the feature point in the best matching area to perform partial template matching with the template image T, which may be specifically performed by the following steps. Implement:

Step S601: calculating a local visual similarity between the neighboring region of the feature point and the template image T; if the local visual similarity is higher than the third threshold TH3, performing step S602; if the local visual similarity Below the third threshold TH3, step S603 is performed.

Step S602: determining that the matching is successful, and calculating the positioning coordinates of the best matching image according to the coordinates obtained by the partial template matching;

Step S603: Perform global multi-scale template matching on the template image T and the source image S.

The partial visual similarity in step S601 is obtained by performing partial template matching with the template image T by the neighboring regions of the feature points in the best matching region.

In the specific implementation, when calculating the local visual similarity, the feature_similarity() function described above may be used for the implementation; and the similarity of the color histogram may also be used for calculation. Specifically, the color histogram H1(i) of the template image T and the color histogram H2(i) of the adjacent region in the source image S may be separately calculated, and then the local visual similarity is solved by using the fourth equation (4):

The adjacent area of the feature point in step S601 may be selected as a rectangular area whose height and width are respectively twice the height and width of the template image T centered on the coordinates of the feature point, and the template image T is obtained on the rectangular area. Matching is performed to select the best matching area. If the visual similarity with the template image T is higher than a certain threshold (TH3), the matching is considered successful, otherwise step S603 is performed.

In an achievable manner, if the local visual similarity is lower than the third threshold TH3, the step S603 includes:

Step S6031: Establish a scale list; the scale list includes a plurality of scale coefficients;

Step S6032: scaling the template image T according to the scale factor in the scale list;

Step S6033: performing global template matching on the template image T after the scaling is performed, and recording the matching value and the matching area obtained by each matching to form a best matching set;

Step S6034: After calculating the global template matching of all the scales, the area corresponding to the maximum matching value in the best matching set is taken as the best matching image, and the central coordinate value of the best matching image is calculated as the The positioning coordinates of the best matching image.

The above process can be implemented by using the function multi_scale_match(), and the above process is represented by pseudo code as follows:

In this embodiment, the multi-scale matching similarity between the template T and the source image S is mainly calculated, and the multi-scale scaling is implemented on the template T, and the problem that the template matching is sensitive to the scale transformation is solved to some extent, if the matching value ( If the matching value of the scaled template image and the source image S is lower than a certain threshold, the matching is considered to be failed, otherwise the visual matching degree of the best matching region and the template T is calculated.

Steps S6031 to S6034 adopt a multi-scale template matching method, which functions as fine filtering to eliminate interference of the source image S which is easy to match errors, and thus is more abundant than the template matching process of step S101 described above.

The image matching method provided by the embodiment of the invention can be implemented in the Python language, and has the characteristics of high efficiency and legibility, and realizes rapid application development of the image matching method.

The image matching method provided by the embodiment of the invention uses the template matching method to perform global template matching on the template image in the source image, and uses the SIFT feature matching algorithm to determine the similarity between the template image and the best matching region, and finally according to the feature. Match the pair of points and calculate the coordinates of the best matching image. And different matching processes are adopted according to the number of matched feature point pairs, which reduces the complexity of the algorithm and improves the accuracy of image matching. The embodiment of the present invention can combine the gray-based template matching method and the SIFT feature matching method, and the rotation invariance of the simple and direct feature-based image matching method based on the gray-based image matching method. And the advantages of scale invariance, thus improving the accuracy and flexibility of image matching.

The invention also provides a mobile phone application testing platform for applying the above image matching method to testing a mobile phone application (Application, APP).

FIG. 4 is a schematic structural diagram of an embodiment of a mobile phone application testing platform provided by the present invention.

The mobile phone application testing platform provided by the embodiment can implement the automatic testing function of the mobile phone application APP based on image matching, and the first problem is the image matching problem. After identifying the correct image position, It is transmitted to the mobile phone to be tested to realize analog click, and realizes the simulation operation of the mobile application (such as mobile game).

The mobile application test platform includes:

The test resource downloading unit 401 is configured to download a test script of the mobile phone application to be tested and the image resource to the tested mobile phone.

The screenshot unit 402 is configured to take a screenshot and upload the test image of the mobile phone application to be tested displayed on the screen of the tested mobile phone;

The image matching unit 403 is configured to perform image matching on the corresponding image resource by using the image matching method as the template image to find the best matching image of the test image. Positioning coordinates; and,

The testing unit 404 is configured to start testing the test code associated with the test image according to the positioning coordinates of the best matching image searched by the image matching unit 403, and feed back the positioning coordinate and the test result data to the tested Mobile phone.

In a specific implementation, the mobile phone application test platform further includes a storage unit 405 that stores a test script of the mobile phone application to be tested and a required image resource for testing; and the test resource download unit 401 downloads a test corresponding to the application to be tested from the storage unit 405. Script and image resources. Performing image matching on the mobile phone application test platform as the template image, finding the positioning coordinates of the best matching image of the image to be tested, and determining the response state of the tested mobile phone; In the case of a template image, the same or similar images are accurately identified from test pictures (such as pictures taken from a mobile game) using the image matching method described above. Further, the mobile application test platform is provided with a plurality of universal interfaces 406, and a corresponding drive layer is disposed on the mobile application test platform for the universal interface 406.

In an implementable manner, as shown in FIG. 5, the mobile phone application test platform can be installed in the server 502, and the server 502 can communicate with the mobile phone 501 to be tested through various communication interfaces.

The mobile phone application testing platform is provided with a plurality of general-purpose interfaces, and a corresponding driving layer is disposed on the mobile phone application testing platform for the universal interface; and the mobile phone 501 and the mobile phone application testing platform pass through The general interface and the driver layer perform data transmission. Specifically, for various interfaces of the server 502, respectively, an operating system (Ios system, Android system, etc.) of the mobile phone 501 to be tested, And the server 502 operating system (windows platform) implements the corresponding driver layer. The communication interface can be implemented by using an open source Appium tool for the Ios system; for the Android system, the ADB (Android Debug Bridge) tool provided by Google can be used to implement the communication interface; for the Windows system, it can be directly used. The underlying API (Application Programming Interface) communicates.

When the mobile application APP is automatically tested by the mobile application test platform: first, the image resources needed in the test code and the test script are prepared; and the screenshot on the mobile phone 501 to be tested is transmitted to the server 502 through the driver layer. Secondly, the position of the image (such as the sun icon in FIG. 5) is recognized on the server 502, and the image is positioned by using any of the image matching methods described above to find the position where the target image is located, that is, The abscissa x value and the ordinate y value constitute a target image position (x, y); then, the (x, y) coordinates are transmitted to the mobile phone 501 to be tested through the communication interface, and the mobile application APP (such as a mobile game) is completed. The analog click operation in ). In a specific implementation, the mobile phone application method further includes: returning the test result data to the test center; the test result data includes the model information of the mobile phone to be tested, the screenshot generated by the test process, the CPU information, the memory information, and the power consumption information. And network card traffic information.

In a preferred embodiment, the mobile phone application APP to be tested is a mobile game application; and the mobile application test platform is a mobile game test platform. Applying the improved image matching method and mobile phone application testing method to the field of mobile game testing can effectively improve the efficiency of existing mobile game testing, lower the threshold of mobile game testing, improve the convenience of mobile game testing, and realize the mobile game. Remote testing.

The mobile phone application testing platform provided by the embodiment utilizes the advantages of the improved image matching method, reduces the defect that the mobile phone of different resolutions needs to repeatedly write test code, realizes automatic testing of the smart phone application, and reduces the manual test mobile phone application. Cost and improve test efficiency and test accuracy. The test code on the mobile app test platform can support programs running on multiple smartphone operating systems at the same time, improving compatibility. When the mobile application test platform is integrated into the mobile application, it can help to test the application APP in the mobile phone anytime and anywhere, especially for ordinary users, and improve the application range of the mobile application test.

The above is a preferred embodiment of the present invention, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present invention. It is the scope of protection of the present invention.

Claims (16)

1. A method for image matching, characterized in that the method comprises:

   Performing global template matching on the template image in the source image, and controlling the template image to slide in the source image to find a best matching area;

   Calculating a feature point and a feature vector of the template image and the source image;

   Calculating a visual similarity between the best matching area and the template image according to the feature point and the feature vector;

   If the visual similarity is zero, determining that the best matching area does not match the template image;

   If the visual similarity is not zero, obtaining a feature matching point pair between the template image and the source image;

   According to the feature matching point pair, the positioning coordinates of the best matching image are calculated.
2. The image matching method according to claim 1, wherein the template image is globally matched in the source image, and the template image is controlled to slide in the source image to find a best matching area, specifically for:

   Obtaining a height and a width of the template image and the source image respectively;

   If the height of the template image is greater than the height of the source image, or the width of the template image is greater than the width of the source image, determining that there is no matching area in the source image;

   If the height of the template image is less than or equal to the height of the source image, and the width of the template image is less than or equal to the width of the source image, then:

   And sliding the template image in the source image by a unit length, calculating a standard correlation coefficient between the template image and the source image one by one, and obtaining a standard correlation coefficient matrix;

   Finding a maximum coefficient value in the standard correlation coefficient matrix, and a coordinate position corresponding to the maximum coefficient value;

   And determining a position of the best matching area according to a coordinate position corresponding to the maximum coefficient value and a height and a width of the template image.
3. The image matching method according to claim 2, wherein the coordinate position corresponding to the maximum coefficient value is (m, n), the height of the template image is h1, and the width is w1;

   Then, the position of the best matching area is: a rectangular area on the source image whose coordinate position (m, n) is the upper left corner, the length is h1, and the height is w1.
4. The method of image matching according to claim 2, wherein the calculating the feature points and the feature vectors of the template image and the source image comprises:

   Searching for image positions of all scales on the image to be detected, and detecting extreme points that are invariant to scale and rotation by a Gaussian differential function;

   Determining the position and scale of the feature points by establishing a fitting model according to the degree of stability of the extreme points;

   Assigning one or more directions to the position of each feature point based on the gradient direction of the image local;

   Measuring a local gradient of the image on a selected scale within a neighborhood around each feature point, transforming the gradient into a feature vector representing local shape deformation and illumination variation;

   When the image to be detected is the template image, the feature point is a SIFT feature point of the template image; the feature vector is a SIFT feature vector of the template image; and when the image to be detected is the In the case of the source image, the feature point is a SIFT feature point of the source image; the feature vector is a SIFT feature vector of the source image.
5. The image matching method according to claim 4, wherein the visual similarity between the best matching area and the template image is calculated according to the feature point and the feature vector, specifically:

   Calculating a length of a SIFT feature point of the template image and a length of a SIFT feature point of the best matching area;

   If the length of the SIFT feature point of the template image is zero, or the length of the SIFT feature point of the best matching area is zero, determining that the visual similarity between the best matching area and the template image is zero ;

   If the length of the SIFT feature point of the template image is not zero, and the length of the SIFT feature point of the best matching area is not zero, then the feature of the template image and the best matching area is calculated The number of matching point pairs; the quotient of dividing the number of the feature matching point pairs by the length of the SIFT feature point of the template image as the visual similarity.
6. The image matching method according to claim 5, wherein if the visual similarity is not zero, obtaining a feature matching point pair between the template image and the source image comprises:

   Calculating a minimum Euclidean distance and a sub-Euclidean distance of the SIFT feature vector of the template image and the SIFT feature vector of the best matching region;

   And when the quotient of the minimum Euclidean distance divided by the second small Euclidean distance is less than the first threshold, the template image and the feature point of the source image are used as the feature matching point pair, and the feature is matched The number of matching point pairs is superimposed.
7. The image matching method according to claim 6, wherein when the number of the feature matching point pairs is higher than the minimum matching number, the positioning of the best matching image is calculated according to the feature matching point pair Coordinates, including:

   Using a single mapping function to find a single mapping matrix corresponding to the pair of feature matching points;

   Calculating, according to the single mapping matrix, a plurality of coordinate points of the best matching region of the template image on the source image by using a perspective transformation function of the vector array;

   The center point coordinates of the best matching area are calculated, and the center point coordinates are used as the positioning coordinates of the best matching image.
8. The method for image matching according to claim 7, wherein said calculating, based on said single mapping matrix, a best matching region of said template image on said source image by using a perspective transformation function of a vector array Multiple coordinate points, including:

   Obtaining, according to the feature matching point pair, coordinates of the SIFT feature points on the template image and coordinates of the SIFT feature points on the source image that are matched one by one;

   The coordinates of the pair of matching points are randomly selected, and a mapping is performed between the template image and the source image to obtain a first equation:

   

   And obtaining corresponding mapping coefficients, forming the mapping coefficients into a coefficient matrix H, and obtaining a second equation:

   

   Wherein N≥4; [x' _i , y' _i ] is the coordinate of the SIFT feature point on the source image; [x _i , y _i ] is the coordinate of the SIFT feature point on the template image; H is Mapping from a SIFT feature point on the template image to a coefficient matrix of SIFT feature points on the source image;

   Calculating, by using the coefficient matrix, the SIFT feature points on the template image to real-time coordinates on the source image;

   When the distance between the coordinates of the SIFT feature point on the source image and the real-time coordinate is less than the second threshold, the coefficient matrix H is updated by the first equation and the second equation until the coefficient matrix H does not change, and the coefficient matrix H that does not change is used as the single mapping matrix;

   According to the single mapping matrix and the first equation, the coordinates (x', y') of the N matching points of the template image in the best matching region are calculated one by one by the following third equation:

   

   The center point coordinates of the coordinates of the N matching points are used as the positioning coordinates of the best matching image.
9. The image matching method according to claim 6, wherein when the number of the feature matching point pairs is lower than the minimum matching number and larger than the specified magnification coefficient, wherein the specified magnification coefficient is smaller than the Minimum number of matches;

   Then, according to the feature matching point, the positioning coordinates of the best matching image are calculated, specifically:

   Performing SIFT strong matching on the template image includes: acquiring coordinates of SIFT feature points on the template image according to the feature matching point pairs, and one-to-one matching SIFT feature points on the source image coordinate;

   The coordinates of the SIFT feature points on the source image are averaged, and the obtained mean coordinate values are used as the positioning coordinates of the best matching image.
10. The image matching method according to claim 6, wherein when the number of the feature matching point pairs is smaller than the specified magnification coefficient, wherein the specified magnification coefficient is smaller than the minimum matching number;

    And calculating, according to the feature matching point, the positioning coordinates of the best matching image, including: selecting a neighboring region of the feature point in the best matching region to perform partial template matching with the template image.
11. The image matching method according to claim 10, wherein selecting a neighboring region of the feature point in the best matching region to perform partial template matching with the template image comprises:

    Calculating a local visual similarity between the neighboring region of the feature point and the template image;

    If the local visual similarity is higher than the third threshold, determining that the matching is successful, and calculating the positioning coordinates of the best matching image according to the coordinates obtained by the partial template matching;

    If the local visual similarity is lower than the third threshold, global multi-scale template matching is performed on the template image and the source image.
12. The image matching method according to claim 11, wherein if the local visual similarity is lower than the third threshold, global multi-scale template matching is performed on the template image and the source image, specifically include:

    Establishing a list of scales; the list of scales includes a plurality of scale factors;

    And scaling the template image according to the scale factor in the scale list;

    Performing global template matching on the template image after scaling, recording matching values and matching regions obtained by each matching, and forming a best matching set;

    After calculating the global template matching of all the scales, the area corresponding to the largest matching value in the best matching set is taken as the best matching image, and the central coordinate value of the best matching image is calculated as the best Match the positioning coordinates of the image.
13. A mobile phone application testing platform, comprising: a test script for a mobile phone application to be tested and an image resource required for testing, wherein the mobile application test platform comprises:

    a test resource downloading unit, configured to download a test script of the mobile phone application to be tested and the image resource to the tested mobile phone;

    a screenshot unit for taking a screenshot and uploading a test image of the mobile phone application to be tested displayed on the screen of the tested mobile phone;

    An image matching unit for performing image matching according to any one of claims 1 to 12, performing image matching on the corresponding image resource as the template image, and finding the most of the test image Good matching image positioning coordinates; and,

    And a test unit, configured to start testing the test code associated with the test image according to the positioning coordinates of the best matching image searched by the image matching unit, and feed back the positioning coordinate and the test result data to the tested mobile phone.
14. The mobile phone application testing platform according to claim 13, wherein the mobile phone application testing platform is provided with a plurality of universal interfaces, and a corresponding driving layer is disposed on the universal interface.
15. The mobile phone application testing platform according to claim 13 or 14, wherein the mobile phone application testing platform is a mobile game testing platform;
16. The mobile phone application testing platform according to claim 15, wherein the platform further comprises a testing center; the testing unit is further configured to transmit test data result data to the testing center; The test result data includes the model information of the mobile phone to be tested, the screenshot generated by the testing process, the CPU information, the memory information, the power consumption information, and the network card traffic information.

Images