CN105740874A - Method and device for determining operation coordinate of automation test script during playback - Google Patents
Method and device for determining operation coordinate of automation test script during playback Download PDFInfo
- Publication number
- CN105740874A CN105740874A CN201610124617.3A CN201610124617A CN105740874A CN 105740874 A CN105740874 A CN 105740874A CN 201610124617 A CN201610124617 A CN 201610124617A CN 105740874 A CN105740874 A CN 105740874A
- Authority
- CN
- China
- Prior art keywords
- user interface
- graphic user
- operation coordinate
- point
- characteristic point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention relates to a method and a device for determining an operation coordinate of an automation test script during playback, comprising steps of zooming a graphic user interface of a tested mobile device to a preset size to obtain a first graph user interface and a second graphic user interface before recording and playing back the test script, recording the operation coordinate and the operation icon during the recording process during the process of recording the test script, calculating the characteristic points of the operation icon and the second graphic user interface during the process of playing back the test script, screening a characteristic point as an effective characteristic point from the characteristic points according to the distance between the operation coordinate, calculating all the matching points between the effective characteristic point and the operation coordinate, and using the mass center of the calculated matching point as the operation coordinate for the playback. The disclosed technical scheme can effectively filter the noise point which is relatively far away from the real object, can accurately match the operation object during recording and improves the matching efficiency.
Description
Technical field
It relates to image identification technical field, particularly relate to a kind of operation determined when automatic test script plays back and sit calibration method and device.
Background technology
It is currently based on the correlation technique of picture script recording and playback to have sikuli (Sikuli is the figure script of a kind of novelty, it it is the figure script of a kind of novelty of the research worker design of MIT, need only there be most basic programming skill in computer user, need not go to write out code line by line, but by the mode of screenshot capture, the program of mystery it is combined into) with the graphic element cut out, image recognition is adopted the stencil matching method based on pixel by this technology, namely compared by according to pixels sleiding form image and original image, by difference of two squares matching algorithm draw each slip ratio compared with difference results, the that time finally obtaining difference minimum slides region as matching result.
Handle icon in the script of this correlation technique does not adapt to the change of screen resolution, when changing test equipment, it is necessary to again recording script, test script can not carry out equipment compatibility test.The above-mentioned stencil matching method based on pixel, the image of the different resolution pixel color value due to distribution cannot be adapted to and produce larger difference along with change resolution, thus making the difference results misalignment that difference of two squares algorithm draws, the coupling for same template image cannot accomplish effective compatibility.
Also having a kind of correlation technique is SIFT (Scale-invariantfeaturetransform, Scale invariant features transform) feature matching method, SIFT feature matching process, by asking the characteristic point (interestpointsorcornerpoints) in two width figure and description about yardstick scale and direction orientation thereof to obtain feature respectively, then carries out the Feature Points Matching of two width images.SIFT algorithm is DavidG.Lowe proposed in 1999 a kind of based on metric space, to image scaling, rotate the image local feature that even affine transformation maintains the invariance and describe operator, it is a local feature vectors collection that piece image is mapped (conversion) by SIFT, characteristic vector has translation, convergent-divergent, rotational invariance, illumination variation, affine and projective transformation is also had certain invariance simultaneously.Therefore SIFT feature coupling energy efficient solution is never with the problem of reorientation handle icon in resolution picture.
But in the practical application of script recording and playback, adopt SIFT feature coupling still to there are some problems, first can cause operation coordinate error hiding, it is not necessary to match point can affect the accuracy rate of matching result and matched position, affect the accuracy rate of target reorientation.When SIFT feature is mated, the different key point in two positions likely can have similar characteristic vector, this the two key point is mated to make algorithm think, these unnecessary match points can affect the accuracy rate of matching result and matched position, affects the accuracy rate of target reorientation.Secondly, matching operation time effects playback real-time, during SIFT feature Point matching, the characteristic vector distance of 128 dimensions calculates, and will produce unnecessary operation time to mating with invalid SIFT feature point, impact playback real-time.
Summary of the invention
For overcoming Problems existing in correlation technique, the disclosure provides a kind of operation determined when automatic test script plays back to sit calibration method and device.
First aspect according to disclosure embodiment, it is provided that calibration method is sat in a kind of operation determined when automatic test script plays back, including:
Recording before test script, zooming to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment；
When recording test script, the operation coordinate on described first graphic user interface when record is recorded, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface；
Playing back before described test script, zooming to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment；
When playing back described test script:
Calculate the characteristic point of described handle icon and described second graph user interface；
From described characteristic point, characteristic point is screened as validity feature point according to the distance between described operation coordinate；
Calculate the match point of described validity feature point and described operation coordinate, operation coordinate during using the barycenter of computed match point as playback.
Second aspect according to disclosure embodiment, it is provided that the device of a kind of operation coordinate determined when automatic test script plays back, including:
Record pretreatment unit, for recording before test script, zoom to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment；
Script recording elements, for when recording test script, the operation coordinate on described first graphic user interface when record is recorded, according to the handle icon that described operation coordinate intercepting recording operation is corresponding on described first graphic user interface；
Playback pretreatment unit, for playing back before described test script, zooms to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment；
Script playback unit, including characteristic point computation subunit, validity feature point screening subelement and operation coordinate computation subunit；
Described characteristic point computation subunit, for when playing back described test script, calculating the characteristic point of described handle icon and described second graph user interface；
Described validity feature point screening subelement, screens characteristic point as validity feature point for the distance between basis and described operation coordinate from described characteristic point；
Described operation coordinate computation subunit, the operation coordinate for calculating the match point of described validity feature point and described operation coordinate, during using the barycenter of computed match point as playback.
Embodiment of the disclosure that the technical scheme of offer can include following beneficial effect:
The technical scheme of the disclosure effectively can filter from realistic objective noise farther out when automatic test, can accurate match operation target when recording, and improve matching efficiency.
It should be appreciated that it is only exemplary and explanatory that above general description and details hereinafter describe, the disclosure can not be limited.
Accompanying drawing explanation
Accompanying drawing herein is merged in description and constitutes the part of this specification, it is shown that meets embodiments of the invention, and is used for explaining principles of the invention together with description.
Fig. 1 is the flow chart that calibration method is sat in a kind of operation determined when automatic test script plays back according to an exemplary embodiment；
Fig. 2 is the flow chart that calibration method is sat in a kind of operation determined when automatic test script plays back according to an exemplary embodiment；
Fig. 3 is the flow chart of a kind of method intercepting recording operation handle icon corresponding on the first graphic user interface based on rim detection according to operation coordinate according to an exemplary embodiment；
Fig. 4 is the flow chart of the method for a kind of automatic test playback phase according to an exemplary embodiment；
Fig. 5 is the block diagram of the device of a kind of operation coordinate determined when automatic test script plays back according to an exemplary embodiment；
Fig. 6 is the block diagram of the device of a kind of operation coordinate determined when automatic test script plays back according to an exemplary embodiment.
Detailed description of the invention
Here in detail exemplary embodiment being illustrated, its example representation is in the accompanying drawings.When as explained below relates to accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represents same or analogous key element.Embodiment described in following exemplary embodiment does not represent all embodiments consistent with the present invention.On the contrary, they only with in appended claims describe in detail, the present invention some in the example of consistent apparatus and method.
Fig. 1 is the flow chart that calibration method is sat in a kind of operation determined when automatic test script plays back according to an exemplary embodiment, the present embodiment is applicable to when automatic test, the situation operating coordinate during playback is determined according to operation coordinate when recording, include as it is shown in figure 1, calibration method is sat in the operation when determination automatic test script described in the present embodiment plays back:
In step s 110, recording before test script, zooming to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment.
In order to make the graphic user interface of the equipment of playback test script adaptation different resolution, it is prevented effectively from test and produces invalid operation because mating inaccurate, the present embodiment based on by record time operation graphic user interface with playback time for graphic user interface zoom to formed objects, matching efficiency during recording and playback can be improved, so that operation target when accurate match is recorded during playback is operated.
Namely respectively the graphic user interface of operation is normalized when recording with playback, operation screen is zoomed in or out into an identical fixing length and width picture.Although position and the size of stretching or the variant icon that may result in picture of reduced scale cannot be completely the same, but different resolution picture can be made to record and the process of playback has the distribution of similar icon.
Such as, this step may be accomplished by: namely obtained the boundary parameter being test for the first graphic user interface moving equipment by preset function, according to described boundary parameter, described first graphic user interface moving equipment is zoomed in and out according to pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device recorded as the first graphic user interface.
In the step s 120, when recording test script, the operation coordinate on described first graphic user interface when record is recorded, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface.
After obtaining the first graphic user interface through step S110 convergent-divergent, the present embodiment requires to be based on this first graphic user interface after scaling to carry out when recording test script.When recording test script, the operation coordinate on described first graphic user interface when record is recorded, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface.
It should be noted that, to those skilled in the art, multiple method can be adopted, for instance can realize intercepting, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on the method for rim detection according to the handle icon that described operation coordinate intercepts recording operation corresponding on described first graphic user interface.
In step s 130, playing back before described test script, zooming to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment.
This step for by playback time for graphic user interface zoom to and when recording operation graphic user interface formed objects.
Similar with step S110, such as, this step realizes also by the following manner: obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
It should be noted that operating procedure when following steps S140 to step S160 belongs to play back described test script.
In step S140, when playing back described test script, calculate the characteristic point of described handle icon and described second graph user interface.
The characteristic point calculating described handle icon and described second graph user interface can adopt multiple method, the present embodiment is in this no limit, for instance can realize based on SIFT algorithm, SURF algorithm, the marginal point algorithm of wavelet transformation or Harris Corner Detection Algorithm.
In step S150, from described characteristic point, screen characteristic point as validity feature point according to the distance between described operation coordinate.
After step S140, the characteristic point of second graph user interface when handle icon during recording and playback can be a lot, wherein can comprise some unnecessary match points, these unnecessary match points can affect the accuracy rate of matching result and matched position, affects the accuracy rate of operation target reorientation during script playback.
This step, based on when playing back described test script, filters out some unnecessary match points as far as possible, to improve the accuracy rate operating coordinate when determining that script play back and efficiency.
The method can include multiple, in view of the present embodiment in step S110 and step 130 respectively by playback time for graphic user interface zoom to and when recording operation graphic user interface formed objects.Therefore, operation coordinate during playback does not answer difference too big with the operation coordinate position on the interface of described formed objects when recording.
Therefore, this step can screen characteristic point as validity feature point according to the distance between described operation coordinate from described characteristic point.The excessive characteristic point of distance being about between described operation coordinate filters out, and retains and characteristic point that distance between described operation coordinate is less.Such as, the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point can be calculated respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
In step S160, calculate the match point of described validity feature point and described operation coordinate, operation coordinate during using the barycenter of computed match point as playback.
After the step S150 distant characteristic point excluded between described operation coordinate, remaining characteristic point, the match point of described validity feature point and described operation coordinate can be calculated, the probability of operation coordinate when these match point correspondences are recorded is all very big, therefore using the barycenter of computed match point as operation coordinate when playing back, correct operation object accuracy rate can be chosen during maximum of probability ground raising playback operation.
The technical scheme of the present embodiment is in automatic test, by respectively the operation screen recorded and play back being zoomed in and out same ratio with before playback before script is recorded respectively, different resolution picture is made to have the distribution of similar icon in the process recorded and play back.And during handle icon when determining playback, with reference to operation coordinate when recording, effectively to filter from realistic objective noise farther out, the accuracy of handle icon when determining playback can be improved.
Fig. 2 is the flow chart that calibration method is sat in a kind of operation determined when automatic test script plays back according to an exemplary embodiment, includes as in figure 2 it is shown, calibration method is sat in the operation when determination automatic test script described in the present embodiment plays back:
In step S210, recording before test script, zooming to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment.
This step is identical with the step S110 of a upper embodiment, and therefore not to repeat here.
In step S220, when recording test script, the operation coordinate on described first graphic user interface when record is recorded, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface.
It should be noted that, to those skilled in the art, multiple method can be adopted, for instance can intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on rim detection according to the handle icon that described operation coordinate intercepts recording operation corresponding on described first graphic user interface.
Fig. 3 is the flow chart of a kind of method intercepting recording operation handle icon corresponding on the first graphic user interface based on rim detection according to operation coordinate according to an exemplary embodiment, as it is shown on figure 3, the above-mentioned method based on rim detection comprises the steps that
In step S310, Laplce's second-order differential is adopted to sharpen the edge of the first graphic user interface described in operator identification.
Laplacian operator utilizes second dervative information, has isotropism, namely unrelated with change in coordinate axis direction, and after coordinate axes rotates, gradient result is constant.Make image after second-order differential, produce a precipitous zero cross point in edge, according to this, zero cross point is judged edge.
In step s 320, from the edge identified, widthwise edge and longitudinal edge are screened.
The edge pixel identified is added filtration, only retains widthwise edge and longitudinal edge.The acquisition methods on transverse and longitudinal limit, mainly by judging whether x, y-axis exist intensive zero cross point (allowing there is certain pixel Discrete Distribution), when the zero cross point of these dense distribution is more than there is a horizontal stroke/longitudinal edge on certain threshold values (empirical value) then this x/y axle any.
In step S330, expand outwardly centered by described operation coordinate, judge whether that two axial symmetry limits are distributed in described operation coordinate both sides and to described operation coordinate distance less than first threshold, if existing, obtain rectangle frame according to described two axial symmetry limits, using the described rectangle frame scope as described handle icon.
After drawing all of transverse and longitudinal limit, when obtaining click coordinate.Then expand outwardly centered by click coordinate, it may be judged whether there is two " axial symmetry limits " (two horizontal strokes or two are indulged, it is allowed to there is discrete and deviant in a small amount) and be centered around operation nearby coordinates.If existing, take this two limit one rectangle frames of composition, in this, as the scope of handle icon.
In step S230, playing back before described test script, zooming to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment.
Similar with step S210, such as, this step realizes also by the following manner: obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
Specifically, in the present embodiment step S210 and step S230, image (graphic user interface) being zoomed in and out, can adopt multiple method, this is not construed as limiting by the present embodiment.Such as image is zoomed in and out by available bilinear interpolation algorithm.In the bilinear interpolation interpolator arithmetic of image, new-create pixel value in target image, is be weighted by average computation by the value of the source images position 4 adjacent pixels in 2*2 region near it to draw.Specifically comprise the following steps that
The first step: calculate source images and target image width and high ratio；
Second step: (x, y), calculates respective coordinates in source images, and result is floating number for a point of target image；
3rd step: seek the weight ratio around shared by four points；
4th step: be filled on target image after trying to achieve value.
In step S240, described second graph user interface is carried out image darkening process.
Obtaining the HSB value that in described second graph user interface, the rgb value of each pixel is corresponding, wherein H (hues) represents that form and aspect, S (saturation) represent saturation, and B (brightness) represents brightness.Judge that whether the brightness value of pixel is lower than the 3rd threshold value, will be set to black lower than the color of the pixel of described 3rd threshold value in described second graph user interface according to described HSB value.Before playback described second graph user interface is carried out image darkening process through this step, by the rgb value of pixel each in image is converted into HSB value, it is known that the brightness value of each pixel, it is thus possible to reduce the low brightness pixel noise impact when image recognition.
It should be noted that operating procedure when following steps S250 to step S270 belongs to play back described test script.
In step s 250, when playing back described test script, the characteristic point of described handle icon and described second graph user interface is calculated.
This step can pass through accomplished in many ways, for instance can realize based on SIFT algorithm, SURF algorithm, the marginal point algorithm of wavelet transformation or Harris Corner Detection Algorithm.
Specifically comprise the steps that
Method one, calculate the SIFT feature point of described handle icon and described second graph user interface based on SIFT algorithm.
Method two, calculate the SURF characteristic point of described handle icon and described second graph user interface based on SURF algorithm.
Method three, marginal point algorithm based on wavelet transformation calculate the marginal point of described handle icon and described second graph user interface.
Method four, calculate the angle point of described handle icon and described second graph user interface based on Harris Corner Detection Algorithm.
For method one, the method can be realized by following steps:
The first step, utilizes Gaussian convolution to check described handle icon and processes, obtain multiscale space image；
Second step, carries out difference of Gaussian process to multiscale space image, builds Gaussian difference scale space image；
3rd step, detects the Local Extremum of this Gaussian difference scale space image, then utilizes matching three-dimensional quadratic function that described extreme point is accurate to sub-pixel, and adopts threshold method and Hessian matrix method screening characteristic point, obtains SIFT feature point set.
In step S260, from described characteristic point, screen characteristic point as validity feature point according to the distance between described operation coordinate.
Such as, the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point can be calculated respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
In step S270, calculate the match point of described validity feature point and described operation coordinate, operation coordinate during using the barycenter of computed match point as playback.
Such as, take the characteristic point A of described handle icon, calculate, by traveling through picture to be matched, the Euclidean distance that all of validity feature point is corresponding with characteristic point A, if distance is less than default Second Threshold, be then judged to that this point is the match point of an A.
Owing to each characteristic point has the characteristic vector of 128 dimensions, if all characteristic points of traversal icon and picture to be matched judge whether that coupling will produce the matching operation time unnecessary in a large number, utilize step S260 that pictorial feature point to be matched is carried out effectiveness screening, the traversal to invalid match point can be reduced, save and calculate these points and the time of the Euclidean distance of characteristic point A, computational efficiency can be improved.
Fig. 4 is the flow chart of the method for a kind of automatic test playback phase according to an exemplary embodiment.As shown in Figure 4, the method for the automatic test playback phase described in the present embodiment includes:
In step S410, operation screen normalization.
Playing back before described test script, zoom to described pre-set dimension by being test for the second graphic user interface moving equipment, with record before equivalently-sized by what be test for after the first graphic user interface convergent-divergent moving equipment.
In the step s 420, the icon intercepted is recorded in input.
The handle icon that this recording operation is corresponding is intercepted according to the operation coordinate on the first graphic user interface after scaling when recording；
In step S430, calculate the characteristic point recording the operation screen after intercepting icon and normalization.
This step can pass through accomplished in many ways, for instance can realize based on SIFT algorithm, SURF algorithm, the marginal point algorithm of wavelet transformation or Harris Corner Detection Algorithm.
Specifically comprise the steps that
Method one, calculate the SIFT feature point of described handle icon and described second graph user interface based on SIFT algorithm.
Method two, calculate the SURF characteristic point of described handle icon and described second graph user interface based on SURF algorithm.
Method three, marginal point algorithm based on wavelet transformation calculate the marginal point of described handle icon and described second graph user interface.
Method four, calculate the angle point of described handle icon and described second graph user interface based on Harris Corner Detection Algorithm.
In step S440, on operation screen, determining whether validity feature point by recording operation coordinate, if then performing step S460, otherwise performing step S450.
In step S450, this characteristic point is not involved in matching operation, skips this characteristic point return step S440 and carries out the judgement of next characteristic point, until all characteristic points have judged to perform step S480.
In step S460, the coordinate of record validity feature point.
In step S470, coupling is recorded intercepting icon characteristics point and is drawn match point with validity feature point, carries out the judgement of next characteristic point, until all characteristic points have judged to perform step S480.
In step S480, calculate all match point barycenter, as playback operation coordinate
Fig. 5 is the block diagram of the device of a kind of operation coordinate determined when automatic test script plays back according to an exemplary embodiment, as it is shown in figure 5, the device of the operation coordinate when determination automatic test script described in the present embodiment plays back includes recording pretreatment unit 510, script recording elements 520, playback pretreatment unit 530 and script playback unit 540.
This recording pretreatment unit 510 is configured to recording before test script, zooms to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment.
This script recording elements 520 is configured to when recording test script, the operation coordinate on described first graphic user interface when record is recorded, and intercepts, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface
This playback pretreatment unit 530 is configured to playing back before described test script, zooms to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment.
This script playback unit 540 is configured to, including characteristic point computation subunit 541, validity feature point screening subelement 542 and operation coordinate computation subunit 543.
Described characteristic point computation subunit 541, for when playing back described test script, calculating the characteristic point of described handle icon and described second graph user interface；
Described validity feature point screening subelement 542, screens characteristic point as validity feature point for the distance between basis and described operation coordinate from described characteristic point；
Described operation coordinate computation subunit 543, the operation coordinate for calculating the match point of described validity feature point and described operation coordinate, during using the barycenter of computed match point as playback.
Further, described recording pretreatment unit 510 is used for, obtain the boundary parameter being test for the first graphic user interface moving equipment, according to described boundary parameter, described first graphic user interface moving equipment is zoomed in and out according to pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device recorded as the first graphic user interface.
Further, described playback pretreatment unit 530 is used for, obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, the described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
Further, described script recording elements 520 is used for, and intercepts, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on rim detection.
Further, described characteristic point computation subunit 541 is used for:
The SIFT feature point of described handle icon and described second graph user interface is calculated based on SIFT algorithm；Or
The SURF characteristic point of described handle icon and described second graph user interface is calculated based on SURF algorithm；Or
The marginal point of described handle icon and described second graph user interface is calculated based on the marginal point algorithm of wavelet transformation；Or
The angle point of described handle icon and described second graph user interface is calculated based on Harris corner detection approach.
Further, described validity feature point screening subelement 542 is used for, calculate the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
Further, described playback pretreatment unit 530 is additionally operable to, and before playing back described test script, described second graph user interface is carried out image darkening process.
Further, described playback pretreatment unit 530 is used for, obtain the HSB value that in described second graph user interface, the rgb value of each pixel is corresponding, judge that whether the brightness value of pixel is lower than the 3rd threshold value, will be set to black lower than the color of the pixel of described 3rd threshold value in described second graph user interface according to described HSB value.
About the device in above-described embodiment, the concrete mode that wherein unit performs to operate has been described in detail in about the embodiment of the method, and explanation will be not set forth in detail herein.
Calibration method is sat in operation during the determination automatic test script that device can perform the embodiment of the present invention one and embodiment two the provides playback of the operation coordinate during determination automatic test script playback that the present embodiment provides, possesses the corresponding functional module of execution method and beneficial effect.
Fig. 6 is the block diagram of the device of a kind of operation coordinate determined when automatic test script plays back according to an exemplary embodiment, and as shown in Figure 6, the device of the operation coordinate when determination automatic test script described in the present embodiment plays back includes:
The device of the operation coordinate when determination automatic test script described in the present embodiment plays back includes recording pretreatment unit 610, script recording elements 620, playback pretreatment unit 630 and script playback unit 640.
This recording pretreatment unit 610 is configured to recording before test script, zooms to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment.
This script recording elements 620 is configured to, for when recording test script, operation coordinate on described first graphic user interface when record is recorded, intercepts, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on rim detection.This script recording elements 620 also includes limb recognition subelement 621, transverse and longitudinal limit screening subelement 622 and icon scope and obtains subelement 623；
Described limb recognition subelement 621, for adopting Laplce's second-order differential to sharpen the edge of the first graphic user interface described in operator identification；
Described transverse and longitudinal limit screening subelement 622, for screening widthwise edge and longitudinal edge from the edge identified；
Described icon scope obtains subelement 623, for expanding outwardly centered by described operation coordinate, judge whether that two axial symmetry limits are distributed in described operation coordinate both sides and to described operation coordinate distance less than first threshold, if existing, obtain rectangle frame according to described two axial symmetry limits, using the described rectangle frame scope as described handle icon.
This playback pretreatment unit 630 is configured to playing back before described test script, zooms to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment.
This script playback unit 640 is configured to, including characteristic point computation subunit 641, validity feature point screening subelement 642 and operation coordinate computation subunit 643.
Described characteristic point computation subunit 641, for when playing back described test script, calculating the characteristic point of described handle icon and described second graph user interface；
Described validity feature point screening subelement 642, screens characteristic point as validity feature point for the distance between basis and described operation coordinate from described characteristic point；
Described operation coordinate computation subunit 643, the operation coordinate for calculating the match point of described validity feature point and described operation coordinate, during using the barycenter of computed match point as playback.
Further, described recording pretreatment unit 610 is used for, obtain the boundary parameter being test for the first graphic user interface moving equipment, according to described boundary parameter, described first graphic user interface moving equipment is zoomed in and out according to pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device recorded as the first graphic user interface.
Further, described playback pretreatment unit 630 is used for, obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, the described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
Further, described characteristic point computation subunit 641 is used for:
The SIFT feature point of described handle icon and described second graph user interface is calculated based on SIFT algorithm；Or
The SURF characteristic point of described handle icon and described second graph user interface is calculated based on SURF algorithm；Or
The marginal point of described handle icon and described second graph user interface is calculated based on the marginal point algorithm of wavelet transformation；Or
The angle point of described handle icon and described second graph user interface is calculated based on Harris corner detection approach.
Further, described validity feature point screening subelement 641 is used for, calculate the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
Further, described playback pretreatment unit 630 is additionally operable to, and before playing back described test script, described second graph user interface is carried out image darkening process.
Further, described playback pretreatment unit 630 is used for, obtain the HSB value that in described second graph user interface, the rgb value of each pixel is corresponding, judge that whether the brightness value of pixel is lower than the 3rd threshold value, will be set to black lower than the color of the pixel of described 3rd threshold value in described second graph user interface according to described HSB value.
About the device in above-described embodiment, the concrete mode that wherein unit performs to operate has been described in detail in about the embodiment of the method, and explanation will be not set forth in detail herein.
Calibration method is sat in operation during the determination automatic test script that device can perform the embodiment of the present invention one and embodiment two the provides playback of the operation coordinate during determination automatic test script playback that the present embodiment provides, possesses the corresponding functional module of execution method and beneficial effect.
Those skilled in the art, after considering description and putting into practice invention disclosed herein, will readily occur to other embodiment of the present invention.The application is intended to any modification of the present invention, purposes or adaptations, and these modification, purposes or adaptations are followed the general principle of the present invention and include the undocumented known general knowledge in the art of the disclosure or conventional techniques means.Description and embodiments is considered only as exemplary, and the true scope of the present invention and spirit are pointed out by claim below.
It should be appreciated that the invention is not limited in precision architecture described above and illustrated in the accompanying drawings, and various amendment and change can carried out without departing from the scope.The scope of the present invention is only limited by appended claim.
Claims (18)
1. calibration method is sat in the operation that a kind is determined when automatic test script plays back, it is characterised in that including:
Recording before test script, zooming to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment；
When recording test script, the operation coordinate on described first graphic user interface when record is recorded, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface；
Playing back before described test script, zooming to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment；
When playing back described test script:
Calculate the characteristic point of described handle icon and described second graph user interface；
From described characteristic point, characteristic point is screened as validity feature point according to the distance between described operation coordinate；
Calculate the match point of described validity feature point and described operation coordinate, operation coordinate during using the barycenter of computed match point as playback.
2. method according to claim 1, it is characterised in that zoom to pre-set dimension and obtain the operation of the first graphic user interface by being test for the first graphic user interface moving equipment and include:
Obtain the boundary parameter being test for the first graphic user interface moving equipment, according to described boundary parameter, described first graphic user interface moving equipment is zoomed in and out according to pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device recorded as the first graphic user interface.
3. method according to claim 1 and 2, it is characterised in that zoom to described pre-set dimension and obtain the operation of second graph user interface by being test for the second graphic user interface moving equipment and include:
Obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, the described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
4. method according to claim 1, it is characterized in that, the operation intercepting recording operation handle icon corresponding on described first graphic user interface according to described operation coordinate includes: intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on rim detection.
5. method according to claim 4, it is characterised in that the operation intercepting recording operation handle icon corresponding on described first graphic user interface according to described operation coordinate based on rim detection includes:
Laplce's second-order differential is adopted to sharpen the edge of the first graphic user interface described in operator identification；
Widthwise edge and longitudinal edge is screened from the edge identified；
Expand outwardly centered by described operation coordinate, judge whether that two axial symmetry limits are distributed in described operation coordinate both sides and to described operation coordinate distance less than first threshold, if existing, obtain rectangle frame according to described two axial symmetry limits, using the described rectangle frame scope as described handle icon.
6. method according to claim 1, it is characterised in that the operation of the characteristic point calculating described handle icon and described second graph user interface includes:
The SIFT feature point of described handle icon and described second graph user interface is calculated based on SIFT algorithm；Or
The SURF characteristic point of described handle icon and described second graph user interface is calculated based on SURF algorithm；Or
The marginal point of described handle icon and described second graph user interface is calculated based on the marginal point algorithm of wavelet transformation；Or
The angle point of described handle icon and described second graph user interface is calculated based on Harris Corner Detection Algorithm.
7. method according to claim 1, it is characterised in that screen characteristic point from described characteristic point according to the distance between described operation coordinate and include as the operation of validity feature point:
Calculate the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
8. method according to claim 1, it is characterised in that described second graph user interface is carried out image darkening process before being additionally included in the described test script of playback by described method.
9. method according to claim 8, it is characterised in that the operation that described second graph user interface carries out image darkening process includes:
Obtain the HSB value that in described second graph user interface, the rgb value of each pixel is corresponding, judge that whether the brightness value of pixel is lower than the 3rd threshold value, will be set to black lower than the color of the pixel of described 3rd threshold value in described second graph user interface according to described HSB value.
10. the device of the operation coordinate that a kind is determined when automatic test script plays back, it is characterised in that including:
Record pretreatment unit, for recording before test script, zoom to pre-set dimension obtain the first graphic user interface by being test for the first graphic user interface moving equipment；
Script recording elements, for when recording test script, the operation coordinate on described first graphic user interface when record is recorded, according to the handle icon that described operation coordinate intercepting recording operation is corresponding on described first graphic user interface；
Playback pretreatment unit, for playing back before described test script, zooms to described pre-set dimension obtain second graph user interface by being test for the second graphic user interface moving equipment；
Script playback unit, including characteristic point computation subunit, validity feature point screening subelement and operation coordinate computation subunit；
Described characteristic point computation subunit, for when playing back described test script, calculating the characteristic point of described handle icon and described second graph user interface；
Described validity feature point screening subelement, screens characteristic point as validity feature point for the distance between basis and described operation coordinate from described characteristic point；
Described operation coordinate computation subunit, the operation coordinate for calculating the match point of described validity feature point and described operation coordinate, during using the barycenter of computed match point as playback.
11. device according to claim 10, it is characterized in that, described recording pretreatment unit is used for, obtain the boundary parameter being test for the first graphic user interface moving equipment, according to described boundary parameter, described first graphic user interface moving equipment is zoomed in and out according to pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device recorded as the first graphic user interface.
12. the device according to claim 10 or 11, it is characterized in that, described playback pretreatment unit is used for, obtain the boundary parameter being test for the second graphic user interface moving equipment, according to described boundary parameter, the described second graphic user interface moving equipment is zoomed in and out according to described pre-set dimension, using the graphical user interface displays after convergent-divergent on the display interface for the device played back as second graph user interface.
13. device according to claim 10, it is characterised in that described script recording elements is used for, intercept, according to described operation coordinate, the handle icon that recording operation is corresponding on described first graphic user interface based on rim detection.
14. device according to claim 13, it is characterised in that described script recording elements includes limb recognition subelement, transverse and longitudinal limit screening subelement and icon scope and obtains subelement；
Described limb recognition subelement, for adopting Laplce's second-order differential to sharpen the edge of the first graphic user interface described in operator identification；
Described transverse and longitudinal limit screening subelement, for screening widthwise edge and longitudinal edge from the edge identified；
Described icon scope obtains subelement, for expanding outwardly centered by described operation coordinate, judge whether that two axial symmetry limits are distributed in described operation coordinate both sides and to described operation coordinate distance less than first threshold, if existing, obtain rectangle frame according to described two axial symmetry limits, using the described rectangle frame scope as described handle icon.
15. device according to claim 10, it is characterised in that described characteristic point computation subunit is used for:
The SIFT feature point of described handle icon and described second graph user interface is calculated based on SIFT algorithm；Or
The SURF characteristic point of described handle icon and described second graph user interface is calculated based on SURF algorithm；Or
The marginal point of described handle icon and described second graph user interface is calculated based on the marginal point algorithm of wavelet transformation；Or
The angle point of described handle icon and described second graph user interface is calculated based on Harris Corner Detection Algorithm.
16. device according to claim 10, it is characterized in that, described validity feature point screening subelement is used for, calculate the Euclidean distance between each characteristic point and described operation coordinate in described characteristic point respectively, using Euclidean distance computed in described characteristic point less than the characteristic point of Second Threshold as validity feature point.
17. device according to claim 10, it is characterised in that described playback pretreatment unit is additionally operable to, before playing back described test script, described second graph user interface is carried out image darkening process.
18. device according to claim 17, it is characterized in that, described playback pretreatment unit is used for, obtain the HSB value that in described second graph user interface, the rgb value of each pixel is corresponding, judge that whether the brightness value of pixel is lower than the 3rd threshold value, will be set to black lower than the color of the pixel of described 3rd threshold value in described second graph user interface according to described HSB value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610124617.3A CN105740874B (en) | 2016-03-04 | 2016-03-04 | Determine the method and device of operation coordinate when automatic test script playback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610124617.3A CN105740874B (en) | 2016-03-04 | 2016-03-04 | Determine the method and device of operation coordinate when automatic test script playback |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105740874A true CN105740874A (en) | 2016-07-06 |
| CN105740874B CN105740874B (en) | 2019-04-16 |
Family
ID=56249985
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610124617.3A Active CN105740874B (en) | 2016-03-04 | 2016-03-04 | Determine the method and device of operation coordinate when automatic test script playback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105740874B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107274442A (en) * | 2017-07-04 | 2017-10-20 | 北京云测信息技术有限公司 | A kind of image-recognizing method and device |
| CN109871172A (en) * | 2019-01-23 | 2019-06-11 | 广州杰赛科技股份有限公司 | Mouse click method, device and readable storage medium storing program for executing in a kind of automatic test |
| CN110046009A (en) * | 2019-02-19 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Method for recording, device, server and readable storage medium storing program for executing |
| CN110647321A (en) * | 2019-08-15 | 2020-01-03 | 平安普惠企业管理有限公司 | Method, device and equipment for playing back operation flow and storage medium |
| CN112685279A (en) * | 2019-10-17 | 2021-04-20 | 腾讯科技（深圳）有限公司 | Script recording method and device and terminal equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040230327A1 (en) * | 2003-05-15 | 2004-11-18 | Fisher-Rosemount Systems, Inc. | Field maintenance tool with enhanced scripts |
| CN104866414A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Method, device and system for testing application program |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
| CN105094385A (en) * | 2015-06-30 | 2015-11-25 | 北京奇虎科技有限公司 | Method, device and system for remote control |
| CN105138463A (en) * | 2015-09-25 | 2015-12-09 | 成都中科创达软件有限公司 | Testing method and system for mobile terminal software |
-
2016
- 2016-03-04 CN CN201610124617.3A patent/CN105740874B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040230327A1 (en) * | 2003-05-15 | 2004-11-18 | Fisher-Rosemount Systems, Inc. | Field maintenance tool with enhanced scripts |
| CN104866414A (en) * | 2014-02-24 | 2015-08-26 | 腾讯科技（深圳）有限公司 | Method, device and system for testing application program |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
| CN105094385A (en) * | 2015-06-30 | 2015-11-25 | 北京奇虎科技有限公司 | Method, device and system for remote control |
| CN105138463A (en) * | 2015-09-25 | 2015-12-09 | 成都中科创达软件有限公司 | Testing method and system for mobile terminal software |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107274442A (en) * | 2017-07-04 | 2017-10-20 | 北京云测信息技术有限公司 | A kind of image-recognizing method and device |
| CN107274442B (en) * | 2017-07-04 | 2020-03-10 | 北京云测信息技术有限公司 | Image identification method and device |
| CN109871172A (en) * | 2019-01-23 | 2019-06-11 | 广州杰赛科技股份有限公司 | Mouse click method, device and readable storage medium storing program for executing in a kind of automatic test |
| CN110046009A (en) * | 2019-02-19 | 2019-07-23 | 阿里巴巴集团控股有限公司 | Method for recording, device, server and readable storage medium storing program for executing |
| CN110046009B (en) * | 2019-02-19 | 2022-08-23 | 创新先进技术有限公司 | Recording method, recording device, server and readable storage medium |
| CN110647321A (en) * | 2019-08-15 | 2020-01-03 | 平安普惠企业管理有限公司 | Method, device and equipment for playing back operation flow and storage medium |
| CN112685279A (en) * | 2019-10-17 | 2021-04-20 | 腾讯科技（深圳）有限公司 | Script recording method and device and terminal equipment |
| CN112685279B (en) * | 2019-10-17 | 2024-02-20 | 深圳市腾讯网域计算机网络有限公司 | Script recording method, script recording device and terminal equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105740874B (en) | 2019-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101722803B1 (en) | Method, computer program, and device for hybrid tracking of real-time representations of objects in image sequence | |
| CN105740874A (en) | Method and device for determining operation coordinate of automation test script during playback | |
| JP3735344B2 (en) | Calibration apparatus, calibration method, and calibration program | |
| US9922443B2 (en) | Texturing a three-dimensional scanned model with localized patch colors | |
| CN107945111B (en) | Image stitching method based on SURF (speeded up robust features) feature extraction and CS-LBP (local binary Pattern) descriptor | |
| CN111161313B (en) | Multi-target tracking method and device in video stream | |
| CN101802842A (en) | System and method for identifying complex tokens in an image | |
| JP2021168143A (en) | System and method for efficiently scoring probe in image by vision system | |
| CN112364865B (en) | Method for detecting small moving target in complex scene | |
| CN106062824A (en) | Edge detection device, edge detection method, and program | |
| Vu et al. | Efficient hybrid tree-based stereo matching with applications to postcapture image refocusing | |
| Okumura et al. | Augmented reality based on estimation of defocusing and motion blurring from captured images | |
| Agrawal et al. | Dense haze removal by nonlinear transformation | |
| CN111105452A (en) | High-low resolution fusion stereo matching method based on binocular vision | |
| CN109658366A (en) | Based on the real-time video joining method for improving RANSAC and dynamic fusion | |
| Chen et al. | A color-guided, region-adaptive and depth-selective unified framework for Kinect depth recovery | |
| US20220245851A1 (en) | Associating device coordinate systems in a multi-person augmented reality system | |
| CN111062895A (en) | Microscopic image restoration method based on multi-view-field segmentation | |
| Aydin et al. | A New Adaptive Focus Measure for Shape From Focus. | |
| Tseng et al. | Depth image super-resolution via multi-frame registration and deep learning | |
| CN113240656A (en) | Visual positioning method and related device and equipment | |
| CN110717910B (en) | CT image target detection method based on convolutional neural network and CT scanner | |
| CN105069764B (en) | A kind of image de-noising method and system based on Edge track | |
| CN113284081B (en) | Depth map super-resolution optimization method and device, processing equipment and storage medium | |
| WO2021110051A1 (en) | Method and system for associating device coordinate systems in a multi‐person ar system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |