CN114445652A - Element identification method in application, application testing method and related hardware - Google Patents
Element identification method in application, application testing method and related hardware Download PDFInfo
- Publication number
- CN114445652A CN114445652A CN202111584833.3A CN202111584833A CN114445652A CN 114445652 A CN114445652 A CN 114445652A CN 202111584833 A CN202111584833 A CN 202111584833A CN 114445652 A CN114445652 A CN 114445652A
- Authority
- CN
- China
- Prior art keywords
- target application
- icon
- interface
- application
- matting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3696—Methods or tools to render software testable
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides an identification method of elements in application, an application testing method and related hardware. The method comprises the following steps: and carrying out interface screenshot on the target application to obtain an interface image of the target application. And carrying out element matting on the interface image of the target application to obtain an element icon in the target application. And inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element classification label corresponding to the sample element icon. The scheme of the embodiment of the invention does not depend on the code attribute information corresponding to the elements to immediately complete element identification, and even if the application continuously iterates to update, excessive adjustment is not needed, thereby effectively reducing the difficulty of later maintenance.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to an identification method of an element in an application, an application testing method, and related hardware.
Background
When the terminal application is subjected to non-manual testing, the program is required to start the functions of elements in the application interface so as to complete the test items. Thus, mechanically identifying elements in an application is a prerequisite for non-manual testing.
An existing way to identify elements in an application is to extract attribute information of each positional element (such as some special field values in the code) from the UI (interface design) framework of the application for element matching. However, the terminal application iteration is fast, the element matching rule needs to be frequently adjusted due to frequent change of codes, and great inconvenience is brought to daily maintenance.
For this reason, how to identify an element mechanically without depending on or deeply depending on attribute information of the element is a technical problem which needs to be solved at present.
Disclosure of Invention
The embodiment of the invention aims to provide an identification method of elements in application, an application test method and related hardware, which can mechanically identify the elements in an application interface on the premise of not depending on attribute information of the elements so as to further develop application related tests.
To solve the above technical problem, the embodiment of the present invention is implemented as follows:
in a first aspect, a method for identifying an element in an application is provided, including:
performing interface screenshot on a target application to obtain an interface image of the target application;
carrying out element matting on the interface image of the target application to obtain an element icon in the target application;
and inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element classification label corresponding to the sample element icon.
In a second aspect, an application testing method is provided, including:
performing interface screenshot on a target application to obtain an interface image of the target application;
carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface;
inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element category label labeled on the sample element icon;
and inputting the information of the identified elements in the target application into a test script of the target application, and testing corresponding functions of the identified elements based on the test script.
In a third aspect, an apparatus for identifying an element in an application interface is provided, including:
the screenshot module is used for carrying out interface screenshot on the target application to obtain an interface image of the target application;
the matting module is used for carrying out element matting on the interface image of the target application to obtain an element icon in the target application;
the identification module is used for inputting the element icons in the target application into a preset element identification model and identifying the elements in the target application, wherein the element identification model is obtained by training based on the sample element icons and the element classification labels corresponding to the sample element icons.
In a fourth aspect, an application testing apparatus is provided, including:
the screenshot module is used for carrying out interface screenshot on the target application to obtain an interface image of the target application;
the matting module is used for carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface;
the identification module is used for inputting the element icons in the target application into a preset element identification model and identifying the elements in the target application, wherein the element identification model is obtained by training based on sample element icons and element category labels labeled on the sample element icons;
and the test module is used for inputting the information of the identified elements in the target application into the test script of the target application and testing the corresponding functions of the identified elements on the basis of the test script.
In a fifth aspect, an electronic device is provided, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the method of the first aspect or to perform the method of the second aspect.
In a sixth aspect, a computer readable storage medium is provided, which stores one or more programs which, when executed by an electronic device comprising a plurality of applications, perform the method of the first aspect or the method of the second aspect.
According to the scheme of the embodiment of the invention, an element identification model for carrying out element identification based on an image is trained through the sample element icons and the corresponding color element classification labels. When the elements in the target application need to be identified, the interface of the target application is subjected to element matting, and the element matting obtained by matting is input into an element identification model to immediately complete identification. Because the identification mode does not depend on the code attribute information corresponding to the elements, even if the application continuously iterates to update, excessive adjustment is not needed, and the difficulty of later maintenance is effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for identifying an element in an application according to an embodiment of the present invention.
Fig. 2 is a first schematic diagram of an application interface screenshot by a method for identifying an element in an application according to an embodiment of the present invention.
Fig. 3 is a second schematic diagram of an application interface screenshot by the method for identifying an element in an application according to the embodiment of the present invention.
Fig. 4 is a third schematic diagram of a screenshot of an application interface according to an identification method of an element in an application provided in an embodiment of the present invention.
Fig. 5 is a fourth schematic diagram of a screenshot of an application interface according to the method for identifying an element in an application according to the embodiment of the present invention.
Fig. 6 is a schematic structural diagram of model output data of an identification method for an element in an application according to an embodiment of the present invention.
FIG. 7 is a schematic flow chart of an application testing method provided in an embodiment of the present invention,
Fig. 8 is a schematic structural diagram of an apparatus for identifying an element in an application according to an embodiment of the present invention.
Fig. 9 is a schematic structural diagram of an application testing apparatus according to an embodiment of the present invention.
Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step should fall within the scope of protection of the present specification.
As previously mentioned, mechanically identifying elements in an application is a prerequisite for non-manual application test development. An existing way to identify elements in an application is to extract attribute information of each positional element (such as some special field values in the code) from the UI (interface design) framework of the application for element matching. However, the terminal application iteration is fast, the element matching rule needs to be frequently adjusted due to frequent change of codes, and great inconvenience is brought to daily maintenance. Therefore, the application aims to provide a scheme for realizing mechanical identification of elements in the application independent of attribute information of the elements and a scheme for developing application tests based on the identified elements.
Fig. 1 is a flowchart of an element identification method in an application according to an embodiment of the present invention, which specifically includes the following steps:
and S102, performing interface screenshot on the target application to obtain an interface image of the target application.
It should be understood that the interface image described herein contains icons of elements in the application, and that normal use of the elements is one of the contents tested as a target application.
And S104, carrying out element matting on the interface image of the target application to obtain an element icon in the target application.
The element icon can be obtained based on any image matting method in the step. Taking element tree positioning method matting as an example, the coordinates and the size of the element in the target application can be obtained from the UI frame information of the target application, and element matting is performed on the interface image of the target application according to the coordinates and the size of the element in the target application.
And S106, inputting the element icons in the target application into a preset element identification model, and identifying the elements in the target application, wherein the element identification model is obtained by training based on the sample element icons and the element classification labels corresponding to the sample element icons.
It should be understood that the sample element icon contains the image features of the sample element icon. And inputting the sample element icon into the element recognition model, so as to obtain a training result output by the element recognition model. The training result is a prediction result given by the current stage of the element identification model, and a true value result corresponding to the element classification label has an error. Here, the loss function of the element recognition model can be derived using a maximum likelihood estimation algorithm. Then, based on the loss function, the error between the training result and the true value result corresponding to the element classification label is calculated, and parameters (such as weights corresponding to the characteristic factors) in the element identification model are adjusted to achieve the training effect by taking the error reduction as the aim.
It should be noted that, the element identification model that can be used herein is not exclusive, but all the learning models that support image data input and have classification capability can be applied to the solution of the embodiment of the present invention, and the present invention is not limited in particular.
It should be understood that in practical applications, the samples may also be preprocessed according to the structural requirements of the learning model on the input data. Here, the YOLOV5 model is taken as an example. The YOLOV5 model requires a combination of five-dimensional feature vectors for the structure of the tag data, and correspondingly, if the YOLOV5 model is adopted in the embodiment of the present application, the element classification tag also needs to adopt five-dimensional information for adaptation, and as an exemplary introduction, the five-dimensional information of the element classification tag may include:
1) the category identification corresponding to the sample element icon;
2) the ratio of the horizontal coordinate value of the upper left corner of the effective area in the sample element icon to the width value of the whole element icon,
3) the ratio of the vertical coordinate value of the upper left corner of the effective area in the sample element icon to the height value of the whole element icon;
4) a ratio of a width value of an effective area in the sample element icon to a width value of the entire element icon;
5) the ratio of the height of the active area in the sample element icon to the height of the entire element icon.
The effective area in the element icon refers to an area where the element icon includes element rendering pixels, and an area not including the element rendering pixels is regarded as other areas except the effective area.
It should be appreciated that the trained sample element icons are capable of identifying elements based on image features. That is, after the element icon of the matting target application is input to the element identification model in this step, the element identification model can provide the identification result of the element (category). After the elements are recognized, the corresponding functions of the elements are also known information, so that automatic testing of the target application can be realized based on the program.
Therefore, the method provided by the embodiment of the invention trains the element identification model for identifying the elements based on the image through the sample element icons and the corresponding color element classification labels. When the elements in the target application need to be identified, the interface of the target application is subjected to element matting, and the element matting obtained by matting is input into an element identification model to immediately complete identification. Because the identification mode does not depend on the code attribute information corresponding to the elements, even if the application continuously iterates to update, excessive adjustment is not needed, and the difficulty of later maintenance is effectively reduced.
The following describes the method for identifying elements in the application of this embodiment in detail with reference to an actual application scenario.
The application scenario of the scheme is based on a YOLOV5 model to identify elements in the target application, and the specific scheme comprises a model training phase and a model application phase.
Wherein, the steps of the model training stage are as follows:
step 1: the interface image of the sample application is automatically intercepted by using the prior art, and the interface image shown in fig. 1 is obtained.
The Android end uses a screenshot interface of an UIAutomator framework, and the iOS end uses a screenshot interface of a WebDriverAgent framework.
Step 2: elements in the interface picture are scratched by using a manual or automatic scratching technology to obtain an element icon as shown in fig. 3.
Specifically, in this step, an element positioned by an element tree may be used for automatic matting, an Android end uses an uiautomation frame, an iOS end uses a WebDriverAgent frame, the element is positioned by an attribute of the element in the element tree, coordinates and a size of the element are obtained, and then static icons including static icons such as camera, red _ packet, new _ weibo, close, forward, comment, and toggle _ up are segmented in a full screen screenshot by using the coordinates and the size through an interface of an image processing open source library opencv. Finally, the 11 icons shown in fig. 3 are obtained in sequence: camera. png, red _ packet. png, new _ weibo.png, topic. png, at.png short _ link. png, video _ link. png, close. png, forward. png, comment. png and thumb _ up. png.
And step 3: the method includes the steps of self-defining element icon type numbers, marking that camera is 0, red _ packet is 1, new _ weibo is 2, topic is 3, at is 4, short _ link is 5, video _ link is 6, close is 7, forward is 8, comment is 9 and stem _ up is 10, and generating element classification labels applicable to a YOLOV5 model according to position data of each element icon, wherein in the element classification labels, the first dimension is an element icon index number, the index number reflects the cutout sequence of the element icon, and the element classification labels can be used as category identification of element classification in the example; the second dimension is the ratio of the horizontal coordinate value of the upper left corner of the effective area of the element icon to the width value of the whole element icon; the third dimension is the ratio of the vertical coordinate value of the upper left corner of the effective area of the element icon to the height value of the whole element icon, and the fourth dimension is the ratio of the width value of the effective area of the element icon to the width value of the whole element icon; the fifth dimension is the token value of the corresponding element class.
For convenience, taking an element icon obtained by ideal matting as an example, if an element effective region is the entire region of the element icon, then the second part and the third part of the element classification label are both 0, and the fourth part and the fifth part are both 1, and finally 11 txt-format markup files are obtained:
0 0 0 1 1：Camera.txt；
1 0 0 1 1：red_packet.txt；
2 0 0 1 1：new_weibo.txt；
3 0 0 1 1：topic.txt；
40 0 1 1：at.txt；
5 0 0 1 1：short_link.txt；
6 0 0 1 1：video_link.txt；
7 0 0 1 1：close.txt；
8 0 0 1 1：forward.txt；
9 0 0 1 1：comment.txt；
10 0 0 1 1：hump_up.txt。
and 4, step 4: and (3) repeating the step 1-3, wherein each element icon needs to be acquired by not less than 12 mobile phones with different resolutions so as to train the Yolov5 model. After training, the recall rate and the accuracy are both more than 95%, the Yolov5 model can be derived and is ready to be applied in the next stage.
Further, the steps of the model application phase are as follows:
step 1: and intercepting the interface picture of the target application to obtain the interface image shown in the figure 4.
Step 2: based on the matting technique, the elements of the interface of the target application are segmented to obtain the element icons as shown in fig. 5.
And step 3: the above-listed panels are input into the YOLOV5 model one by one for object detection, and the data content shown in fig. 6 is output. Referring to fig. 6, taking the first row as an example, the "frame" field is the coordinate information of the detected element, where "71" is the abscissa of the upper left corner of the element icon in the belonging interface image, "769" is the ordinate of the upper left corner of the element icon in the belonging interface image, "101" is the abscissa of the lower right corner of the element icon in the belonging interface image, "795" is the ordinate of the lower right corner of the element icon in the belonging interface image, "icon _ conf" field is the result coefficient, 0.57329 indicates that the model considers that this element has% 57.329 and may be a forward tag, "icon _ name" field is the detected tag name, and "index" field is the sequential index value in the last step of the thumbnail segmentation. The coordinate values of the element icons in the corresponding interface image can be obtained in the matting process, and are not described herein again.
And 4, step 4: and traversing the output content monitored by the target of the upper graph, identifying each element according to the icon _ name field, and identifying the coordinates of the elements in the interface of the target application, namely completing positioning. Wherein the element identification and positioning information is an execution parameter for subsequent testing of the target application.
In addition, the embodiment of the invention also provides an application test method on the basis of the identification method shown in fig. 1. Fig. 7 is a flowchart of an application testing method according to an embodiment of the present invention, which specifically includes the following steps:
and S702, performing interface screenshot on the target application to obtain an interface image of the target application.
S704, carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface.
S706, inputting the element icons in the target application into a preset element identification model, and identifying the elements in the target application, wherein the element identification model is obtained by training based on the sample element icons and the element category labels labeled on the sample element icons.
S708, inputting the information of the identified elements in the target application into the test script of the target application, and testing corresponding functions of the identified elements based on the test script.
The method provided by the embodiment of the invention trains an element identification model for identifying elements based on images through sample element icons and corresponding color element classification labels. When the target application is tested, element matting is firstly carried out on an interface of the target application, the element matting obtained by the matting is input into an element identification model to complete element identification, information of an element in the target application is input into a test script of the target application, and the identified element is automatically subjected to corresponding function development testing based on the test script. Because the identification mode does not depend on the code attribute information corresponding to the elements, even if the application continuously iterates to update, the test script does not need to be adjusted too much, and the difficulty of later maintenance is effectively reduced.
Correspondingly, fig. 8 is a structural diagram of an apparatus for identifying elements in an application interface according to an embodiment of the present invention, including:
the screenshot module 810 is used for performing interface screenshot on the target application to obtain an interface image of the target application;
a matting module 820, performing element matting on the interface image of the target application to obtain an element icon in the target application;
the identifying module 830 is configured to input the element icon in the target application to a preset element identification model, and identify the element in the target application, where the element identification model is trained based on a sample element icon and an element classification label corresponding to the sample element icon.
Optionally, the screenshot module 810 performs element matting on the interface image of the target application based on an element tree positioning method to obtain an element icon in the interface of the target application.
Optionally, the matting module 820 specifically obtains the coordinates and the size of the element in the target application from the UI frame information of the target application; and then carrying out element matting on the interface image of the target application based on the coordinates and the sizes of the elements in the target application.
Optionally, the element recognition model is a YOLOV5 model.
The element icons are divided into effective areas containing element rendering pixels and other areas not containing the element rendering pixels, and the element classification labels corresponding to the sample element icons comprise the following five dimensional information:
the matting sequence of the sample element icons corresponding to the interface images to which the sample element icons belong;
the ratio of the horizontal coordinate value of the upper left corner of the effective area in the sample element icon to the width value of the whole element icon,
the ratio of the vertical coordinate value of the upper left corner of the effective area in the sample element icon to the height value of the whole element icon;
a ratio of a width value of an effective area in the sample element icon to a width value of the entire element icon;
the ratio of the height of the active area in the sample element icon to the height of the entire element icon.
Obviously, the identification apparatus of the embodiment of the present invention may be used as an execution subject of the method shown in fig. 1, and thus, the steps and corresponding functions of the method shown in fig. 1 may be implemented. Since the principle is the same, detailed description is omitted herein.
Correspondingly, fig. 9 is a structural diagram of an application testing apparatus according to an embodiment of the present invention, including:
the screenshot module 910 performs interface screenshot on a target application to obtain an interface image of the target application.
And a matting module 920, configured to perform element matting on the interface image of the target application to obtain an element icon in the target application interface.
The identification module 930 is configured to input the element icon in the target application to a preset element identification model, and identify the element in the target application, where the element identification model is trained based on a sample element icon and an element category label labeled on the sample element icon;
and the test module 940 is used for inputting the information of the identified elements in the target application into the test script of the target application and testing the corresponding functions of the identified elements based on the test script.
The device provided by the embodiment of the invention trains an element identification model for identifying elements based on images through sample element icons and corresponding color element classification labels. When the target application is tested, element matting is firstly carried out on an interface of the target application, the element matting obtained by the matting is input into an element identification model to complete element identification, information of an element in the target application is input into a test script of the target application, and the identified element is automatically subjected to corresponding function development testing based on the test script. Because the identification mode does not depend on the code attribute information corresponding to the elements, even if the application continuously iterates to update, the test script does not need to be adjusted too much, and the difficulty of later maintenance is effectively reduced.
Optionally, the screenshot module 910 performs element matting on the interface image of the target application based on an element tree positioning method to obtain an element icon in the interface of the target application.
Optionally, the matting module 920 specifically obtains the coordinates and the sizes of the elements in the target application from the UI frame information of the target application; and then carrying out element matting on the interface image of the target application based on the coordinates and the sizes of the elements in the target application.
Optionally, the element recognition model is a YOLOV5 model.
The element icons are divided into effective areas containing element rendering pixels and other areas not containing the element rendering pixels, and the element classification labels corresponding to the sample element icons comprise the following five dimensional information:
the matting sequence of the sample element icons corresponding to the interface images to which the sample element icons belong;
the ratio of the horizontal coordinate value of the upper left corner of the effective area in the sample element icon to the width value of the whole element icon,
the ratio of the vertical coordinate value of the upper left corner of the effective area in the sample element icon to the height value of the whole element icon;
a ratio of a width value of an effective area in the sample element icon to a width value of the entire element icon;
the ratio of the height of the active area in the sample element icon to the height of the entire element icon.
Obviously, the application testing apparatus of the embodiment of the present invention may be used as an execution main body of the method shown in fig. 7, and thus, the steps and corresponding functions of the method shown in fig. 7 may be implemented. Since the principle is the same, detailed description is omitted herein.
Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present specification. Referring to fig. 10, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 10, but this does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the nonvolatile memory into the memory and then runs the computer program, and the identification device of the elements in the application is formed on a logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
and carrying out interface screenshot on the target application to obtain an interface image of the target application.
And carrying out element matting on the interface image of the target application to obtain an element icon in the target application.
And inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element classification label corresponding to the sample element icon.
Or, the processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program, and the application testing device is formed on the logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
and carrying out interface screenshot on the target application to obtain an interface image of the target application.
And carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface.
Inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is trained on a sample element icon and an element category label labeled on the sample element icon.
And inputting the information of the identified elements in the target application into a test script of the target application, and testing corresponding functions of the identified elements based on the test script.
The method disclosed in the embodiment of fig. 1 in this specification can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
It should be understood that the electronic device of the embodiment of the present invention may implement the functions of the method shown in fig. 1 or fig. 7. Since the principle is the same, the detailed description is omitted here.
Of course, besides the software implementation, the electronic device in this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.
Furthermore, an embodiment of the present invention also provides a computer-readable storage medium storing one or more programs, the one or more programs including instructions.
When executed by a portable electronic device including a plurality of application programs, the instructions enable the portable electronic device to perform the method in the embodiment shown in fig. 1, and are specifically configured to perform the following steps:
and carrying out interface screenshot on the target application to obtain an interface image of the target application.
And carrying out element matting on the interface image of the target application to obtain an element icon in the target application.
And inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element classification label corresponding to the sample element icon.
Alternatively, the above instructions, when executed by a portable electronic device including a plurality of application programs, can cause the portable electronic device to perform the method in the embodiment shown in fig. 7, and specifically to perform the following steps:
and carrying out interface screenshot on the target application to obtain an interface image of the target application.
And carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface.
And inputting the element icon in the target application into a preset element recognition model, and recognizing the element in the target application, wherein the element recognition model is trained on a sample element icon and an element category label labeled on the sample element icon.
And inputting the information of the identified elements in the target application into a test script of the target application, and testing corresponding functions of the identified elements based on the test script.
As will be appreciated by one of ordinary skill in the art, the embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and changes may occur to those skilled in the art to which it pertains. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification. Moreover, all other embodiments obtained by persons of ordinary skill in the art without making any inventive step shall fall within the scope of protection of this document.
Claims (10)
1. A method for identifying an element in an application, comprising:
performing interface screenshot on a target application to obtain an interface image of the target application;
carrying out element matting on the interface image of the target application to obtain an element icon in the target application;
and inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element classification label corresponding to the sample element icon.
2. The method of claim 1,
performing element matting on the interface image of the target application to obtain an element icon in the target application interface, including:
and carrying out element matting on the interface image of the target application based on an element tree positioning method to obtain an element icon in the target application interface.
3. The method of claim 2,
carrying out element matting on the interface image of the target application based on an element tree positioning method to obtain an element icon in the target application interface, wherein the element icon comprises the following steps:
acquiring coordinates and sizes of elements in the target application from UI frame information of the target application;
and carrying out element matting on the interface image of the target application based on the coordinates and the sizes of the elements in the target application.
4. The method according to any one of claims 1 to 3,
the element identification model is a YOLOV5 model, the element icons are divided into effective areas containing element rendering pixels and other areas not containing the element rendering pixels, and the element classification labels corresponding to the sample element icons comprise the following five dimensional information:
the category identification corresponding to the sample element icon;
the ratio of the horizontal coordinate value of the upper left corner of the effective area in the sample element icon to the width value of the whole element icon;
the ratio of the vertical coordinate value of the upper left corner of the effective area in the sample element icon to the height value of the whole element icon;
a ratio of a width value of an effective area in the sample element icon to a width value of the entire element icon;
the ratio of the height of the active area in the sample element icon to the height of the entire element icon.
5. The method of claim 1,
the information that the element recognition model recognizes the element in the target application includes a category of the element.
6. An application testing method, comprising:
performing interface screenshot on a target application to obtain an interface image of the target application;
carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface;
inputting the element icon in the target application into a preset element identification model, and identifying the element in the target application, wherein the element identification model is obtained by training based on a sample element icon and an element category label labeled on the sample element icon;
and inputting the information of the identified elements in the target application into a test script of the target application, and testing corresponding functions of the identified elements based on the test script.
7. An apparatus for identifying an element in an application interface, comprising:
the screenshot module is used for carrying out interface screenshot on the target application to obtain an interface image of the target application;
the matting module is used for carrying out element matting on the interface image of the target application to obtain an element icon in the target application;
the identification module is used for inputting the element icons in the target application into a preset element identification model and identifying the elements in the target application, wherein the element identification model is obtained by training based on the sample element icons and the element classification labels corresponding to the sample element icons.
8. An application testing apparatus, comprising:
the screenshot module is used for carrying out interface screenshot on the target application to obtain an interface image of the target application;
the matting module is used for carrying out element matting on the interface image of the target application to obtain an element icon in the target application interface;
the identification module is used for inputting the element icons in the target application into a preset element identification model and identifying the elements in the target application, wherein the element identification model is obtained by training based on sample element icons and element category labels labeled on the sample element icons;
and the test module is used for inputting the information of the identified elements in the target application into the test script of the target application and testing the corresponding functions of the identified elements on the basis of the test script.
9. An electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the method of any one of claims 1 to 5 or to perform the method of claim 6.
10. A computer readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of applications, perform the method of any of claims 1 to 5 or perform the method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111584833.3A CN114445652A (en) | 2021-12-22 | 2021-12-22 | Element identification method in application, application testing method and related hardware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111584833.3A CN114445652A (en) | 2021-12-22 | 2021-12-22 | Element identification method in application, application testing method and related hardware |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114445652A true CN114445652A (en) | 2022-05-06 |
Family
ID=81362954
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111584833.3A Pending CN114445652A (en) | 2021-12-22 | 2021-12-22 | Element identification method in application, application testing method and related hardware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114445652A (en) |
-
2021
- 2021-12-22 CN CN202111584833.3A patent/CN114445652A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111309618B (en) | Page element positioning method, page testing method and related devices | |
| WO2020133442A1 (en) | Text recognition method and terminal device | |
| CN111078908A (en) | Data annotation detection method and device | |
| CN111310826B (en) | Method and device for detecting labeling abnormality of sample set and electronic equipment | |
| CN111709414A (en) | AR device, character recognition method and device thereof, and computer-readable storage medium | |
| CN112926621B (en) | Data labeling method, device, electronic equipment and storage medium | |
| CN112149570B (en) | Multi-person living body detection method, device, electronic equipment and storage medium | |
| CN113505707A (en) | Smoking behavior detection method, electronic device and readable storage medium | |
| CN113763348A (en) | Image quality determination method and device, electronic equipment and storage medium | |
| CN111368632A (en) | Signature identification method and device | |
| CN112651311A (en) | Face recognition method and related equipment | |
| CN109978044B (en) | Training data generation method and device, and model training method and device | |
| KR102440198B1 (en) | VIDEO SEARCH METHOD AND APPARATUS, COMPUTER DEVICE, AND STORAGE MEDIUM | |
| CN111898610A (en) | Card unfilled corner detection method and device, computer equipment and storage medium | |
| CN112597940B (en) | Certificate image recognition method and device and storage medium | |
| CN110969640A (en) | Video image segmentation method, terminal device and computer-readable storage medium | |
| CN113138916A (en) | Automatic testing method and system for picture structuring algorithm based on labeled sample | |
| CN114255493A (en) | Image detection method, face detection device, face detection equipment and storage medium | |
| CN112287905A (en) | Vehicle damage identification method, device, equipment and storage medium | |
| CN111783561A (en) | Picture examination result correction method, electronic equipment and related products | |
| CN114445652A (en) | Element identification method in application, application testing method and related hardware | |
| US20230401809A1 (en) | Image data augmentation device and method | |
| CN115908977A (en) | Image data labeling method and device, electronic equipment and storage medium | |
| CN115294505A (en) | Risk object detection and model training method and device and electronic equipment | |
| CN113221718B (en) | Formula identification method, device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |