CN116450510A - Interactive element detection and model training method, device, equipment and storage medium - Google Patents

Interactive element detection and model training method, device, equipment and storage medium Download PDF

Info

Publication number
CN116450510A
CN116450510A CN202310346941.XA CN202310346941A CN116450510A CN 116450510 A CN116450510 A CN 116450510A CN 202310346941 A CN202310346941 A CN 202310346941A CN 116450510 A CN116450510 A CN 116450510A
Authority
CN
China
Prior art keywords
interactive element
application
sample data
preset
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310346941.XA
Other languages
Chinese (zh)
Inventor
高柳洋
胡骏飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Netease Hangzhou Network Co Ltd
Original Assignee
Netease Hangzhou Network Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Netease Hangzhou Network Co Ltd filed Critical Netease Hangzhou Network Co Ltd
Priority to CN202310346941.XA priority Critical patent/CN116450510A/en
Publication of CN116450510A publication Critical patent/CN116450510A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3684—Test management for test design, e.g. generating new test cases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3696—Methods or tools to render software testable
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/96—Management of image or video recognition tasks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The application provides an interactive element detection and model training method, device, equipment and storage medium, and relates to the technical field of image processing. The method comprises the following steps: traversing an interface of a preset application according to information of a preset interaction element to obtain a plurality of groups of sample data, wherein each group of sample data comprises: information of a sample interaction element in a sample application screenshot; correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data; and performing model training according to the multiple groups of target sample data to obtain the interactive element detection model. Compared with the prior art, the method and the device avoid the problem that the application scene is inconvenient enough, and the information of the interaction element acquired based on the mode can be subjected to missing report or false report.

Description

Interactive element detection and model training method, device, equipment and storage medium
Technical Field
The application relates to the technical field of image recognition, in particular to an interactive element detection and model training method, device and equipment and a storage medium.
Background
In the process of software development, the fields such as testing and the like have a great amount of requirements for acquiring interactive element information, and the interactive element information is utilized for secondary development and software testing. It is important to obtain what interactive element information exists in the page, where the interactive element information is located, what the type of interactive element information is, whether a button can be clicked or a sliding window can be slid.
Traditional interactive element information acquisition scheme: and acquiring information of the interactive element information tree through a system interface, analyzing and acquiring information such as the type of the interactive element information, the position of the interactive element information, the size of the interactive element information and the like.
However, such a detection manner is not convenient enough for its application scenario because it is analyzed and acquired based on the system interface, and the information of the interaction element acquired based on such a manner may have a problem of missing report or false report.
Disclosure of Invention
The application aims to provide an interactive element detection and model training method, device, equipment and storage medium for overcoming the defects in the prior art, so as to solve the problems that the application scene is too limited in the prior art, and the information of the interactive element acquired based on the mode possibly has missing report or false report.
In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
in a first aspect, an embodiment of the present application provides a training method of an interactive element detection model, where the method includes:
traversing an interface of a preset application according to information of a preset interaction element to obtain a plurality of groups of sample data, wherein each group of sample data comprises: information of a sample interaction element in a sample application screenshot;
correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data;
and performing model training according to the multiple groups of target sample data to obtain the interactive element detection model.
In a second aspect, an embodiment of the present application provides an interactive element detection method, where the method includes:
acquiring an application screenshot to be identified;
adopting a preset interactive element detection model to detect the interactive element of the application screenshot to be identified, and obtaining information of a target interactive element in the application screenshot to be identified; the preset interactive element detection model is obtained by training the interactive element detection model training method according to any one of the first aspect.
In a third aspect, another embodiment of the present application provides a training apparatus for an interactive element detection model, where the apparatus includes: the system comprises a traversing module, a correcting module and a training module, wherein:
the traversing module is configured to traverse an interface of a preset application according to information of a preset interaction element, so as to obtain multiple groups of sample data, where each group of sample data includes: information of a sample interaction element in a sample application screenshot;
the correction module is used for correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data;
and the training module is used for carrying out model training according to the multiple groups of target sample data to obtain the interactive element detection model.
In a fourth aspect, another embodiment of the present application provides an interactive element detection apparatus, including: the device comprises an acquisition module and a detection module, wherein:
the acquisition module is used for acquiring an application screenshot to be identified;
the detection module is used for detecting the interactive element of the application screenshot to be identified by adopting a preset interactive element detection model to obtain information of a target interactive element in the application screenshot to be identified; the preset interactive element detection model is obtained by training the interactive element detection model training method according to any one of the first aspect.
In a fifth aspect, another embodiment of the present application provides an electronic device, including: a processor, a storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating over a bus when the electronic device is running, the processor executing the machine-readable instructions to perform the steps of the method as described in any of the first aspects above.
In a sixth aspect, another embodiment of the present application provides a storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method according to any of the first aspects described above.
The beneficial effects of this application are: according to the training method for the interactive element detection model, the sample data are obtained by traversing the interface of the preset application according to the information of the preset interactive element, each sample data comprises one sample application screenshot and the information of the sample interactive element in the sample application screenshot, after the plurality of groups of sample data are corrected, a plurality of groups of target sample data are obtained, model training is carried out based on the plurality of groups of target sample data, and the interactive element detection model is obtained, so that the obtained interactive element detection model is the sample application screenshot, and the information of the interactive element in the application screenshot can be directly determined according to the application screenshot in the subsequent application process.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered limiting the scope, and that other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a training method of an interactive element detection model according to an embodiment of the present application;
FIG. 2 is a flowchart of a training method of an interactive element detection model according to another embodiment of the present application;
FIG. 3 is a flow chart of an interactive element detection method according to an embodiment of the present application;
fig. 4 is a flow chart of an interactive element detection method according to another embodiment of the present application;
FIG. 5 is a schematic structural diagram of a training device for an interactive element detection model according to an embodiment of the present disclosure;
FIG. 6 is a schematic structural diagram of a training device for an interactive element detection model according to another embodiment of the present application;
fig. 7 is a schematic structural diagram of an interactive element detection device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an interactive element detection device according to another embodiment of the present application;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments.
The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.
Additionally, a flowchart, as used in this application, illustrates operations implemented in accordance with some embodiments of the present application. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to the flow diagrams and one or more operations may be removed from the flow diagrams as directed by those skilled in the art.
The following explains a training method of an interactive element detection model provided in the embodiment of the present application in conjunction with a plurality of specific application examples. Fig. 1 is a flow chart of a training method of an interactive element detection model according to an embodiment of the present application, as shown in fig. 1, the method includes:
s101: traversing the interface of the preset application according to the information of the preset interaction element to obtain a plurality of groups of sample data.
Each set of sample data includes: information of sample interaction elements in a sample application screenshot; wherein the sample interaction element may be, for example, a sliding control, a scrolling control, a progress bar control, an operable control, or a selectable control on some sample application screen shot (User Interface); it should be understood that the types included in the specific sample interaction elements may be flexibly adjusted according to the needs of the user, and are not limited to the above embodiments.
In the embodiment of the application, the interface of the preset application is traversed, which can be traversed on any terminal device or webpage, for example, the preset application running on the android device, the IOS device, the desktop computer device, the notebook computer device, the H5 and the webpage; wherein H5 is a fifth version of HTML (Hyper Text Markup Language ); that is, by adopting the traversal method provided by the application, not only the preset application running on the android device or the IOS device can be traversed, but also the preset application running on other forms of terminal devices can be traversed, and the problem that interactive elements running on the preset application on other forms of terminal devices are difficult to acquire in the prior art is solved.
S102: and correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data.
In the embodiment of the application, for example, multiple groups of sample data may be displayed on preset data labeling software; and correcting the plurality of groups of sample data according to the correction operation for the plurality of groups of sample data so as to obtain a plurality of groups of target sample data.
In some possible embodiments, after the multiple sets of sample data are displayed on the preset data labeling software, based on a manual correction labeling mode, the multiple sets of sample data are checked to determine whether a false alarm, a missing report or a popup window problem exists in the multiple sets of sample data, so as to correct the sample data with a problem in the multiple sets of sample data, and the correction may include, for example, fine tuning the multiple sets of sample data or labeling information of real sample interaction elements in each set of sample data in a manual labeling mode, so that the problem that the information of the sample interaction elements in the sample data is inaccurate is avoided, and the corrected multiple sets of sample data are determined to be target sample data.
S103: and performing model training according to the multiple groups of target sample data to obtain an interactive element detection model.
That is, the interactive element detection model in the application is obtained by training according to multiple groups of target sample data, wherein each group of target sample data comprises a sample application screenshot, in the subsequent detection process, the interactive element detection model obtained by the training mode does not need to acquire related information in the application to be identified running on the equipment to be identified based on a system interface, but can directly identify and determine the interactive element in the image information based on the screenshot of the application to be identified, that is, based on the image information, and the application scene is wider.
In the embodiment of the application, in order to improve model convergence, that is, improve the speed of model training, model training is performed on a pre-training image detection network according to multiple sets of target sample data to obtain an interactive element detection model, wherein the pre-training image detection network is a preset general image detection network.
That is, in the embodiment of the present application, training is performed according to the pre-training image detection network, and then training in the field of interactive element detection is performed on the pre-training image detection network according to multiple sets of target sample data, so as to obtain an interactive element detection model.
In some possible embodiments, to facilitate model training of the interactive element detection model, the data format of the target sample data may be converted to a cos format, for example, to facilitate training of the interactive element detection model.
According to the training method for the interactive element detection model, the sample data are obtained by traversing the interface of the preset application according to the information of the preset interactive element, each sample data comprises one sample application screenshot and the information of the sample interactive element in the sample application screenshot, after the plurality of groups of sample data are corrected, a plurality of groups of target sample data are obtained, model training is carried out based on the plurality of groups of target sample data, and the interactive element detection model is obtained, so that the obtained interactive element detection model is the sample application screenshot, and the information of the interactive element in the application screenshot can be directly determined according to the application screenshot in the subsequent application process.
Optionally, on the basis of the foregoing embodiment, the embodiment of the present application may further provide a training method of the interactive element detection model, and an implementation process of obtaining multiple sets of sample data in the foregoing method is illustrated with reference to the accompanying drawings. Fig. 2 is a flow chart of a training method of an interactive element detection model according to another embodiment of the present application, as shown in fig. 2, S101 may include:
s111: traversing the interface of the preset application according to the information of the preset interactive elements, and capturing the current interface of the preset application when the preset interactive elements are obtained through traversing.
In this embodiment of the present application, for example, the automation framework POCO may obtain information of preset interactive elements of each terminal device, then traverse an interface of a preset application based on the preset interactive element information obtained by POCO, determine whether the interface of the preset application includes or has preset interactive element information, and if it is determined in the traversal process that the interface of the preset application includes the preset interactive elements, screenshot a current interface of the preset application.
S112: and determining the current interface screenshot of the preset application as a sample application screenshot.
S113: and saving the information of the sample interaction element in the sample application screenshot.
In some possible embodiments, after the information of the sample application screenshot and the sample interaction elements in the sample application screenshot is stored, the sample application screenshot of a plurality of preset applications and the information of the sample interaction elements in the sample application screenshot can be stored, an interface interaction element information tree corresponding to the preset application is generated, and the information of each corresponding sample interaction element on each sample application screenshot can be determined in the interface interaction information tree; it should be understood that the foregoing embodiments are merely exemplary, or the sample application screenshot of each preset application and the information of the sample interaction element in the sample application screenshot may be stored in a tabular manner, and only the correspondence between the sample application screenshot and the information of the sample interaction element in the sample application screenshot needs to be established, which may be specifically and flexibly adjusted according to the needs of the user, and is not limited to the foregoing embodiments.
In the embodiment of the application, if the sample application screenshots of the plurality of preset applications and the information of the sample interaction elements in the sample application screenshots are stored in the interface interaction information tree, in the process of correcting the plurality of groups of sample data, each group of sample data in the interface interaction information tree can be acquired in a crawling manner, the acquired groups of sample data are placed on preset data labeling software for display, and then the groups of sample data are corrected in a manual correction labeling manner.
By adopting the training method of the interactive element detection model, the model is trained based on a plurality of sample application screenshots and the information of the sample interactive elements in each sample application screenshot so as to obtain the interactive element detection model, the interactive element detection model obtained through training can be directly detected only based on picture image information in the subsequent application process by adopting the training mode, and the information of the interactive elements in the picture image information is detected and determined, and because the target sample data in the application are all corrected, the accuracy of the obtained interactive element detection model is high, and the method can be widely applied to the fields of service and software development, software testing or software anomaly detection and the like.
The embodiment of the application further provides an interactive element detection method, and in combination with a plurality of specific application examples, the method for detecting an interactive element provided in the embodiment of the application is explained in fig. 3, which is a schematic flow diagram of an interactive element detection method provided in another embodiment of the application, and as shown in fig. 3, the method may include:
s121: and acquiring an application screenshot to be identified.
In some possible embodiments, the obtaining manner of the application screenshot to be identified may be, for example, to respond to a screenshot operation initiated by a user, and perform screenshot on the application interface to be identified, where the obtained screenshot is the application screenshot to be identified; or, after screenshot is performed based on a section of interface video of the existing application to be identified, the obtained screenshot is the screenshot of the application to be identified; or photographing the application to be identified, wherein the obtained photographing content is the screenshot of the application to be identified; it should be understood that the above embodiment is merely an exemplary illustration, and the specific manner of acquiring the screenshot of the application to be identified may be flexibly adjusted according to the needs of the user, which is not limited to the above embodiment, and only the acquired screenshot of the application to be identified may be a static image of the application interface containing the application to be identified.
S122: and adopting a preset interactive element detection model to detect the interactive element of the application screenshot to be identified, and obtaining the information of the target interactive element in the application screenshot to be identified.
In the embodiment of the application, after the information of the target interaction element in the application screenshot to be identified is obtained, the application screenshot to be identified and the information of the target interaction element can be packaged and sent to the preset operation and maintenance equipment, so that the preset operation and maintenance personnel can carry out subsequent processing based on the obtained information package of the application screenshot to be identified and the information package of the target interaction element.
In the embodiment of the application, before the application screenshot to be identified is subjected to interactive element detection by adopting a preset interactive element detection model, normalization processing is required to be performed on the application screenshot to be identified in advance, wherein the normalization processing mode can be, for example, normalization processing is performed on the application screenshot to be identified according to numerical distribution of sample data; the data processing is performed before model training, the pixel value of the image is generally in the range of 0-255, the neural network learning is generally converted into 0-1, the later training is facilitated, and the image is generally divided by 255 at the same time, so that the image becomes a normalized graph. I.e. the normalization in this application is to change the data distribution from 0-255 to 0-1.
In addition, in order to ensure the accuracy of detection, in the embodiment of the present application, the size of the application screenshot to be identified needs to be processed, the size of the application screenshot to be identified needs to be scaled and adjusted to be the preset size of the preset interactive element detection model, that is, the size of the specific preset size can be flexibly adjusted according to the user's needs, and the application does not limit the application.
The preset interactive element detection model is obtained by training any one of the interactive element detection models in the embodiments provided in fig. 1-2.
According to the interactive element detection method, the preset interactive element detection model is obtained by training according to the training method of any interactive element detection model in the embodiment provided by the figures 1-2, therefore, only the application screenshot to be identified is provided as image input and is input into the preset interactive element detection model, information of the target interactive element in the application screenshot to be identified can be automatically identified, and because the element and the interactive detection model are corrected in the training process when the based target sample data is trained, the accuracy of the information of the target interactive element determined based on the application screenshot to be identified is higher in the using process of the preset interactive element detection model obtained based on the training mode.
Optionally, on the basis of the foregoing embodiment, an interactive element detection method may also be provided in an embodiment of the present application, and an implementation procedure of the method is described below with reference to the accompanying drawings. Fig. 4 is a flow chart of an interactive element detection method according to another embodiment of the present application, as shown in fig. 4, where the method may further include:
s123: and labeling the target interaction element on the application screenshot to be identified based on the information of the target interaction element, and obtaining a target image.
In some possible embodiments, the labeling manner may be, for example: and marking the target interaction element at the corresponding position on the application screenshot to be identified based on the position information of the target interaction element, and marking attribute information of the target interaction element at one side of the corresponding position.
In the embodiment of the application, after the target image is obtained, the target image containing the label of the target interaction element can be output to the preset operation and maintenance equipment, so that the preset operation and maintenance personnel can perform subsequent processing according to the received target image.
By adopting the interactive element detection method provided by the application, the preset interactive element detection model is obtained by training according to the training method of any interactive element detection model in the embodiment provided by fig. 1-2, that is, the interactive element detection method is an application side method of the training method of the interactive element detection model, so that the beneficial effects brought by the method are consistent with those of the training method of the interactive element detection model provided by fig. 1-2, and the application is not repeated herein.
The following explanation is made on the training device of the interactive element detection model provided by the present application in combination with the accompanying drawings, and the training device of the interactive element detection model may execute the training method of any one of the interactive element detection models shown in fig. 1-2, and specific implementation and beneficial effects of the training device refer to the foregoing, and are not repeated herein.
Fig. 5 is a schematic structural diagram of a training device for an interactive element detection model according to an embodiment of the present application, where, as shown in fig. 5, the device includes: a traversal module 301, a correction module 302, and a training module 303, wherein:
the traversing module 301 is configured to traverse an interface of a preset application according to information of a preset interaction element, so as to obtain multiple groups of sample data, where each group of sample data includes: information of sample interaction elements in a sample application screenshot;
the correction module 302 is configured to correct multiple groups of sample data to obtain multiple groups of target sample data;
and the training module 303 is used for performing model training according to the multiple groups of target sample data to obtain an interactive element detection model.
Optionally, on the basis of the foregoing embodiment, an embodiment of the present application may further provide a training device for an interactive element detection model, where an implementation process of the device provided in fig. 5 is described below with reference to the accompanying drawings. Fig. 6 is a schematic structural diagram of a training device for an interactive element detection model according to another embodiment of the present application, where, as shown in fig. 6, the device further includes: a determination module 304 and a save module 305, wherein:
the traversing module 301 is specifically configured to traverse an interface of a preset application according to information of the preset interactive element, and when the traversing obtains the preset interactive element, screenshot is performed on a current interface of the preset application;
the determining module 304 is configured to determine that a current screenshot of a preset application is a sample application screenshot;
a saving module 305, configured to save the sample application screenshot and information of the sample interaction element in the sample application screenshot.
Optionally, as shown in fig. 6, the apparatus further includes: a display module 306 and a correction module 307, wherein:
the display module 306 is configured to display a plurality of groups of sample data on preset data labeling software;
the correction module 307 is configured to correct the multiple sets of sample data according to the correction operation for the multiple sets of sample data, so as to obtain multiple sets of target sample data.
Optionally, the training module 303 is specifically configured to perform model training on a pre-training image detection network according to multiple sets of target sample data to obtain an interactive element detection model, where the pre-training image detection network is a preset general image detection network.
The foregoing apparatus is used for executing the method provided in the foregoing embodiment, and its implementation principle and technical effects are similar, and are not described herein again.
The following explains the interactive element detection device provided in the present application with reference to the accompanying drawings, where the interactive element detection device may execute any one of the interactive element detection methods shown in fig. 3 to fig. 4, and specific implementation and beneficial effects of the interactive element detection device refer to the foregoing, and are not repeated herein.
Fig. 7 is a schematic structural diagram of an interactive element detection device according to an embodiment of the present application, as shown in fig. 7, where the device includes: an acquisition module 401 and a detection module 402, wherein:
an obtaining module 401, configured to obtain an application screenshot to be identified;
the detection module 402 is configured to detect an interactive element of the application screenshot to be identified by using a preset interactive element detection model, so as to obtain information of a target interactive element in the application screenshot to be identified; the preset interactive element detection model is obtained by training the interactive element detection model by adopting the training method of any one of the figures 1-2.
Optionally, on the basis of the foregoing embodiment, an apparatus for detecting an interactive element may be further provided in an embodiment of the present application, and an implementation process of the apparatus provided in fig. 7 is described below with reference to the accompanying drawings. Fig. 8 is a schematic structural diagram of an interactive element detection device according to another embodiment of the present application, where, as shown in fig. 8, the device further includes: the labeling module 403 is configured to label the target interaction element on the application screenshot to be identified based on the information of the target interaction element, so as to obtain a target image.
Optionally, the labeling module 403 is specifically configured to label the target interaction element at a corresponding position on the application screenshot to be identified based on the position information of the target interaction element, and label attribute information of the target interaction element at one side of the corresponding position.
Optionally, as shown in fig. 8, the apparatus further includes: and the processing module 404 is used for normalizing the application screenshot to be identified.
The foregoing apparatus is used for executing the method provided in the foregoing embodiment, and its implementation principle and technical effects are similar, and are not described herein again.
The above modules may be one or more integrated circuits configured to implement the above methods, for example: one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as ASICs), or one or more microprocessors, or one or more field programmable gate arrays (Field Programmable Gate Array, abbreviated as FPGAs), etc. For another example, when a module above is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU) or other processor that may invoke the program code. For another example, the modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application, where the electronic device may be integrated in a terminal device or a chip of the terminal device.
As shown in fig. 9, the electronic device includes: a processor 501, a bus 502, and a storage medium 503.
The processor 501 is configured to store a program, and the processor 501 invokes the program stored in the storage medium 503 to perform the method embodiments corresponding to fig. 1-4. The specific implementation manner and the technical effect are similar, and are not repeated here.
Optionally, the present application also provides a program product, such as a storage medium, on which a computer program is stored, including a program which, when being executed by a processor, performs the corresponding embodiments of the above-mentioned method.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units.
The integrated units implemented in the form of software functional units described above may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (english: processor) to perform part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: u disk, mobile hard disk, read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.

Claims (12)

1. A method for training an interactive element detection model, comprising:
traversing an interface of a preset application according to information of a preset interaction element to obtain a plurality of groups of sample data, wherein each group of sample data comprises: information of a sample interaction element in a sample application screenshot;
correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data;
and performing model training according to the multiple groups of target sample data to obtain the interactive element detection model.
2. The method of claim 1, wherein traversing the interface of the preset application according to the information of the preset interaction element to obtain a plurality of sets of sample data comprises:
traversing the interface of the preset application according to the information of the preset interactive element, and capturing a current interface of the preset application when the preset interactive element is obtained through traversing;
determining the current interface screenshot of the preset application as the sample application screenshot;
and saving information of sample interaction elements in the sample application screenshot.
3. The method of claim 1, wherein said modifying said plurality of sets of sample data to obtain a plurality of sets of target sample data comprises:
displaying the plurality of groups of sample data on preset data labeling software;
and correcting the plurality of groups of sample data according to the correction operation on the plurality of groups of sample data so as to obtain the plurality of groups of target sample data.
4. The method according to claim 1, wherein the performing model training according to the multiple sets of target sample data to obtain the interactive element detection model includes:
and performing model training on a pre-training image detection network according to the multiple groups of target sample data to obtain the interactive element detection model, wherein the pre-training image detection network is a preset general image detection network.
5. An interactive element detection method, characterized in that the method comprises:
acquiring an application screenshot to be identified;
adopting a preset interactive element detection model to detect the interactive element of the application screenshot to be identified, and obtaining information of a target interactive element in the application screenshot to be identified; the preset interactive element detection model is obtained by training the interactive element detection model according to any one of claims 1-4.
6. The method of claim 5, wherein the method further comprises:
and labeling the target interaction element on the application screenshot to be identified based on the information of the target interaction element, so as to obtain a target image.
7. The method of claim 6, wherein the information of the target interactive element comprises: the position information and attribute information of the target interaction element; the labeling of the target interaction element on the application screenshot to be identified based on the information of the target interaction element to obtain a target image comprises the following steps:
and marking the target interaction element at a corresponding position on the application screenshot to be identified based on the position information of the target interaction element, and marking attribute information of the target interaction element at one side of the corresponding position.
8. The method of claim 5, wherein the method further comprises, before performing interactive element detection on the application screenshot to be identified by using a preset interactive element detection model to obtain information of a target interactive element in the application screenshot to be identified:
and normalizing the application screenshot to be identified.
9. A training device for an interactive element detection model, the device comprising: the system comprises a traversing module, a correcting module and a training module, wherein:
the traversing module is configured to traverse an interface of a preset application according to information of a preset interaction element, so as to obtain multiple groups of sample data, where each group of sample data includes: information of a sample interaction element in a sample application screenshot;
the correction module is used for correcting the plurality of groups of sample data to obtain a plurality of groups of target sample data;
and the training module is used for carrying out model training according to the multiple groups of target sample data to obtain the interactive element detection model.
10. An interactive element detection apparatus, the apparatus comprising: the device comprises an acquisition module and a detection module, wherein:
the acquisition module is used for acquiring an application screenshot to be identified;
the detection module is used for detecting the interactive element of the application screenshot to be identified by adopting a preset interactive element detection model to obtain information of a target interactive element in the application screenshot to be identified; the preset interactive element detection model is obtained by training the interactive element detection model according to any one of claims 1-4.
11. An electronic device, the device comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating over the bus when the electronic device is operating, the processor executing the machine-readable instructions to perform the method of any of the preceding claims 1-8.
12. A storage medium having stored thereon a computer program which, when executed by a processor, performs the method of any of the preceding claims 1-8.
CN202310346941.XA 2023-04-03 2023-04-03 Interactive element detection and model training method, device, equipment and storage medium Pending CN116450510A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310346941.XA CN116450510A (en) 2023-04-03 2023-04-03 Interactive element detection and model training method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310346941.XA CN116450510A (en) 2023-04-03 2023-04-03 Interactive element detection and model training method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116450510A true CN116450510A (en) 2023-07-18

Family

ID=87133067

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310346941.XA Pending CN116450510A (en) 2023-04-03 2023-04-03 Interactive element detection and model training method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116450510A (en)

Similar Documents

Publication Publication Date Title
CN109948507B (en) Method and device for detecting table
CN108229485B (en) Method and apparatus for testing user interface
US9135151B2 (en) Automatic verification by comparing user interface images
KR102002024B1 (en) Method for processing labeling of object and object management server
CN107491536B (en) Test question checking method, test question checking device and electronic equipment
US8392887B2 (en) Systems and methods for identifying graphic user-interface components
CN110955590A (en) Interface detection method, image processing method, device, electronic equipment and storage medium
CN110647523B (en) Data quality analysis method and device, storage medium and electronic equipment
CN110826494A (en) Method and device for evaluating quality of labeled data, computer equipment and storage medium
CN113763348A (en) Image quality determination method and device, electronic equipment and storage medium
CN110826646A (en) Robot vision testing method and device, storage medium and terminal equipment
CN114419035A (en) Product identification method, model training device and electronic equipment
Shamash et al. OnePetri: accelerating common bacteriophage Petri dish assays with computer vision
CN112989256B (en) Method and device for identifying web fingerprint in response information
CN117788444A (en) SMT patch offset detection method, SMT patch offset detection device and SMT patch offset detection system
CN110083540B (en) Interface testing method and device
CN116450510A (en) Interactive element detection and model training method, device, equipment and storage medium
CN115238212A (en) Page designated element detection and replacement method, device, equipment and storage medium
CN111124862B (en) Intelligent device performance testing method and device and intelligent device
CN112966671A (en) Contract detection method and device, electronic equipment and storage medium
CN113127858A (en) Anomaly detection model training method, anomaly detection method and anomaly detection device
CN112418217A (en) Method, apparatus, device and medium for recognizing characters
CN113641933A (en) Abnormal webpage identification method, abnormal site identification method and device
CN110633197B (en) Method and device for detecting excessive drawing
CN111768261B (en) Display information determining method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination