CN116501431A - Page element positioning method, device and storage medium - Google Patents

Page element positioning method, device and storage medium Download PDF

Info

Publication number
CN116501431A
CN116501431A CN202310472142.7A CN202310472142A CN116501431A CN 116501431 A CN116501431 A CN 116501431A CN 202310472142 A CN202310472142 A CN 202310472142A CN 116501431 A CN116501431 A CN 116501431A
Authority
CN
China
Prior art keywords
file
page
uix
screen capture
positioning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310472142.7A
Other languages
Chinese (zh)
Inventor
陈刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Suning Bank Co Ltd
Original Assignee
Jiangsu Suning Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Suning Bank Co Ltd filed Critical Jiangsu Suning Bank Co Ltd
Priority to CN202310472142.7A priority Critical patent/CN116501431A/en
Publication of CN116501431A publication Critical patent/CN116501431A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/451Execution arrangements for user interfaces
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/445Program loading or initiating
    • G06F9/44521Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The invention provides a page element positioning method, a page element positioning device and a storage medium, wherein the method comprises the following steps: acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file; if the screen capture is forbidden, extracting element coordinate information based on the uix file, wherein the screen capture file is an empty file; drawing an element position picture based on the element coordinate information, and taking the element position picture as a screen capture file; and loading uix files and screen capture files, and positioning page elements. The invention can be compatible with the positioning requirements of two page elements which can be captured and not captured, the difference between the two page elements is shielded, a user does not need to pay attention to whether a specific page can be captured, the positioning of the two types of page elements can be realized through unified operation steps, the whole process automation operation of the extra learning cost is avoided, and the positioning of the page elements is based on a uiAutomatorviewer tool familiar to the user and the extra learning cost is avoided.

Description

Page element positioning method, device and storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a storage medium for locating page elements.
Background
Page element positioning is a precondition for mobile APP page automation, and various existing tools can assist in page element positioning, and the free positioning tools uiautomatic viewer and application software matched with the android system are provided, and the webator is developed based on python codes; the charging tools include cloud testing software and the like. The software supports various methods for positioning the page elements, such as ID, name, xpath, class name and image comparison and the like based on the elements, and the comprehensive use of various means can meet the positioning requirements of most page elements.
The application principle of the tool is that the page image of the APP and the layout file thereof are processed, a positioning method and a value of specific elements are provided, and the tool is used on the premise of having the APP page screen capturing authority.
However, with the increasing popularity of use of bank APP, every year, the use of APP related funds security event is endless, and from the viewpoints of regulatory requirements and bank reputation, each bank APP controls the screen capturing authority, and there is a tendency to prohibit the gradual expansion of the interface page scope, for example: asset pages, transfer process & results pages, login process pages, etc. are all prohibited from screen capturing.
For the forbidden screenshot page element positioning, it is common practice in the industry for the development team to go out an APP test specific package that turns off the "forbidden screenshot" function, bypassing this limitation. This practice has the following disadvantages: 1. the switch of the detected object for prohibiting screen capture is required to be closed, and the improper switch state is easily brought into the production environment; 2. the core pages to be protected are forbidden to be used in the use scene without closing the screen capturing function, a switching mechanism is not introduced, the forbidden screen capturing function is deeply inserted into various business process pages, the cost of closing the forbidden screen capturing function one by one is high, and the risk is high; 3. some output functions, the entry is in the external APP, the source code modification capability is not possessed, and the function of prohibiting screen capture cannot be closed.
Disclosure of Invention
In view of the above problems, the invention provides a method, a device and a storage medium for locating page elements, which have the characteristics of simplicity and low learning cost, and are used for stably and reliably locating page elements compared with an APP test special packet for closing a 'forbidden screen capture' function.
In order to solve the technical problems, the invention adopts the following technical scheme: a page element positioning method comprises the following steps: acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file; if the screen capture is forbidden, extracting element coordinate information based on the uix file, wherein the screen capture file is an empty file; drawing an element position picture based on the element coordinate information, and taking the element position picture as a screen capture file; and loading uix files and screen capture files, and positioning page elements.
Preferably, the obtaining the page layout file includes: and connecting the mobile phone with a computer through an Android debug bridge, manually opening a designated page, acquiring a uix layout file and a screen capture file of the current page through a shell command, and downloading the uix layout file and the screen capture file to a local place.
Preferably, the extracting element coordinate information based on the uix file includes: opening a uix file, and extracting coordinate information at one time through a regular expression; replacing each coordinate character string, and analyzing each coordinate character string as a tuple object; and combining the coordinate information of the upper left point and the lower right point into a large tuple object which is used as an input parameter for drawing the picture.
Preferably, the drawing the element position picture based on the element coordinate information includes: generating a blank picture according to the screen size; and drawing an element position rectangular chart on the blank picture according to the element coordinate information.
Preferably, the loading uix file and the screen capturing file perform page element positioning, including: opening uiautoformationviewer tools using bat commands; linking uiautomatic viewer processes by using a Pywinauto module; sequentially clicking ' Open ', ' Screen ', ' OpenScreen ', ' filename (N) > ' Open (O) ', by using a Pywinauto module, and completing loading of the screen capture file; sequentially clicking 'Open' > 'UIXMLUMP' > 'OpenUIXMLUMpFILE' > 'filename (N) >' Open (O) ", and completing uix file loading; clicking an OK button, entering a uiAutomation viewer to analyze the page, and completing the positioning of the page elements.
In a preferred scheme, the uiAutomation viewer tool can be replaced by a weditor tool, and when page element positioning is performed, a local screen capture file and a uix file are added for operation.
The invention also provides a page element positioning device, which comprises: the acquisition module is used for acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file; the extraction module is used for extracting element coordinate information based on the uix file if the screen capture file is blank if the screen capture file is forbidden; the drawing module draws element position pictures based on the element coordinate information and takes the element position pictures as screen capturing files; and the positioning module is used for loading uix files and screen capturing files and positioning page elements.
The invention also provides a computer readable storage medium storing a computer program which when executed by a processor implements the steps of the method as claimed in any one of the preceding claims.
Compared with the prior art, the invention has the beneficial effects that: element coordinate information is extracted based on a uix file, element position pictures are drawn according to the element coordinate information, the element position pictures are used as screen capturing files, positioning requirements of two page elements which can be captured and cannot be captured can be met, the difference between the two page elements is shielded, a user does not need to pay attention to whether a specific page can be captured, and the two types of page elements can be positioned through unified operation steps. The whole process is automatically operated without extra learning cost, and the page element positioning is based on a uiAutomation viewer tool familiar to a user, so that the extra learning cost is avoided. The method is realized by using a uiAutomatorviewer or a weditor tool, does not depend on a specific element positioning tool, and has good expandability. The page element positioning method is simple in process, the tool is based on a mature python module, no complex processing process exists, and stable and reliable operation can be guaranteed.
Drawings
The disclosure of the present invention is described with reference to the accompanying drawings. It is to be understood that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the invention. In the drawings, like reference numerals are used to refer to like parts. Wherein:
FIG. 1 is a flow chart of a page element positioning method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another flow chart of a page element positioning method according to an embodiment of the present invention;
FIG. 3 is an interface diagram of page element positioning resolution in screen capturing according to an embodiment of the present invention;
FIG. 4 is an interface diagram of a page element positioning error report when a screen shot is forbidden according to an embodiment of the invention;
FIG. 5 is a diagram showing a comparison of an original page and an element position picture according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of resolving element information according to an element position picture according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a page element positioning device according to an embodiment of the present invention.
Detailed Description
It is to be understood that, according to the technical solution of the present invention, those skilled in the art may propose various alternative structural modes and implementation modes without changing the true spirit of the present invention. Accordingly, the following detailed description and drawings are merely illustrative of the invention and are not intended to be exhaustive or to limit the invention to the precise form disclosed.
An embodiment according to the invention is shown in connection with fig. 1 and 2. A page element positioning method comprises the following steps:
s101, acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file.
Wherein, obtain the page layout file, include: and connecting the mobile phone with a computer through an Android debug bridge, manually opening a designated page, acquiring a uix layout file and a screen capture file of the current page through a shell command, and downloading the uix layout file and the screen capture file to a local place. The code implementation commands are as follows:
adb shell rm/sdcard/app.uix
adb shell rm/sdcard/app.png
adb shell uiautomator dump/sdcard/app.uix
adb pull/sdcard/app.uix app.uix
adb shell screencap-p/sdcard/app.png
adb pull/sdcard/app.pngapp.png
it should be appreciated that Android debug bridges (adbs) are a variety of command line tools that enable communication with devices. The adb command may be used to perform various device operations, such as installing and debugging applications. adb provides access to Unixshells (which can be used to run various commands on a device). It is a client-server program comprising three components, client, daemon and server: the client runs on the development machine and is used for sending a command and calling the client from the command line terminal by sending an adb command; daemons (adbd) for running commands on the devices, daemons running as background processes on each device; the server is used to manage communications between the client and daemon, and the server runs as a background process on the development machine.
S102, if the page prohibits screen capture and the screen capture file is an empty file, extracting element coordinate information based on the uix file. When the screen capturing is forbidden, the interface diagram of the page element positioning error reporting is shown in fig. 4.
Specifically, extracting element coordinate information based on uix file includes:
(1) And opening a uix file, and extracting coordinate information at one time through a regular expression.
Uix is taken as a common text file, and the element coordinate information is as follows: "[1193,2266] [1275,2322]", one-time extraction is performed by a regular expression. The regular expression: '0-9 +, [0-9 + ]'.
(2) And replacing each coordinate character string, and analyzing each coordinate character string as a tuple object.
(3) And combining the coordinate information of the upper left point and the lower right point into a large tuple object which is used as an input parameter for drawing the picture.
For example: each coordinate string is parsed into a tuple object by replacing each coordinate '[' ] 'string with' (')' such as "[1193,2266] [1275,2322]" becomes "(1193,2266) (1275,2322)". The coordinate information of the upper left and lower right points is combined into a large tuple object which is used as an input parameter for drawing the following rectangle.
And S103, drawing an element position picture based on the element coordinate information, and taking the element position picture as a screen capture file. The position rectangle of the corresponding element can be easily found by combining with the mobile phone page, so that the corresponding original information is obtained. As shown in fig. 5, the left picture is an original page, and the right picture is an element position picture drawn under the condition of prohibiting screen capturing.
Specifically, drawing an element position picture based on element coordinate information includes: generating a blank picture according to the screen size by using opencv; and drawing an element position rectangular chart on the blank picture according to the element coordinate information.
The pseudocode is implemented as follows:
s104, using Pywinauto to operate a uiautomatic viewer tool, loading uix files and screen capturing files, and positioning page elements.
When screen capturing is allowed, an interface diagram of page element positioning analysis is shown in fig. 3. When the screen capturing is forbidden, a schematic diagram of resolving the element information according to the element position picture is shown in fig. 6. And analyzing the page by using a uiAutomation viewer tool to obtain attribute values, such as id and text information, of the page elements, thereby completing the positioning of the page elements.
Specifically, loading uix files and screen capture files to locate page elements includes:
(1) Opening uiautoformationviewer tools using bat commands;
(2) Linking uiautomatic viewer processes by using a Pywinauto module;
(3) Sequentially clicking ' Open ', ' Screen ', ' OpenScreen ', ' filename (N) > ' Open (O) ', by using a Pywinauto module, and completing loading of the screen capture file;
(4) Sequentially clicking 'Open' > 'UIXMLUMP' > 'OpenUIXMLUMpFILE' > 'filename (N) >' Open (O) ", and completing uix file loading;
(5) Clicking an OK button, entering a uiAutomation viewer to analyze the page, and completing the positioning of the page elements.
The pseudocode is implemented as follows:
the code start= = = = = = = = = = = = = = =
# import uix, screen file to Automatovier
pro=app.process_from_module('java.exe')
app=app.Application(backend='uia')
app.connect(process=pro)
dlg=app['UI Automator Viewer']
dlg.maximize()
dlg.Open.click()
#screenshot file
dlg.child_window(title="Screenshot",
control_type="Group").Button.click()
dlg.child_window(title="Open Screenshot File",
control_type="Window")
# selection screenshot file
dlg.child_window (title= "filename (N):,
control_type="Edit").set_text(screenshot_file)
# click on open button
dlg.child_window (title= "open (O)",
control_type="Button").click()
#XML file
dlg.child_window(title="UI XML Dump",
control_type="Group").Button.click()
dlg.child_window(title="Open UI XML Dump File",
control_type="Window")
# select uix layout file
dlg.child_window (title= "filename (N):,
control_type="Edit").set_text("app.uix")
# click on open button
dlg.child_window (title= "open (O)",
control_type="Button").click()
# click OK button
dlg.child_window(title="OK",control_type="Button").click()
= = = = = = = = = = = = = = = = = = = = = = = = = = = = =
In the embodiment of the invention, the uiautomatic viewer tool can be replaced by a weditor and other tools, and when page element positioning is performed, a local screen capture file and a uix file are added for operation, and the uiautomatic viewer tool is not dependent on a specific element positioning tool, so that the expandability is good.
Referring to fig. 5, the present invention further provides a page element positioning device, including:
an obtaining module 101, configured to obtain a page layout file, where the page layout file includes a uix file and a screen capture file;
the extraction module 102 extracts element coordinate information based on the uix file if the page prohibits screen capture and the screen capture file is an empty file;
the drawing module 103 draws element position pictures based on the element coordinate information and takes the element position pictures as screen capturing files;
and the positioning module 104 is used for loading uix files and screen capturing files and positioning page elements.
The invention also provides a computer readable storage medium storing a computer program which when executed by a processor performs the steps of any of the methods described above.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In summary, the beneficial effects of the invention include: element coordinate information is extracted based on a uix file, element position pictures are drawn according to the element coordinate information, the element position pictures are used as screen capturing files, positioning requirements of two page elements which can be captured and cannot be captured can be met, the difference between the two page elements is shielded, a user does not need to pay attention to whether a specific page can be captured, and the two types of page elements can be positioned through unified operation steps. The whole process is automatically operated without extra learning cost, and the page element positioning is based on a uiAutomation viewer tool familiar to a user, so that the extra learning cost is avoided. The method is realized by using a uiAutomatorviewer or a weditor tool, does not depend on a specific element positioning tool, and has good expandability. The page element positioning method is simple in process, the tool is based on a mature python module, no complex processing process exists, and stable and reliable operation can be guaranteed.
It should be appreciated that the integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The technical scope of the present invention is not limited to the above description, and those skilled in the art may make various changes and modifications to the above-described embodiments without departing from the technical spirit of the present invention, and these changes and modifications should be included in the scope of the present invention.

Claims (8)

1. The page element positioning method is characterized by comprising the following steps:
acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file;
if the screen capture is forbidden, extracting element coordinate information based on the uix file, wherein the screen capture file is an empty file;
drawing an element position picture based on the element coordinate information, and taking the element position picture as a screen capture file;
and loading uix files and screen capture files, and positioning page elements.
2. The method for locating page elements according to claim 1, wherein the obtaining a page layout file includes: and connecting the mobile phone with a computer through an Android debug bridge, manually opening a designated page, acquiring a uix layout file and a screen capture file of the current page through a shell command, and downloading the uix layout file and the screen capture file to a local place.
3. The page element positioning method according to claim 1, wherein the extracting element coordinate information based on the uix file includes:
opening a uix file, and extracting coordinate information at one time through a regular expression;
replacing each coordinate character string, and analyzing each coordinate character string as a tuple object;
and combining the coordinate information of the upper left point and the lower right point into a large tuple object which is used as an input parameter for drawing the picture.
4. The page element positioning method according to claim 1, wherein the drawing of the element position picture based on the element coordinate information includes:
generating a blank picture according to the screen size;
and drawing an element position rectangular chart on the blank picture according to the element coordinate information.
5. The method for locating page elements according to claim 1, wherein loading uix files and screen capturing files for locating page elements comprises:
opening uiautoformationviewer tools using bat commands;
linking uiautomatic viewer processes by using a Pywinauto module;
sequentially clicking ' Open ', ' Screen ', ' OpenScreen ', ' filename (N) > ' Open (O) ', by using a Pywinauto module, and completing loading of the screen capture file;
sequentially clicking 'Open' > 'UIXMLUMP' > 'OpenUIXMLUMpFILE' > 'filename (N) >' Open (O) ", and completing uix file loading;
clicking an OK button, entering a uiAutomation viewer to analyze the page, and completing the positioning of the page elements.
6. The method for locating page elements according to claim 1, wherein the uiautomatiorviewer tool can be replaced by a weditor tool, and when locating page elements, the operations of opening local screen capture file and uix file are added.
7. A page element positioning device, comprising:
the acquisition module is used for acquiring a page layout file, wherein the page layout file comprises a uix file and a screen capture file;
the extraction module is used for extracting element coordinate information based on the uix file if the screen capture file is blank if the screen capture file is forbidden;
the drawing module draws element position pictures based on the element coordinate information and takes the element position pictures as screen capturing files;
and the positioning module is used for loading uix files and screen capturing files and positioning page elements.
8. A computer-readable storage medium, characterized in that a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1-6.
CN202310472142.7A 2023-04-27 2023-04-27 Page element positioning method, device and storage medium Pending CN116501431A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310472142.7A CN116501431A (en) 2023-04-27 2023-04-27 Page element positioning method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310472142.7A CN116501431A (en) 2023-04-27 2023-04-27 Page element positioning method, device and storage medium

Publications (1)

Publication Number Publication Date
CN116501431A true CN116501431A (en) 2023-07-28

Family

ID=87324379

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310472142.7A Pending CN116501431A (en) 2023-04-27 2023-04-27 Page element positioning method, device and storage medium

Country Status (1)

Country Link
CN (1) CN116501431A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117827357A (en) * 2024-03-04 2024-04-05 北京云测信息技术有限公司 Remote operation method, device and equipment of terminal equipment

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117827357A (en) * 2024-03-04 2024-04-05 北京云测信息技术有限公司 Remote operation method, device and equipment of terminal equipment

Similar Documents

Publication Publication Date Title
CN110427331B (en) Method for automatically generating performance test script based on interface test tool
CN109918055B (en) Application program generation method and device
CN106484611B (en) Fuzzy test method and device based on automatic protocol adaptation
CN107171825B (en) Repeated log filtering method for terminal
US20100281463A1 (en) XML based scripting framework, and methods of providing automated interactions with remote systems
US9442822B2 (en) Providing a visual representation of a sub-set of a visual program
CN104899016B (en) Allocating stack Relation acquisition method and device
CN111831538A (en) Debugging method, device and storage medium
CN116501431A (en) Page element positioning method, device and storage medium
CN111324352A (en) Code generation method of application page and related equipment
CN107632921A (en) A kind of method and system that server info is shown by Quick Response Code
CN105760761A (en) Software behavior analyzing method and device
CN109871205B (en) Interface code adjustment method, device, computer device and storage medium
CN111177623A (en) Information processing method and device
CN117076338B (en) Method and system for dynamically debugging Linux kernel based on kprobe
JP4102365B2 (en) Data linkage support method between applications
CN114661623B (en) UI automatic test method for testing React Native application
CN111078529A (en) Client write-in module testing method and device and electronic equipment
CN113360379B (en) Program test environment creation method and program test environment creation apparatus
CN106327560A (en) File version identification method and identification client side
CN111026631B (en) Automatic interface detection method, device and server
CN113326193A (en) Applet testing method and device
CN113010434A (en) System and method for realizing lightweight application point burying
CN112685305B (en) User interface performance test method and device, electronic equipment and readable storage medium
CN113094265B (en) Analysis method and analysis device for test script and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination