CN116303024A - Method, system, electronic device and storage medium for acquiring interface information of android application during running - Google Patents

Method, system, electronic device and storage medium for acquiring interface information of android application during running Download PDF

Info

Publication number
CN116303024A
CN116303024A CN202310269447.8A CN202310269447A CN116303024A CN 116303024 A CN116303024 A CN 116303024A CN 202310269447 A CN202310269447 A CN 202310269447A CN 116303024 A CN116303024 A CN 116303024A
Authority
CN
China
Prior art keywords
interface
attribute information
transparency
interface element
android application
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310269447.8A
Other languages
Chinese (zh)
Inventor
马骏
黄伟祥
曹春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University
Original Assignee
Nanjing University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University filed Critical Nanjing University
Priority to CN202310269447.8A priority Critical patent/CN116303024A/en
Publication of CN116303024A publication Critical patent/CN116303024A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3684Test management for test design, e.g. generating new test cases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3688Test management for test execution, e.g. scheduling of test suites
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/57Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention discloses a method, a system, electronic equipment and a storage medium for acquiring interface information of android application in a runtime, wherein the method comprises the following steps: 1. acquiring a multi-layer interface list, and if the interface name does not contain the name of the file package of the current application, removing the interface layer which does not belong to the current application; 2. extracting and storing interface element attribute information of the uppermost interface; 3. preliminarily removing invisible interface elements according to attribute information of the interface elements, capturing a screenshot of the rest interface elements, calculating transparency of the screenshot, and further removing transparent interface elements; 4. returning the attribute information of the extracted interface element if the current interface contains the visible interface element, returning to the blank if the current interface does not contain the visible interface element and is the bottommost layer, otherwise, entering the next layer of the current interface, extracting the attribute information of the interface element of the interface layer, and returning to the step 3. The method improves the accuracy of the android application interface test and enhances the resistance to repackaging attacks.

Description

Method, system, electronic device and storage medium for acquiring interface information of android application during running
Technical Field
The invention belongs to the technical field of software testing, and particularly relates to a method, a system, electronic equipment and a storage medium for acquiring interface information of android application in a running process.
Background
Because the android system is open and the android application is huge in number, the android application test is always popular, and the android application is interacted with a user mainly through an application interface, so that most of researches in the android application test are concentrated on a graphical interface, the android application interface comprises a series of interface elements such as layouts and controls, when a developer designs the user interface, the layouts and the controls are nested and stacked, the elements in the application interface are in a tree structure, the controls are mainly responsible for interacting with the user, most of layouts are used for optimally placing the controls, and the layouts are basically invisible. When developing android applications, developers may also add small or transparent interface elements, and these invisible interface elements are mainly used for special functions or appearance designs. Some attackers decompil, modify and repackage the android application, and illegally achieve the goal by inserting advertisements and malicious programs in the android application, the attacks can be detected by generating and comparing software probes, and the attackers can confuse the repackaged application by various methods, such as code or interface confusion, in order to prevent the repackaged application from being detected. Researchers find that the repackaging application can be detected more accurately by dynamically acquiring the interface to generate the software fetal marks, so that the means for confusing the interface of the attacker is more and more, and the application of invisible controls is more and more, for example, by inserting a plurality of invisible layouts or controls to change the tree structure of the interface, so that the inspector cannot accurately acquire the interface information and generate the software fetal marks in the process of dynamically running the application. From the standpoint of testing or repacking, it is important that the interface information be accurately obtained so that the obtained interface information is consistent with what the user actually sees.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a method, a system, electronic equipment and a storage medium for acquiring the interface information of the android application in the running process, so as to solve the technical problem that invisible elements in an android graphical interface cause trouble to the existing graphical interface testing tool, and an attacker uses the invisible elements to confuse the interface when performing repackaging attack.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a method for acquiring interface information of android application in runtime comprises the following steps:
step 1: when the android application runs, a multi-layer interface list is obtained, if the name of the interface does not contain the name of the file package of the current application, the interface layer which does not belong to the application is removed, and the interface list of the application is obtained;
step 2: extracting attribute information of all interface elements of the uppermost interface in the interface list of the application and storing the attribute information in a key value pair; the interface elements comprise layouts and controls, and the attribute information comprises IDs, transparency, sizes and positions;
step 3: removing invisible interface elements according to attribute information of the interface elements, then capturing the rest interface elements, calculating transparency of the capturing, and further removing transparent interface elements;
step 4: returning the attribute information of the extracted interface element if the current interface contains the visible interface element, returning to the blank if the current interface does not contain the visible interface element and is the bottommost layer, otherwise, entering the next layer of the current interface, continuing to extract the attribute information of the interface element of the interface layer, and returning to the step 3.
In order to optimize the technical scheme, the specific measures adopted further comprise:
further, in step 3, the excluding the invisible interface element according to the attribute information of the interface element specifically includes:
step 3.1.1: if the transparency of the interface element is smaller than the transparency threshold alpha, judging that the interface element is invisible;
step 3.1.2: if the width or height of the interface element is smaller than the length threshold value beta, judging that the interface element is invisible;
step 3.1.3: and finding out the area occupied by the interface element according to the position, the width and the height, and if the area occupied by the interface element and the interface area of the android application have no overlapping part or the width or the height of the overlapping part is smaller than a length threshold value beta, indicating that the interface element exceeds the interface area, and judging that the interface element is invisible.
Further, in step 3, the capturing is performed on the remaining interface elements, and the transparency of the capturing is calculated, so that the transparent interface elements are further excluded, specifically:
obtaining the screenshot of the rest interface element, sampling the pixel points in the screenshot, starting sampling from the pixel position with the upper left corner coordinate of (0, 0), setting the sampling frequency in the width direction as m+1 and the sampling frequency in the height direction as n+1, and setting the sampling interval M in the width direction as the screenshot width
Figure SMS_1
The sampling interval N in the height direction is +.>
Figure SMS_2
Taking n+1 rows and m+1 columns, and (m+1) x (n+1) sample pixel points, wherein each sample pixel point comprises four channels, namely a red channel, a green channel, a blue channel and a transparency channel, calculating the average value of transparency values of the transparency channels of the sample pixel points, and judging that the interface element is transparent if the average value of the transparency values is smaller than a transparency threshold alpha.
Further, in step 3, when the remaining interface elements are subjected to screenshot, a screenshot instruction may be sent, an ID of the interface element is specified, and the specified interface element is separately subjected to screenshot.
The invention also provides a system for acquiring the interface information of the android application in the running process, which comprises a data acquisition module and a module for eliminating invisible elements;
the data acquisition module is used for obtaining an interface list of the application when the android application runs, extracting attribute information of all interface elements in the interface layer and storing the attribute information in the key value pair;
the invisible element eliminating module is used for eliminating invisible interface elements according to attribute information of the interface elements, capturing the rest interface elements, calculating transparency of the capturing, and eliminating transparent interface elements.
The invention also provides an electronic device, comprising: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the method for acquiring the interface information of the android application when the processor executes the computer program.
The invention also provides electronic equipment, which stores a computer program, and the computer program enables the computer to execute the method for acquiring the interface information of the android application in the running process.
The beneficial effects of the invention are as follows:
the invisible elements in the android graphical interface are eliminated, and the original tree structure is compressed, so that a tester can acquire less and more accurate information;
the invisible interface elements are eliminated, the interface acquired by the graphic testing tool is prevented from being inconsistent with the interface actually seen by human eyes, only the visible elements in the graphic interface are reserved, and the similarity calculation between the android application interfaces is more practical;
the accuracy of the android application interface test is improved, the resistance to repackaging attacks is enhanced, and a method for acquiring interface information can be provided for a third-party dynamic repackaging detection method.
Drawings
FIG. 1 is a block diagram of a system according to the present invention;
FIG. 2 is a flow chart of invisible interface element removal;
FIG. 3 is an exemplary diagram of an android application interface;
FIG. 4 is a diagram of an example of taking a pixel sample of a screenshot;
fig. 5 is a flow chart for obtaining visual graphical interface information.
Detailed Description
The invention will now be described in further detail with reference to the accompanying drawings.
The android application is interacted with a user mainly through a graphical interface when the android application runs, the graphical interface of the android application is composed of a series of layouts and controls, the controls are mainly responsible for interacting with the user, most layouts are used for optimally placing the controls, and information of the layouts and the controls can be obtained through a third-party tool such as a ViewServer when the android application runs. The invisible interface element is removed to obtain the graphical interface information of the android application more accurately. A large number of invisible layouts and controls exist in the graphical interface of the android application, and the invisible interface elements are not removed by the graphical interface information acquired by the third-party tool during operation, so that a developer or an attacker can change the structural information of the graphical interface by inserting some invisible layouts or controls on the premise of not changing the visual effect of the graphical interface, thereby influencing the information acquired by the third-party tool and further influencing the test and comparison of the android graphical interface. The above problems can be effectively solved by removing interface elements such as invisible layouts and controls.
In an embodiment, the present invention provides a method for obtaining interface information of an android application during runtime, and an overall flowchart of the method is shown in fig. 5, including the following steps:
step 1: when the android application runs, a multi-layer interface list is obtained, because some system interfaces such as notification bars and other graphic interface layers can be covered on the android application, in order to eliminate the influence of the system interfaces, when the interfaces are selected, whether the interfaces belong to the application is firstly judged, and if the names of the interfaces do not contain the names of file packages of the current application, the interface layers which do not belong to the application are removed, so that an interface list of the application is obtained;
step 2: extracting attribute information of all interface elements of the uppermost interface in the interface list of the application and storing the attribute information in a key value pair; the interface elements comprise layouts and controls, and the attribute information comprises IDs, transparency, sizes and positions;
the acquired attribute information of the interface elements is in a text format, each line of text represents a layout or control, and nesting and containing relations of the line are represented by retracting the line head. The values of each attribute of the layout and the control are recorded in the text in the form of key-value, and detailed information, such as ID, transparency, size and position, of the corresponding layout or control can be obtained by analyzing each line of text.
Step 3: the invisible interface elements are removed, and the specific flow is shown in fig. 2, the invisible interface elements are removed according to the attribute information of the interface elements, whether the transparency value of the layout or the control is smaller than the transparency threshold value alpha (the default setting alpha is 0.1) is judged first, if the transparency value is smaller than the transparency threshold value alpha, the layout or the control is considered to be transparent or almost transparent, the invisible interface elements are required to be removed, most of the layouts are used for more optimally placing the control, the layouts are basically transparent, and a large number of layouts can be removed at the step. And secondly, judging whether the control is visible according to the layout and the size of the control, wherein the default length threshold value beta is three pixel distances, and if the width or the height of one interface element is smaller than beta, the interface element is basically invisible to a user, and the interface element is regarded as invisible. It is then determined whether an interface element is visible based on its layout and the position and size, and if it is not within the range of the current interface and its occupied area and interface range do not overlap, or the overlap range is wider or higher than the length threshold β, then it is considered invisible.
After the above steps, the invisible layout and control are basically removed, but some attackers may implement a layout or control which can avoid the three steps but is invisible, and the transparency, size and position of the layout or control indicate that the layout or control is visible, but in fact, the layout or control only contains a transparent background image, so that the user cannot see the control. To exclude such controls, we screen capture the remaining interface elements and calculate the transparency of the screen capture, further excluding the transparent interface elements.
We can send a screenshot instruction to a third party tool specifying the ID of the layout or control to screenshot the layout or control individually.
Since the android system allows different transparency in different areas of the background image in the control, in order to avoid misjudgment of gradual change designs of some developers, sampling pixels of the screenshot, judging whether the whole image is transparent or not by averaging transparency channel values of the pixels, sending a screenshot instruction to a third party tool, designating a "LOGIN" button in the screenshot to perform screenshot, and taking sample pixels of the screenshot, as shown in fig. 4. Setting the sampling interval M in the width direction as one ninth of the screenshot width, setting the sampling interval N in the height direction as one fifth of the screenshot height, and then taking six rows and ten columns from the pixel position with the upper left corner coordinates of (0, 0) of the screenshot, wherein the total number of the pixel points is 60. Each pixel point comprises four channels, namely a red channel, a green channel, a blue channel and a transparency channel, the transparency values of the transparency channels of the sample pixel points are averaged, and if the result is smaller than a transparency threshold alpha, the interface element is judged to be transparent and invisible.
Step 4: returning the attribute information of the extracted interface element if the current interface contains the visible interface element, returning to the blank if the current interface does not contain the visible interface element and is the bottommost layer, otherwise, entering the next layer of the current interface, continuing to extract the attribute information of the interface element of the interface layer, and returning to the step 3.
In another embodiment, the invention provides a system for acquiring the runtime interface information of the android application, which corresponds to the method for acquiring the runtime interface information of the android application provided in the first embodiment, wherein a system block diagram is shown in fig. 1, and the system block diagram comprises a data acquisition module and an invisible element elimination module;
the data acquisition module is used for obtaining an interface list of the application when the android application runs, extracting attribute information of all interface elements in the interface layer and storing the attribute information in the key value pair;
the invisible element eliminating module is used for eliminating invisible interface elements according to attribute information of the interface elements, capturing the rest interface elements, calculating transparency of the capturing, and eliminating transparent interface elements.
In another embodiment, the present invention provides an electronic device, including: the method for acquiring the interface information of the android application in the running process according to the first embodiment is realized when the processor executes the computer program.
In another embodiment, the present invention provides a computer readable storage medium storing a computer program, where the computer program causes a computer to execute the method for acquiring the interface information of the android application in the first embodiment.
In the embodiments disclosed herein, a computer storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer storage medium would include one or more wire-based electrical connections, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the invention without departing from the principles thereof are intended to be within the scope of the invention as set forth in the following claims.

Claims (7)

1. The method for acquiring the interface information of the android application in the running process is characterized by comprising the following steps of:
step 1: when the android application runs, a multi-layer interface list is obtained, if the name of the interface does not contain the name of the file package of the current application, the interface layer which does not belong to the application is removed, and the interface list of the application is obtained;
step 2: extracting attribute information of all interface elements of the uppermost interface in the interface list of the application and storing the attribute information in a key value pair; the interface elements comprise layouts and controls, and the attribute information comprises IDs, transparency, sizes and positions;
step 3: removing invisible interface elements according to attribute information of the interface elements, then capturing the rest interface elements, calculating transparency of the capturing, and further removing transparent interface elements;
step 4: returning the attribute information of the extracted interface element if the current interface contains the visible interface element, returning to the blank if the current interface does not contain the visible interface element and is the bottommost layer, otherwise, entering the next layer of the current interface, continuing to extract the attribute information of the interface element of the interface layer, and returning to the step 3.
2. The method for obtaining interface information during android application runtime according to claim 1, wherein in step 3, the excluding the invisible interface element according to the attribute information of the interface element specifically includes:
step 3.1.1: if the transparency of the interface element is smaller than the transparency threshold alpha, judging that the interface element is invisible;
step 3.1.2: if the width or height of the interface element is smaller than the length threshold value beta, judging that the interface element is invisible;
step 3.1.3: and finding out the area occupied by the interface element according to the position, the width and the height, and if the area occupied by the interface element and the interface area of the android application have no overlapping part or the width or the height of the overlapping part is smaller than a length threshold value beta, indicating that the interface element exceeds the interface area, and judging that the interface element is invisible.
3. The method for obtaining interface information during android application runtime of claim 1, wherein in step 3, the remaining interface elements are captured, and transparency of the captured image is calculated, and further excluding transparent interface elements specifically includes:
obtaining the screenshot of the rest interface element, sampling the pixel points in the screenshot, starting sampling from the pixel position with the upper left corner coordinate of (0, 0), setting the sampling frequency in the width direction as m+1 and the sampling frequency in the height direction as n+1, and setting the sampling interval M in the width direction as the screenshot width
Figure FDA0004134145810000011
The sampling interval N in the height direction is +.>
Figure FDA0004134145810000012
Taking n+1 rows and m+1 columns, and (m+1) x (n+1) sample pixel points, wherein each sample pixel point comprises four channels, namely a red channel, a green channel, a blue channel and a transparency channel, calculating the average value of transparency values of the transparency channels of the sample pixel points, and judging that the interface element is transparent if the average value of the transparency values is smaller than a transparency threshold alpha.
4. The method for obtaining interface information during android application runtime of claim 1, wherein in step 3, when the remaining interface elements are captured, a capture instruction may be sent, an ID of the interface element is specified, and the specified interface element is captured separately.
5. The system for acquiring the android application runtime interface information is characterized by comprising a data acquisition module and an invisible element elimination module;
the data acquisition module is used for obtaining an interface list of the application when the android application runs, extracting attribute information of all interface elements in the interface layer and storing the attribute information in the key value pair;
the invisible element eliminating module is used for eliminating invisible interface elements according to attribute information of the interface elements, capturing the rest interface elements, calculating transparency of the capturing, and eliminating transparent interface elements.
6. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of obtaining android application runtime interface information as claimed in any one of claims 1-4 when the computer program is executed.
7. A computer-readable storage medium, characterized by: computer program is stored which causes a computer to perform the method of obtaining android application runtime interface information as claimed in any of claims 1-4.
CN202310269447.8A 2023-03-20 2023-03-20 Method, system, electronic device and storage medium for acquiring interface information of android application during running Pending CN116303024A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310269447.8A CN116303024A (en) 2023-03-20 2023-03-20 Method, system, electronic device and storage medium for acquiring interface information of android application during running

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310269447.8A CN116303024A (en) 2023-03-20 2023-03-20 Method, system, electronic device and storage medium for acquiring interface information of android application during running

Publications (1)

Publication Number Publication Date
CN116303024A true CN116303024A (en) 2023-06-23

Family

ID=86779337

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310269447.8A Pending CN116303024A (en) 2023-03-20 2023-03-20 Method, system, electronic device and storage medium for acquiring interface information of android application during running

Country Status (1)

Country Link
CN (1) CN116303024A (en)

Similar Documents

Publication Publication Date Title
CN111124888B (en) Method and device for generating recording script and electronic device
US9454836B2 (en) Object display device and object display method
CN108875534B (en) Face recognition method, device, system and computer storage medium
US9804955B2 (en) Method and apparatus for creating reference images for an automated test of software with a graphical user interface
CN107710280A (en) Object method for visualizing
WO2015074521A1 (en) Devices and methods for positioning based on image detection
CN108664364A (en) A kind of terminal test method and device
US9176043B2 (en) Cell image analysis apparatus, cell image analysis method, and program
CN110865753B (en) Application message notification method and device
CN111414948B (en) Target object detection method and related device
CN108961316B (en) Image processing method and device and server
CN113505707A (en) Smoking behavior detection method, electronic device and readable storage medium
CN111461070B (en) Text recognition method, device, electronic equipment and storage medium
CN112559341A (en) Picture testing method, device, equipment and storage medium
CN111062854A (en) Method, device, terminal and storage medium for detecting watermark
CN109753883A (en) Video locating method, device, storage medium and electronic equipment
CA3152854C (en) Human-computer similar trajectory detection method and apparatus based on screen segmentation
CN109542546A (en) A kind of throwing screen method and device at application process window and interface
CN116303024A (en) Method, system, electronic device and storage medium for acquiring interface information of android application during running
CN106682669A (en) Image processing method and mobile terminal
CN109934185A (en) Data processing method and device, medium and calculating equipment
US20190149878A1 (en) Determining and correlating visual context on a user device with user behavior using digital content on the user device
CN113537026A (en) Primitive detection method, device, equipment and medium in building plan
JP5882883B2 (en) Telop detection device, telop detection method, and telop detection program
CN111292374A (en) Method and equipment for automatically plugging and unplugging USB interface

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination