CN114546740A - Touch screen testing method, device and system and storage medium - Google Patents
Touch screen testing method, device and system and storage medium Download PDFInfo
- Publication number
- CN114546740A CN114546740A CN202210099863.3A CN202210099863A CN114546740A CN 114546740 A CN114546740 A CN 114546740A CN 202210099863 A CN202210099863 A CN 202210099863A CN 114546740 A CN114546740 A CN 114546740A
- Authority
- CN
- China
- Prior art keywords
- control
- touch screen
- operated
- image data
- tested
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 116
- 238000000034 method Methods 0.000 claims abstract description 18
- 238000013507 mapping Methods 0.000 claims description 5
- 238000005192 partition Methods 0.000 claims description 5
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000010998 test method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008909 emotion recognition Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000002195 synergetic effect Effects 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2205—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested
- G06F11/2221—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested to test input/output devices or peripheral units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides a touch screen testing method, a device, a system and a storage medium, wherein the method comprises the following steps: acquiring first image data displayed in a touch screen of equipment to be tested in real time; acquiring feature data of a control to be operated according to the configuration information of the test task, and determining whether the control to be operated exists in the first image data according to the feature data; if the control to be operated exists in the first image data, acquiring a corresponding actual coordinate of the control according to the image coordinate of the operation control; controlling the execution tail end of the mechanical arm to move to the actual coordinate of the control, and executing a first operation on the touch screen of the equipment to be tested; and acquiring second image data displayed in the touch screen of the device to be tested, and determining a test result of the test task according to the second image data. According to the invention, the control of the touch screen interface is accurately positioned in real time through machine vision without depending on the attribute of the UI control, so that the test efficiency is improved while manual operation is simulated.
Description
Technical Field
The invention relates to the technical field of product testing, in particular to a touch screen testing method, device and system and a storage medium.
Background
At present, touch screen type electronic products on the market are various in types, rich in functions and high in system version iteration updating speed. The automatic test of such touch screen products is usually performed by obtaining the properties of the UI control or by manipulating the UI control through test instructions. However, this method has the following drawbacks: sometimes, the control property cannot be acquired, so that the UI interface cannot be operated, for example: the third-party customized application does not support the user to acquire the control attributes or part of the controls after the Android system is upgraded cannot acquire the attributes.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the embodiment of the invention provides a touch screen testing method, a device, a system and a storage medium.
In one aspect, an embodiment of the present invention provides a touch screen testing method, including: acquiring first image data displayed in a touch screen of equipment to be tested in real time; acquiring feature data of a control to be operated according to configuration information of a test task, and determining whether the control to be operated exists in the first image data according to the feature data; if the control to be operated is determined to exist, acquiring an image coordinate of the control to be operated in the first image data, and acquiring a corresponding control actual coordinate according to the image coordinate, wherein the control actual coordinate is used for representing the position of the control to be operated on the touch screen side of the equipment to be operated; controlling an execution tail end of a mechanical arm to move to the actual coordinates of the control, and executing a first operation on a touch screen of the equipment to be tested, wherein the first operation is determined by the configuration of the test task; acquiring second image data displayed in a touch screen of the equipment to be tested, and determining a test result of the test task according to the second image data; and the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
The touch screen testing method provided by the embodiment of the invention at least has the following beneficial effects: the method comprises the steps of identifying a corresponding to-be-operated control in a test task by shooting a touch screen of the to-be-tested equipment in real time, obtaining a corresponding actual coordinate of the control through an image coordinate of the to-be-operated control, moving an execution tail end of a mechanical arm to the actual coordinate of the control, carrying out corresponding operation on the to-be-operated control on the touch screen, obtaining a display image responded by the to-be-tested equipment, and generating a corresponding test result; the method has the advantages that the UI control attribute is not depended on, the controls of the touch screen interface are accurately positioned in real time through machine vision, the manipulator is controlled to carry out UI interface control operation on the touch screen products, the test efficiency is improved while manual operation is simulated, and test errors caused by the fact that the operation is carried out by bypassing touch screen use commands or serial ports and the like and is inconsistent with the actual operation of a user are avoided.
According to some embodiments of the invention, the feature data of the control to be operated comprises third image data of the control to be operated; the determining whether the control to be operated exists in the first image data according to the feature data includes: acquiring a matching result through image identification, wherein the matching result is used for representing whether an area matched with the third image data exists in the first image data; and if the matching result is that an area matched with the third image data exists, determining that the control to be operated exists in the first image data.
According to some embodiments of the present invention, the feature data of the control to be operated includes a first text, where the first text is used to represent text information displayed by the control to be operated; the determining whether the control to be operated exists in the first image data according to the feature data includes: acquiring a character recognition result of the first image data; and if the character recognition result comprises the first characters, determining that the control to be operated exists in the first image data.
According to some embodiments of the invention, the configuration information of the test task comprises: the first position of the control to be operated is used for representing partition information where the control to be operated is displayed on the touch screen of the equipment to be tested; the determining whether the control to be operated exists in the first image data according to the feature data comprises: acquiring a corresponding identification area image from the first image data according to the first position; and determining whether the control to be operated exists in the image of the identification area according to the feature data.
According to some embodiments of the present invention, the obtaining the corresponding actual coordinates of the control according to the image coordinates includes: acquiring corresponding calibration data according to the type of the equipment to be tested; the calibration data is used for representing the mapping relation between the actual moving distance of the execution tail end and the image display moving distance of the touch screen of the equipment to be tested; and converting the image coordinates into actual coordinates of the control according to the calibration data.
According to some embodiments of the invention, the method of obtaining calibration data comprises: displaying a real-time shot image of the touch screen of the equipment to be tested through an interactive interface; in the real-time shot image, determining a touch screen area of the equipment to be tested as a first area, and determining a first calibration point and a second calibration point in the first area; controlling the executing tail end to move from an original point position so that the executing tail end is aligned with the first calibration point in the real-time shooting image to obtain a first actual coordinate of the executing tail end; controlling the execution terminal to move so that the execution terminal is aligned with the second calibration point in the real-time shot image, and obtaining a second actual coordinate of the execution terminal; and acquiring the calibration data according to the screen coordinate of the first calibration point, the first actual coordinate and the second actual coordinate.
According to some embodiments of the invention, the method further comprises configuring an axial movement speed and a touch delay of a horizontal plane of the robotic arm; the control is that the execution tail end of the mechanical arm moves to the actual coordinate of the control, and the execution of a first operation on the touch screen of the equipment to be tested comprises the following steps: adjusting the height of the execution tail end according to the thickness of the equipment to be tested; controlling the execution tail end of the mechanical arm to move to the actual coordinate of the control according to the axial moving speed of the horizontal plane of the mechanical arm; and waiting for the touch delay, and executing the first operation on the touch screen of the equipment to be tested.
In another aspect, an embodiment of the present invention provides a touch screen testing apparatus, including: the first module is used for acquiring first image data displayed in a touch screen of the equipment to be tested in real time; the second module is used for acquiring the characteristic data of a control to be operated according to the configuration information of the test task and determining whether the control to be operated exists in the first image data or not according to the characteristic data; a third module, configured to, if it is determined that the control to be operated exists, obtain an image coordinate of the control to be operated in the first image data, and obtain a corresponding actual control coordinate according to the image coordinate, where the actual control coordinate is used to represent a position of the control to be operated on a touch screen side of the device to be tested; the fourth module is used for controlling the execution tail end of the mechanical arm to move to the actual coordinates of the control and executing first operation on the touch screen of the equipment to be tested, wherein the first operation is determined by the configuration of the test task; the fifth module is used for acquiring second image data displayed in the touch screen of the device to be tested and determining a test result of the test task according to the second image data; and the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
The touch screen testing device provided by the embodiment of the invention at least has the following beneficial effects: the method comprises the steps of identifying a corresponding to-be-operated control in a test task by shooting a touch screen of the to-be-tested equipment in real time, obtaining a corresponding actual coordinate of the control through an image coordinate of the to-be-operated control, moving an execution tail end of a mechanical arm to the actual coordinate of the control, carrying out corresponding operation on the to-be-operated control on the touch screen, obtaining a display image responded by the to-be-tested equipment, and generating a corresponding test result; the method does not depend on UI control attributes, accurately positions the controls of the touch screen interface in real time through machine vision, controls the manipulator to carry out UI interface control operation on the touch screen products, improves the test efficiency while simulating manual operation, and avoids the test error caused by the inconsistency between the test error and the actual operation of a user and the operation of bypassing modes such as a touch screen use command or a serial port.
In another aspect, an embodiment of the present invention provides a touch screen testing system, including: the mechanical arm is connected with the computer equipment through a serial port and used for receiving an instruction of the computer equipment to control the execution tail end to move and execute touch operation on the touch screen of the equipment to be tested; the camera device is erected above the equipment to be tested and is used for shooting images displayed on a touch screen of the equipment to be tested in real time; computer apparatus comprising at least one processor and at least one memory storing at least one program which, when executed by the at least one processor, causes the at least one processor to implement a touch screen testing method as described above.
In another aspect, the present invention provides a computer readable storage medium, in which program instructions are stored, and the program instructions, when executed by a processor, implement the touch screen testing method described above.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart illustrating steps of a touch screen testing method according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating steps of a method for acquiring calibration data according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of converting image coordinates into actual coordinates for different types of test equipment in the embodiment of the present invention.
Fig. 4 is a schematic block diagram of a touch screen testing apparatus according to an embodiment of the present invention.
Fig. 5 is a schematic connection diagram of each device in the touch screen test system according to the embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. In the following description, suffixes such as "module", "part", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no peculiar meaning in itself. Thus, "module", "component" or "unit" may be used mixedly. "first", "second", etc. are used for the purpose of distinguishing technical features only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the precedence of the indicated technical features. In the following description, the method steps are labeled continuously for convenience of examination and understanding, and the implementation sequence of the steps is adjusted without affecting the technical effect achieved by the technical scheme of the invention in combination with the overall technical scheme of the invention and the logical relationship among the steps. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
At present, a method for automatically testing a touch screen product mainly operates a control by acquiring UI control attributes or operates a control by a test command and the like. However, such test methods all have certain disadvantages. For example, an automation test framework such as Uiautomator, Appium, Airtest and the like is adopted to obtain the attributes of the UI interface control and perform automation test on the operation of the control. There are two drawbacks. On one hand, sometimes, the control attribute cannot be acquired, so that the corresponding UI interface control cannot be operated, for example: the third-party customized application does not support the user to acquire the control attributes, or partial control elements cannot acquire the attributes after the Android system is upgraded. On the other hand, the method of testing through the test command is separated from the actual use scene of the user, and therefore a certain test deviation may exist.
Therefore, the touch screen testing method provided by the invention can be used for accurately positioning the control of the touch screen interface through machine vision and controlling the manipulator to perform UI (user interface) control operation on the touch screen products so as to simulate the actual use scene of a user. Because the invention does not depend on the acquisition of the UI interface control attribute, the invention can avoid the programming of the operating system depending on the product, reduce the technical requirements on testers and improve the testing efficiency.
Referring to fig. 1, the present embodiment discloses a touch screen testing method, which can test a device under test having a touch screen, including but not limited to the following steps S100 to S500.
Step S100, first image data displayed in a touch screen of the equipment to be tested is obtained in real time.
Specifically, the camera device is erected above the device to be tested, and an area including a touch screen of the device to be tested is shot so as to acquire a display picture in the touch screen of the device to be tested in real time as first image data. The image pickup apparatus may transmit the photographed first image data through a USB cable.
And step S200, acquiring characteristic data of the control to be operated according to the configuration information of the test task, and determining whether the control to be operated exists in the first image data according to the characteristic data.
In this embodiment, the test task may exist in a test script manner. The test task may include at least one test case, and each test case may operate at least one control to be operated. The configuration information of the test task includes related information of at least one to-be-operated control, and the related information of each to-be-operated control includes, for example: characteristic data of the control to be operated, a first operation, an expected result and the like. The characteristic data of the control to be operated is used for searching and positioning the position of the control to be operated. The first operation is used for executing corresponding touch operation on the control to be operated, such as: click, drag, etc. The expected result may be a response screen obtained by performing the first operation, and may be an image feature or a text feature of the response screen.
Illustratively, the feature data of the control to be operated may include at least one of: third image data, or, a first text. The third image data includes the image to be operated and controlled. The first text is used for representing text information displayed by the control to be operated, for example: all or part of the characters displayed on the control to be operated.
And when the feature data of the control to be operated is third image data, acquiring a region matched with the third image data from the first image data through image recognition. And if the area matched with the third image data exists, determining that the control to be operated exists in the first image data. When the image is identified, an identification threshold value may be configured, and when the matching degree between the image area in the first image data and the third image data is greater than the identification threshold value, it is determined that the control to be operated exists in the first image data, and the image coordinate of the control to be operated in the first image data is obtained. In this embodiment, whether the image recognition is gray scale recognition or color recognition may also be set. When the gray level identification is carried out, the first image data and the third image data are converted into a gray level image and then are matched; when color identification is performed, the color of the matching area is compared with the color of the third image data to determine whether the error range is included in the matching. The third image data can be captured before the test is implemented, and stored in a corresponding address; the address is configured in the test task.
And when the characteristic data of the control to be operated is a first character, acquiring a character recognition result of the first image data through character recognition, and searching the first character in the character recognition result. And if the first character is found, determining that the corresponding control to be operated exists in the first image data, and acquiring the image coordinate of the control to be operated in the first image data.
In this embodiment, the touch screen of the device to be tested may be further divided into a plurality of partitions, and a first position of the control to be operated is configured in the test task, where the first position is used to represent in which partition the control to be operated is displayed. The positioning process to be operated and controlled is as follows: acquiring image data of a corresponding subarea from the first image data as an identification area image according to the first position; and determining whether the control to be operated exists in the identification area image according to the characteristic data of the control to be operated. And if the control to be operated exists, acquiring the image coordinate of the control to be operated in the first image data. Through screen partition recognition, control positioning misjudgment can be prevented, and recognition accuracy is improved.
In this embodiment, when there is no corresponding control to be operated in the first image data, information of a matching failure is recorded, for example: the operation type of the device under test, the time of the matching failure, etc. And when the matching fails, the test of the control to be operated is stopped, or the test of the control to be operated is suspended, retries is carried out after waiting for a period of time, and when the retries exceed a certain number of times, the test of the control to be operated is stopped. After the test of the control to be operated is terminated, the next control to be operated can be tested until all the controls to be operated in the test task are executed.
Step S300, if the control to be operated is determined to exist, acquiring the image coordinate of the control to be operated in the first image data, and acquiring the corresponding actual coordinate of the control according to the image coordinate.
And the actual coordinates of the control are used for representing the position of the control to be operated on the touch screen side of the equipment to be tested.
Specifically, corresponding calibration data is acquired according to the type of the equipment to be tested; and converting the image coordinates into actual coordinates of the control according to the calibration data. The calibration data is used for representing the mapping relation between the actual moving distance of the executing tail end of the mechanical arm and the image display moving distance of the touch screen of the equipment to be tested. By converting the image coordinate into the actual coordinate of the control, the control arm can be conveniently controlled to move the executing tail end to the control to be operated of the touch screen of the equipment to be tested to carry out corresponding operation.
Referring to fig. 2, the method of acquiring calibration data includes, but is not limited to, steps S710 to S750.
And step S710, displaying the real-time shot image of the touch screen of the device to be tested through the interactive interface.
For example, the touch screen of the device to be tested is photographed in real time by the camera device, and the acquired image is displayed on the display interface of the terminal connected with the camera device.
Step S720, in the real-time shooting image, the touch screen area of the device to be tested is determined as a first area, and a first calibration point and a second calibration point are determined in the first area.
The touch screen area of the device to be tested can be determined as the first area through the interactive interface, and the first area can also be calibrated through a software program by utilizing the fact that the brightness of the touch screen is different from the brightness of the periphery when the touch screen is displayed. The first calibration point and the second calibration point may be any two different points on the touch screen of the device under test. In this embodiment, as shown in fig. 3, B' is a first area of the device under test, with the point at the top left corner (x1, y1) being a first calibration point and the point at the bottom right corner (x2, y2) being a second calibration point.
In step S730, the execution terminal is controlled to move from the origin position so that the execution terminal is aligned with the first calibration point in the real-time captured image, and the first actual coordinate of the execution terminal is obtained.
Specifically, the robot arm may be controlled to perform the X-axis direction and the Y-axis direction movement in the first region by sending a movement command to the robot arm, and the sent movement command may be stored. The movement instructions may be sent to the robot in pulses, for example, sending 1000 pulses to advance the robot 5mm along the X-axis. First movement data of the execution tip from the origin can be derived from the movement instruction transmitted during the movement of the execution tip of the robot arm from the origin position to the alignment with the first calibration point. The first movement data includes movement data in an X-axis direction and a Y-axis direction. Further, a first actual coordinate of the execution end may be obtained from the first movement data. For example, if the coordinates of the origin position are (0, 0) and the first movement data are (87mm, 52mm), it is obvious that the first actual coordinates are (87, 52) (unit: mm).
In step S740, the execution terminal is controlled to move so that the execution terminal is aligned with the second calibration point in the real-time photographed image, and a second actual coordinate of the execution terminal is obtained.
Specifically, the execution tip may be controlled to move from the origin position to be aligned with the second calibration point, and may also be controlled to move from the first calibration point to the second calibration point.
And step S750, acquiring calibration data according to the screen coordinate of the first calibration point, the first actual coordinate and the second actual coordinate.
Referring to fig. 3, the robot arm has a coordinate unit of millimeter (mm), an origin position of the robot arm is Machine _0, and X-and Y-axes of a coordinate system of the robot arm are respectively denoted as Machine _ X and Machine _ Y.
In this embodiment, the display interface is a PC display interface, and therefore, in fig. 3, the origin of coordinates corresponding to the display interface is denoted as PC _0, the X-axis and the Y-axis of the coordinate system of the display interface are denoted as PC _ X and PC _ Y, respectively, the coordinate unit is a pixel (pixel), and the unit of the first area (B' area) framed is also a pixel (pixel).
In fig. 3, (x1, y1) and (x2, y2) are the screen coordinates of the first calibration point and the second calibration point, respectively. The actual coordinates of the two calibration points of the first region (B' region) corresponding to the execution end of the robot arm are (Xadj1, Yadj1) and (Xadj2, Yadj2), respectively. Here, (Xadj1, Yadj1) corresponds to (x1, y1), and (Xadj2, Yadj2) corresponds to (x2, y 2).
From fig. 3, calibration data may be obtained, the calibration data comprising: a mapping relationship X _ scale of the actual movement distance of the X-axis of the execution tip to the display movement distance, and a mapping relationship Y _ scale of the actual movement distance of the X-axis of the execution tip to the display movement distance. In fig. 3, the calibration data can be understood as a proportional relationship of how many pixels on the display interface correspond to each unit (mm) of movement of the robot arm in the X-axis direction and the Y-axis direction. Namely, the method comprises the following steps:
X_scale＝(x2-x1)/(Xadj2-Xadj1)
Y_scale＝(y2-y1)/(Yadj2-Yadj1)
if the image coordinate point corresponding to the control to be operated is (x, y), the actual coordinate (Xm, Ym) of the control corresponding to the control to be operated can be calculated in the following manner, and the calculation method is as follows:
Xm＝Xadj1+(x-x1)/X_scale
Ym＝Yadj1+(y-y1)/Y_scale
because the camera is generally erected above the equipment to be operated, when the same object in the field of view of the camera device with a certain erection height is at different distances from the camera, the size displayed in the preview or imaging of the camera is different. That is to say, the thickness of the equipment to be measured can lead to the formation of image that the camera obtained not of uniform size, and then leads to the control of arm to appear the error. Compared with thinner mobile equipment such as a tablet personal computer and a smart phone, when the control operation is tested on equipment with larger thickness such as an attendance machine, the automatic test on the control can hardly be carried out due to the great error caused by the difference of imaging.
According to the embodiment, two calibration points are selected in the touch screen area of the equipment to be tested for different types of equipment to be tested, calibration is carried out, calibration data is obtained, the image coordinate to be operated and controlled is converted into the actual seat end of the execution tail end of the mechanical arm according to the calibration data, the control precision of the mechanical arm for the equipment to be operated with different thicknesses can be improved, and the purpose of accurately touching the control in the screen of the equipment to be operated is achieved.
And S400, controlling the execution tail end of the mechanical arm to move to the actual coordinate of the control, and executing a first operation on the touch screen of the device to be tested.
Wherein the first operation is determined by a configuration of the test task.
In this embodiment, corresponding parameters of the robot arm are also configured, and taking a three-axis robot arm as an example, the parameters include moving speeds of two axial directions (i.e., an X axis and a Y axis) of a horizontal plane, moving speed in a Z axis direction perpendicular to the horizontal plane, touch delay, whether the robot arm returns to an original point after each operation is completed, and the like.
Specifically, the height of the execution tail end of the mechanical arm is adjusted according to the thickness of the equipment to be tested; then, according to the axial moving speed of the horizontal plane of the mechanical arm, the execution tail end of the mechanical arm moves to the actual coordinate of the control; and waiting for touch delay, and executing a first operation on the touch screen of the equipment to be tested.
Step S500, second image data displayed in the touch screen of the device to be tested is obtained, and a test result of the test task is determined according to the second image data.
And the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
In this embodiment, a corresponding test report may also be generated according to the test result of the test task, where the report includes, for example: the type of the device to be tested, the execution starting and ending time (including the starting time and the ending time) of the test task, the total number of the test cases, the total number of the operation controls, the number of the pass/fail items and the running result of each test case.
According to the method, the UI interface control attributes are not acquired, the control of the touch screen interface can be accurately positioned through machine vision, and the manipulator is controlled to carry out UI interface control operation on the touch screen products, so that the actual use scene of a user is simulated. The programming of an operating system depending on a product is avoided, the technical requirements on testers are reduced, and the testing efficiency is improved.
Referring to fig. 4, the present embodiment discloses a touch screen testing apparatus 400, which includes, but is not limited to, a first module 410, a second module 420, a third module 430, a fourth module 440, and a fifth module 450.
The first module 410 is configured to obtain first image data displayed in a touch screen of a device under test in real time. The second module 420 is configured to obtain feature data of the control to be operated according to the configuration information of the test task, and determine whether the control to be operated exists in the first image data according to the feature data. A third module 430, configured to, if it is determined that the control to be operated exists, obtain an image coordinate of the control to be operated in the first image data, and obtain a corresponding actual coordinate of the control according to the image coordinate. And the actual coordinates of the control are used for representing the position of the control to be operated on the touch screen side of the equipment to be tested. And a fourth module 440, configured to control the execution end of the manipulator to move to the actual coordinate of the control, and execute a first operation on the touch screen of the device under test, where the first operation is determined by the configuration of the test task. A fifth module 450, configured to obtain second image data displayed in the touch screen of the device under test, and determine a test result of the test task according to the second image data. And the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
The touch screen testing device shown in fig. 4 is based on the same inventive concept as the touch screen testing method shown in fig. 1, and therefore, the emotion recognition device has at least the same beneficial effects as the touch screen testing method shown in fig. 1 under the synergistic effect of the modules.
Referring to fig. 5, the present embodiment provides a touch screen testing system, including: a robotic arm 510, a camera 520, and a computer device 530. As shown in fig. 5, the mechanical arm 510 is connected to a computer device 530 through a serial port, receives an instruction from the computer device to control the execution terminal to move, and performs a touch operation on the touch screen of the device to be tested. The camera device 520 is erected above the device to be tested, and is configured to capture an image displayed on a touch screen of the device to be tested in real time and send the captured image to the computer device 530. A computer device 530 comprising at least one processor and at least one memory storing at least one program which, when executed by the at least one processor, causes the at least one processor to implement the touch screen testing method as described above. The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip having signal processing capabilities. The processor may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The general purpose processor may be a microprocessor, but in the alternative, the general purpose processor may be any conventional processor or the like. The memory may include various components (e.g., machine-readable media), including but not limited to random access memory components, read only components, and any combination thereof. It is to be understood that, for avoiding redundancy, the content not referred to in this embodiment may refer to the content not referred to in this embodiment and refer to the touch screen testing method described above.
The embodiment provides a computer-readable storage medium, in which program instructions are stored, and when the program instructions are executed by a processor, the touch screen testing method can be implemented. It is to be understood that, for avoiding redundancy, the touch screen test method described above may be referred to for what is not referred to in this embodiment.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods disclosed above, functional modules/units in the devices, may be implemented as software, firmware, hardware, and suitable combinations thereof.
In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer-readable media (storage media for short), which can include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer readable media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and are not to be construed as limiting the scope of the invention. Any modifications, equivalents and improvements which may occur to those skilled in the art without departing from the scope and spirit of the present invention are intended to be within the scope of the claims.
Claims (10)
1. A touch screen testing method is characterized by comprising the following steps:
acquiring first image data displayed in a touch screen of equipment to be tested in real time;
acquiring feature data of a control to be operated according to configuration information of a test task, and determining whether the control to be operated exists in the first image data according to the feature data;
if the control to be operated is determined to exist, acquiring an image coordinate of the control to be operated in the first image data, and acquiring a corresponding control actual coordinate according to the image coordinate, wherein the control actual coordinate is used for representing the position of the control to be operated on the touch screen side of the equipment to be operated;
controlling an execution tail end of a mechanical arm to move to the actual coordinates of the control, and executing a first operation on a touch screen of the equipment to be tested, wherein the first operation is determined by the configuration of the test task;
acquiring second image data displayed in a touch screen of the equipment to be tested, and determining a test result of the test task according to the second image data; and the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
2. The touch screen testing method according to claim 1, wherein the feature data of the control to be operated includes third image data of the control to be operated; the determining whether the control to be operated exists in the first image data according to the feature data includes:
acquiring a matching result through image identification, wherein the matching result is used for representing whether an area matched with the third image data exists in the first image data;
and if the matching result is that an area matched with the third image data exists, determining that the control to be operated exists in the first image data.
3. The touch screen testing method according to claim 1 or 2, wherein the feature data of the control to be operated comprises first characters, and the first characters are used for representing character information displayed by the control to be operated; the determining whether the control to be operated exists in the first image data according to the feature data includes:
acquiring a character recognition result of the first image data;
and if the character recognition result comprises the first characters, determining that the control to be operated exists in the first image data.
4. The touch screen testing method of claim 1, wherein the configuration information of the test task comprises: the first position of the control to be operated is used for representing partition information where the control to be operated is displayed on the touch screen of the equipment to be tested; the determining whether the control to be operated exists in the first image data according to the feature data comprises:
acquiring a corresponding identification area image from the first image data according to the first position;
and determining whether the control to be operated exists in the image of the identification area according to the feature data.
5. The touch screen testing method of claim 1, wherein the obtaining of the corresponding actual coordinates of the control according to the image coordinates comprises:
acquiring corresponding calibration data according to the type of the equipment to be tested; the calibration data is used for representing the mapping relation between the actual moving distance of the execution tail end and the image display moving distance of the touch screen of the equipment to be tested;
and converting the image coordinates into actual coordinates of the control according to the calibration data.
6. The touch screen testing method of claim 5, wherein the calibration data obtaining method comprises:
displaying a real-time shot image of the touch screen of the equipment to be tested through an interactive interface;
in the real-time shot image, determining a touch screen area of the equipment to be tested as a first area, and determining a first calibration point and a second calibration point in the first area;
controlling the executing tail end to move from an original point position so that the executing tail end is aligned with the first calibration point in the real-time shooting image to obtain a first actual coordinate of the executing tail end;
controlling the execution terminal to move so that the execution terminal is aligned with the second calibration point in the real-time shot image, and obtaining a second actual coordinate of the execution terminal;
and acquiring the calibration data according to the screen coordinate of the first calibration point, the first actual coordinate and the second actual coordinate.
7. The touch screen testing method of claim 1, further comprising configuring an axial movement speed and a touch delay of a horizontal plane of the robotic arm; the control is that the execution tail end of the mechanical arm moves to the actual coordinate of the control, and the execution of a first operation on the touch screen of the equipment to be tested comprises the following steps:
adjusting the height of the execution tail end according to the thickness of the equipment to be tested;
controlling the execution tail end of the mechanical arm to move to the actual coordinate of the control according to the axial moving speed of the horizontal plane of the mechanical arm;
and waiting for the touch delay, and executing the first operation on the touch screen of the equipment to be tested.
8. A touch screen testing device, comprising:
the first module is used for acquiring first image data displayed in a touch screen of the equipment to be tested in real time;
the second module is used for acquiring feature data of a control to be operated according to configuration information of a test task and determining whether the control to be operated exists in the first image data or not according to the feature data;
a third module, configured to, if it is determined that the control to be operated exists, obtain an image coordinate of the control to be operated in the first image data, and obtain a corresponding actual control coordinate according to the image coordinate, where the actual control coordinate is used to represent a position of the control to be operated on a touch screen side of the device to be tested;
the fourth module is used for controlling the execution tail end of the mechanical arm to move to the actual coordinates of the control and executing first operation on the touch screen of the equipment to be tested, wherein the first operation is determined by the configuration of the test task;
the fifth module is used for acquiring second image data displayed in the touch screen of the device to be tested and determining a test result of the test task according to the second image data; and the second image data is used for representing a display image of the touch screen of the equipment to be tested after the first operation is executed.
9. A touch screen testing system, comprising:
the mechanical arm is connected with the computer equipment through a serial port and used for receiving an instruction of the computer equipment to control the execution tail end to move and execute touch operation on the touch screen of the equipment to be tested;
the camera device is erected above the equipment to be tested and is used for shooting images displayed on a touch screen of the equipment to be tested in real time;
computer apparatus comprising at least one processor, and at least one memory storing at least one program which, when executed by the at least one processor, causes the at least one processor to carry out a touch screen testing method according to any one of claims 1 to 7.
10. A computer readable storage medium having stored therein program instructions which, when executed by a processor, implement the touch screen testing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210099863.3A CN114546740A (en) | 2022-01-27 | 2022-01-27 | Touch screen testing method, device and system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210099863.3A CN114546740A (en) | 2022-01-27 | 2022-01-27 | Touch screen testing method, device and system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114546740A true CN114546740A (en) | 2022-05-27 |
Family
ID=81674251
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210099863.3A Pending CN114546740A (en) | 2022-01-27 | 2022-01-27 | Touch screen testing method, device and system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114546740A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116955207A (en) * | 2023-09-18 | 2023-10-27 | 中国兵器装备集团兵器装备研究所 | Automatic test method, system and medium for identifying software and hardware completed by test case |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103235661A (en) * | 2013-04-13 | 2013-08-07 | 北京汇冠新技术股份有限公司 | Positioning clicking device and method and touch display screen calibration system and method |
| CN103472953A (en) * | 2013-09-25 | 2013-12-25 | 成都吉锐触摸技术股份有限公司 | Calibration method of touch display screen |
| CN109815947A (en) * | 2018-12-14 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Test method, device and the computer equipment of application |
-
2022
- 2022-01-27 CN CN202210099863.3A patent/CN114546740A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103235661A (en) * | 2013-04-13 | 2013-08-07 | 北京汇冠新技术股份有限公司 | Positioning clicking device and method and touch display screen calibration system and method |
| CN103472953A (en) * | 2013-09-25 | 2013-12-25 | 成都吉锐触摸技术股份有限公司 | Calibration method of touch display screen |
| CN109815947A (en) * | 2018-12-14 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Test method, device and the computer equipment of application |
Non-Patent Citations (1)
| Title |
|---|
| 刘淼: "《嵌入式系统接口设计与Linux驱动程序开发》", 31 May 2006, 《北京航空航天大学出版社》, pages: 101 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116955207A (en) * | 2023-09-18 | 2023-10-27 | 中国兵器装备集团兵器装备研究所 | Automatic test method, system and medium for identifying software and hardware completed by test case |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6960980B2 (en) | Image-based tray alignment and tube slot positioning in visual systems | |
| US9756261B2 (en) | Method for synthesizing images and electronic device thereof | |
| CN106845508B (en) | The methods, devices and systems of release in a kind of detection image | |
| CN103377119A (en) | Automatic nonstandard control testing method and device | |
| CN110666359A (en) | Laser marking method, marking device and computer readable storage medium | |
| US20200164520A1 (en) | Image processing device, control method thereof, and program storage medium | |
| CN114546740A (en) | Touch screen testing method, device and system and storage medium | |
| CN114952856A (en) | Mechanical arm hand-eye calibration method, system, computer and readable storage medium | |
| CN113172636B (en) | Automatic hand-eye calibration method and device and storage medium | |
| CN113310997A (en) | PCB defect confirmation method and device, automatic optical detection equipment and storage medium | |
| CN110853102A (en) | Novel robot vision calibration and guide method, device and computer equipment | |
| CN114184143A (en) | PCB flatness measuring method and device | |
| CN111901589A (en) | Automatic test system for television mainboard | |
| CN115914616A (en) | Light source point detection method, device, equipment and computer readable storage medium | |
| CN114543669B (en) | Mechanical arm calibration method, device, equipment and storage medium | |
| CN110460841B (en) | Television detection method and device, electronic equipment and storage medium | |
| CN112819760A (en) | Image detection method, device and system | |
| CN117325160A (en) | Mechanical arm calibration method and device, electronic equipment and storage medium | |
| TWI799025B (en) | Processing system, device and method for automatic testing interface card | |
| CN113643358B (en) | External parameter calibration method, device, storage medium and system of camera | |
| KR102597459B1 (en) | System and method of correcting of image for diagnoisis | |
| CN111060800B (en) | Flying probe testing method, flying probe testing device, flying probe testing apparatus, and storage medium | |
| US11892923B2 (en) | Testing electronic products for determining abnormality | |
| US20230364798A1 (en) | Information processing method, image processing method, robot control method, product manufacturing method, information processing apparatus, image processing apparatus, robot system, and recording medium | |
| CN112750188B (en) | Method and terminal for automatically rendering object |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |