CN105653460A - Method and system for recording and playback of man-machine interface testing actions of touch screen product - Google Patents
Method and system for recording and playback of man-machine interface testing actions of touch screen product Download PDFInfo
- Publication number
- CN105653460A CN105653460A CN201610114356.7A CN201610114356A CN105653460A CN 105653460 A CN105653460 A CN 105653460A CN 201610114356 A CN201610114356 A CN 201610114356A CN 105653460 A CN105653460 A CN 105653460A
- Authority
- CN
- China
- Prior art keywords
- test
- touch screen
- actuator
- recording
- machine interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 290
- 230000009471 action Effects 0.000 title claims abstract description 137
- 238000000034 method Methods 0.000 title claims abstract description 29
- 230000009466 transformation Effects 0.000 claims description 17
- 230000005540 biological transmission Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 description 13
- 239000011159 matrix material Substances 0.000 description 8
- 238000005070 sampling Methods 0.000 description 8
- 238000013515 script Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 239000000243 solution Substances 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000012482 calibration solution Substances 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 210000001508 eye Anatomy 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 210000000214 mouth Anatomy 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2205—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested
- G06F11/2221—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested to test input/output devices or peripheral units
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a method and system for recording and playback of man-machine interface testing actions of a touch screen product. The method comprises the steps that S10, the man-machine interface testing actions are recorded; S20, the man-machine interface testing actions recorded in the S10 are played back. The system comprises a recording module used for recording the man-machine interface testing actions, a playback module used for playing back the man-machine interface testing actions recorded by the recording module and an actuator used for executing the testing actions on a man-machine interface of the touch screen product according to an obtained testing action instruction. The method and system are used for automatic testing of the man-machine interface of the touch screen product and applicable to touch screen products produced by different manufacturers.
Description
Technical Field
The invention relates to the field of touch screen type product man-machine interface testing, in particular to a method and a system for recording and playing back touch screen type product man-machine interface testing actions.
Background
The man-machine interface test of the touch screen type product aims to test whether the performance of the whole machine meets the preset functional requirements. The main modes include manual testing, special machine testing and intelligent equipment testing.
The traditional human-computer interface test is mainly a manual test, but in recent years, the shipment volume of touch screen products such as smart phones and tablets is rapidly increased, the labor cost is continuously increased, and the limitation of the manual test is more and more obvious, such as low test efficiency, poor test consistency and negative influence caused by personnel mobility.
A special machine testing mode is adopted for part of the man-machine interfaces, although the special machine testing efficiency is high, the testing function is single, the model of a testing object is fixed, personnel are required to complete the operation of the machine and the judgment and recording of the testing result, the universality is not high, and the requirements can not be met gradually.
The intelligent test equipment is provided with a programmable control system, a flexible mechanism capable of simulating manual operation and an intelligent sensor capable of simulating sense organs such as human eyes, ears, mouths and the like, and can be quickly suitable for one-stop test and result analysis of product model updating and numerous test items. The intelligent test equipment can reduce labor cost and improve test efficiency and test quality. However, the intelligent test equipment belongs to high-end equipment, the technical content is high, the threshold is high, the test program and the script need to be written, and the later maintenance work is heavy.
Disclosure of Invention
The technical problem is as follows: the technical problem to be solved by the invention is as follows: the method and the system for recording and replaying the test action of the man-machine interface of the touch screen type product are used for automatically testing the man-machine interface of the touch screen type product and are generally used for touch screen type products produced by different manufacturers.
The technical scheme is as follows: in order to solve the technical problem, the embodiment of the invention adopts the following technical scheme:
in a first aspect, the present embodiment provides a method for recording and playing back a human-machine interface test action of a touch-screen product, including the following steps:
s10 recording the man-machine interface test action;
s20 plays back the human-machine interface test action recorded in S10.
With reference to the first aspect, as a first possible implementation manner, the S10 further includes:
s101) taking a qualified touch screen type product as a prototype, collecting an image of a man-machine interface of the prototype, transmitting the image to a touch screen teaching terminal which is provided with a touch screen and can record the position and time sequence of an operation point in real time, and displaying the man-machine interface image in real time by the touch screen teaching terminal;
s102) saving a teaching action finished on the touch screen teaching terminal as a sequence of positions and time of a group of operation points;
s103) converting the sequence of the positions and the time of the group of operation points into a test action command, and transmitting the test action command to an actuator for execution;
s104) acquiring an execution result image of the actuator on the prototype, and taking the execution result image as a standard test result image;
s105) storing the standard test result image in the S104) and a corresponding test action instruction;
s106) returns to step S102 until the recording of all teaching actions for testing is completed.
With reference to the first possible implementation manner of the first aspect, as a second possible implementation manner, in S103), the specific process of transforming the sequence of positions and times of the group of operation points into a test action command is: and converting the position sequence of the group of operation points from a touch screen coordinate system of a touch screen teaching terminal to an actuator end pose sequence under the actuator coordinate system through an image coordinate system, performing kinematic inversion on the actuator end pose sequence to generate a joint position of the actuator, generating an interpolation track, namely a motion track of the actuator, from the joint position and a corresponding time sequence, and converting the motion track of the actuator into a corresponding actuator execution instruction to form a test action instruction.
With reference to the first aspect, as a third possible implementation manner, the S20 further includes:
s201) transmitting the nth test action command to the actuator for execution, where n is 1+ q, q is the number of times that step S204) returns to step S201), and the initial value of q is 0;
s202) acquiring an execution result image of an actuator on a touch screen type product of the same type as a prototype;
s203) comparing the acquired test result image with the standard test result image, if the acquired test result image is consistent with the standard test result image, the test comparison result is passed, and the step S204) is executed; if not, the comparison result of the test is failed, and the test is finished or the step S204 is entered);
s204) returns to step S201) until all test action instructions have been executed.
With reference to the third possible implementation manner of the first aspect, as a fourth possible implementation manner, the S20 further includes S205): replacing the products to be tested with the same model as the prototype, and returning to S201) until the test of all the products to be tested in the same batch is completed.
In a second aspect, the present embodiment provides a system for recording and playing back a man-machine interface test action of a touch-screen product, the system comprising:
a recording module: the device is used for recording the human-computer interface test action;
a playback module: the device is used for playing back the human-computer interface test action recorded by the recording module;
an actuator: and the testing device is used for executing the testing action on the man-machine interface of the touch screen type product according to the acquired testing action instruction.
With reference to the second aspect, as a first possible implementation manner, the recording module further includes:
a first transmission submodule: the touch screen teaching terminal is used for transmitting the human-computer interface image displayed by the human-computer interface of the prototype to the touch screen teaching terminal which is provided with a touch screen and can record the position and the time sequence of an operation point;
a recording submodule: the teaching terminal is used for recording teaching actions finished on the touch screen teaching terminal and storing the teaching actions as a sequence of positions and time of a group of operation points;
a transformation submodule: the sequence of the positions and the time of the group of operation points is converted into a test action command, and the test action command is transmitted to the actuator;
a first acquisition submodule: the system comprises a test system, a test device and a control device, wherein the test system is used for acquiring a result image of a test action instruction executed by an actuator on a prototype human-computer interface and taking the result image as a standard test result image;
saving the submodule: and the standard test result image and the corresponding test action instruction are stored.
With reference to the first possible implementation manner of the second aspect, as a second possible implementation manner, the transformation submodule further includes:
a conversion unit: the position sequence of the group of operation points is converted into an actuator terminal pose sequence under an actuator coordinate system through a touch screen coordinate system of a touch screen teaching terminal through an image coordinate system;
a first generation unit: the system is used for performing kinematic inversion on the end pose sequence of the actuator to generate the joint position of the actuator;
a second generation unit: the system is used for generating an interpolation track, namely a motion track of an actuator, according to the joint position and the corresponding time sequence;
a third generation unit: and the test motion command is generated by converting the motion trail of the actuator.
With reference to the second aspect, as a third possible implementation manner, the playback module further includes:
a second transmission submodule: the test action command is transmitted to the executor;
a second acquisition submodule: the test device is used for acquiring a test result of the executor executing the received test action instruction on the product to be tested with the same model as the prototype;
a comparison submodule: comparing the test result acquired by the second acquisition submodule with the standard test result image, and if the test result is consistent with the standard test result image, judging that the test comparison result is passed; if not, the comparison result of the test is failed.
Has the advantages that: compared with the prior art, the technical scheme of the embodiment of the invention has the following beneficial effects: all programmable test equipment at present needs to prefabricate or write test scripts or codes, has high technical threshold and needs to be completed by professional personnel. Meanwhile, programming languages and script formats of different manufacturers are different, so that the method is not universal and difficult to transplant. The embodiment of the invention reduces the use threshold and the programming cost of the equipment, realizes the portability of the test case among different equipment in a factory, realizes the automatic generation of the test program, or can obtain the test case without programming. The robot teaching method based on the touch screen and the vision in the embodiment of the invention adopts the existing principle of teaching of off-line programming and field separation and the on-line mode of teaching box programming, becomes a brand new program automatic generation technology superior to the existing teaching programming technology by integrating the touch screen, the vision and the robot control technology, and is a breakthrough of the robot programming technology. According to the embodiment, robot language programming or test script writing is completely not needed, the required test actions can be demonstrated once only by performing touch operation on the image interface of the tested object, the test case can be easily obtained, then the same test can be repeatedly executed by the machine, all product test tasks of the same batch are completed, and testers are thoroughly liberated from fussy test preparations.
Drawings
FIG. 1 is a flow chart of a method of an embodiment of the present invention.
Fig. 2 is a block diagram of the flow of step S10 in the embodiment of the present invention.
Fig. 3 is a block diagram of the flow of step S20 in the embodiment of the present invention.
Fig. 4 is a block diagram of a recording module in the system according to the embodiment of the present invention.
Fig. 5 is a block diagram of a transformation submodule in the system according to the embodiment of the present invention.
Fig. 6 is a block diagram of a playback module in the system according to an embodiment of the present invention.
Detailed Description
The technical solution of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a method for recording and playing back a man-machine interface test action of a touch screen product of the present embodiment includes the following steps:
s10 recording the man-machine interface test action;
s20 plays back the human-machine interface test action recorded in S10.
The method of the embodiment tests the man-machine interface of the touch screen type product, firstly recording the test action of a prototype (the prototype is a qualified touch screen type product) to form a standard test result; and then testing the touch screen type product to be tested, and comparing the test result with the standard test result to obtain the information whether the test result is accurate or not so as to complete the test of the man-machine interface of the touch screen type product. The whole testing process comprises two steps of recording the human-computer interface testing action and replaying the human-computer interface testing action. The testing method of the embodiment of the invention is different from the existing manual testing. The embodiment of the invention combines the robot technology and the intelligent vision processing and analyzing technology, the machine executes the test action instead of manual execution, and a worker can easily obtain the test case only by demonstrating the required test action once through touch operation on the image interface of the tested object, and then the machine can repeatedly execute the same test to complete the test tasks of all products in the same batch, thereby completely liberating the tester from fussy test preparations. The mode of manual teaching recording and playback is adopted, and any test case can be adopted regardless of tablet computers or mobile phones of different models. Different from the existing special machine test, one special machine can only be used for testing products of one type, and the embodiment of the invention uses the actuator with more than three degrees of freedom to execute the test action instead of the special machine test, thereby improving the universality. Different from the existing testing means, the programming languages and script formats of the testing equipment from different manufacturers are different, are not universal and are difficult to transplant, and for the actuators which are produced by different manufacturers or have more than three degrees of freedom and different models, the embodiment of the invention can realize the reproduction and transplant of functions only by acquiring corresponding actuator parameters and a method of kinematic forward and inverse solution. Different from the existing programmable test equipment which needs to prefabricate or write test scripts or codes, the technical threshold is high, and the test equipment needs to be completed by professional personnel, the test program in the embodiment of the invention can be automatically generated, the professional personnel is not needed to write the test program and the scripts, and the technical threshold is reduced.
As a preferred example, as shown in fig. 2, the S10 further includes:
s101) taking a qualified touch screen type product as a prototype, collecting images of a man-machine interface of the prototype, transmitting the images to a touch screen teaching terminal which is provided with a touch screen and can record positions and time sequences of operation points in real time, and receiving and displaying the man-machine interface images in real time by the touch screen teaching terminal.
In the step S101), the human-computer interface image of the qualified touch screen type product can be sampled in real time through image sampling equipment such as an industrial camera and the like, and is transmitted to a touch screen teaching terminal for displaying. The touch screen teaching terminal can be a tablet computer of a windows7 system. In other embodiments, a tablet computer or a touch screen mobile phone or a computer provided with an operating system such as android series, saiban series, Unix, Linux and the like can be selected. In other embodiments, the human-computer interface image of the touch screen type product can be transmitted to the teaching terminal through the upper computer by adopting the image sampling device instead of being directly transmitted to the teaching terminal.
S102) saving a teaching action finished on the touch screen teaching terminal as a sequence of positions and time of a group of operation points.
And in step S102), operating the human-computer interface image displayed on the touch screen teaching terminal to finish teaching actions for testing. And recording the teaching action, and recording the teaching action as a sequence of the positions and time of a group of operation points. The specific actions for operating the human-computer interface image displayed on the touch screen teaching terminal comprise clicking, double clicking, sliding, dragging and the like. A user can use a finger or a touch pen to execute a test action on a touch screen product man-machine interface displayed on the touch screen teaching terminal, the test action comprises clicking, double clicking, sliding and dragging, and the sequence of the position and the time of an operation point recorded by the touch screen teaching terminal can be extracted through programming. In this embodiment, touch data can be acquired by calling an API function on a touch screen teaching terminal in the windows7 system, so as to obtain a sequence of positions and times of operation points, and the sequence of positions and times of the operation points is sent to an upper computer by the touch screen teaching terminal. It is understood that in other embodiments, touch data may be obtained through a corresponding system interface for other types of touch screen devices as well.
S103) converting the sequence of the positions and the time of the group of operation points into a test action command, and transmitting the test action command to the actuator.
In this embodiment, the sequence of the position and time of the operating point can be converted into a test action command on the upper computer and transmitted to the control unit of the actuator, and the control unit of the actuator controls the execution mechanism to execute the test action.
And converting the sequence of the positions and the time of the group of operation points into a test action command, taking a point-to-point track of the six-axis robot as an example, calling a corresponding robot control function Move to generate a command moveARMtoA, and thus realizing the motion of the tail end of the robot to the point A in the track.
The specific process of converting the sequence of the positions and the time of the group of operation points into a test action command is as follows: and converting the position sequence of the group of operation points from a touch screen coordinate system of a touch screen teaching terminal to an actuator end pose sequence under an actuator coordinate system through an image coordinate system, performing kinematic inversion on the actuator end pose sequence to generate a joint position of the actuator, generating an interpolation track by the joint position and a corresponding time sequence, namely generating a motion track of the actuator, and converting the motion track of the actuator into a corresponding actuator execution instruction, namely forming a test action instruction.
In this embodiment, the position sequence of the operation point is transformed from the touch screen coordinate system to the actuator end pose sequence in the actuator coordinate system through the image coordinate system. In the implementation, a space coordinate system is established firstly, including an actuator world coordinate system { W }, a camera coordinate system { C }, and an image coordinate system { U }. The transformation sequence of each operation point between the coordinate systems is as follows: firstly, the image coordinate system { U } is transformed into a camera coordinate system { C }, and then the camera coordinate system { C } is transformed into an actuator world coordinate system { W }. For example: knowing the corresponding pose of a point P in the image coordinate system { U }The point is formulated from the image coordinate system { U }Transformation to the camera coordinate system C,is the pose of the operation point on the man-machine interface of the lower model machine { C },a matrix of intrinsic parameters representing the camera is shown,and (4) obtaining the target through camera calibration solution. From the camera coordinate system { C } equationTransforming into an actuator world coordinate system W,is the C to W transform matrix,is the pose of the operation point on the man-machine interface of the lower model machine { W }. Then by the formulaTransforming the pose of the operating point on the man-machine interface of the lower model machine into the pose T of the end of the actuator6，Is a known quantity, similar to the relationship of a gripper and a cube.Can be obtained by means of a target surface. Establishing a target surface coordinate system { B }, calibrating by a relation based on space geometric vectors, and then calculating a transformation matrix from { B } to { W }And extracting characteristic points after shooting the target surface by a camera, and solving a transformation matrix from { B } to { C } by utilizing a P3P space point pose solving algorithmAnd then haveWherein,is the C to B transform matrix.
The coordinate systems are not uniquely established, and some intermediate coordinate systems such as the workpiece coordinate system can be added, or the relationship between some coordinate systems can be determined during calibration, so that some coordinate systems are omitted.
In this embodiment, the end pose sequence of the actuator is subjected to inverse kinematics solution to generate the joint position of the actuator, then trajectory planning is performed according to the corresponding time sequence to generate the motion trajectory of the actuator, and then the joint position and the motion trajectory of the actuator are combinedThe trace translation generates test action instructions. In the specific implementation, taking a six-joint robot as an example, firstly establishing a connecting rod coordinate system of the robot according to D-H (translating Chinese into connecting rod parameters), and then calculating the inverse kinematics of the robot according to the end pose of an actuator. The specific process is that a motion equation T containing joint positions and the end pose of the actuator is established according to the connecting rod parameters6＝A1A2A3A4A5A6，AiAnd i is an integer of 1-6 and is a pose matrix of each joint. A. theiFrom the position theta of each jointiDetermining and solving the motion equation to obtain the positions of the joints corresponding to the end pose of the actuator, wherein 6 unknowns are required for 6 equations, and the variables of the joints are required to be separated for solving, and the specific solving mode is as follows: for known pose transformation matrix T6By left or right multiplication by an Ai -1And the right side of the equation has no corresponding joint position, and elements of sine and cosine values corresponding to the joint position can be found from the joint position, so that the angle is obtained. A. thei -1Is AiThe inverse matrix of (c). And then generating an interpolation track according to the corresponding time sequence. Taking the generation of the interpolation track from point to point as an example, the specific implementation mode is that a motion function of the joint position is constructed, for example, a cubic polynomial with the joint position equal to time is made, so that the function must meet the initial condition of the joint position given by the inverse kinematics solution, the motion equation meeting the condition is finally solved to be the solved track, then a series of intermediate points are uniformly selected on the obtained track, a series of points through which the end of the actuator needs to pass are obtained, and the solved interpolation track is obtained. And finally, sequentially calling corresponding control instructions according to the interpolation track and the corresponding time sequence thereof to generate a test action instruction. Taking a point-to-point track of a six-axis robot as an example, a specific implementation manner may be that a corresponding robot control function move is called, and a generated instruction is moveARMtoA, so that the motion that the tail end of the robot reaches a point a can be realized.
S104) acquiring an execution result image of the actuator on the prototype, and taking the execution result image as a standard test result image.
After the execution mechanism receives the test action instruction, the test action is executed on the sample, the upper computer sends a sampling instruction to the image sampling equipment, and the image sampling equipment samples and executes a result image. Meanwhile, the execution result image is taken as a standard test result.
S105) storing the standard test result image in S104) and the corresponding test action command.
Storing the standard test result image in the S104) and the corresponding test action instruction can be completed in the upper computer. Of course, the test action command may be stored locally by the upper computer, or may be generated and exported.
In this embodiment, after the executor finishes executing during recording, the user may manually confirm whether the execution result image is the required standard test result image, and if the requirement is met, the recording is confirmed to be successful; if not, S10 is repeatedly executed until an execution result image satisfying the requirements is obtained as the standard test result image. Thus, the same batch of products can be played back by only completing the recording process for one sample product.
As a preferred example, as shown in fig. 3, the S20 further includes:
s201) transmits the nth test action command to the actuator, where n is 1+ q, q is step S204) and returns to step S201), and the initial value of q is 0.
S202) acquiring a test result of the test action instruction received by the executor to the same type of touch screen type product to be tested.
S203) comparing the acquired test result with the standard test result image, if the acquired test result is consistent with the standard test result image, the test comparison result is passed, and the step S204) is executed; if not, the comparison result of the test is failed, and the test is ended or the process proceeds to step S204). If the test is inconsistent, the test is continued or stopped, the test can be set by the user before the test, and then the system executes the test according to the setting of the user.
S204) returns to step S201) until all test action instructions are completed.
In this embodiment, during playback, the upper computer may directly transmit the locally stored test action command to the control unit of the actuator, or may import the locally stored test action command into the control unit of the actuator. After receiving a test action instruction called locally or from other places by an upper computer, an execution mechanism of the actuator executes the test action instruction on the same type of touch screen type product to be tested. The image sampling equipment can be used for collecting the image displayed after the same model touch screen type product to be tested executes the test action instruction, and the image is used as the test result. The image sampling device can transmit the test result to the upper computer, and the test result is compared with the standard test result in the upper computer. The comparison method between the images can adopt the prior image comparison technology such as SIFT algorithm and the like for comparison. The comparison results are two types, one is that the test result is consistent with the standard test result, and the other is that the test result is inconsistent with the standard test result. And recording and storing the comparison result. The comparison result can be stored in the upper computer or other equipment with a storage function.
As a preferred example, the step S20 further includes S205: replacing the products to be tested, and returning to S201) until the testing of all the products to be tested in the same batch is completed.
The embodiment provides a system for recording and playing back a man-machine interface test action of a touch screen type product, which comprises the following steps:
a recording module: the device is used for recording the human-computer interface test action;
a playback module: the device is used for playing back the human-computer interface test action recorded by the recording module;
an actuator: and the testing device is used for executing the testing action on the man-machine interface of the touch screen type product according to the acquired testing action instruction.
In the embodiment, a recording module and an actuator are utilized to record the test action to form a standard test result; and then testing the touch screen type product to be tested by utilizing the playback module and the actuator, and comparing the test result with the standard test result so as to obtain the information whether the test result is accurate or not, thereby completing the test of the man-machine interface of the touch screen type product.
As a preferred example, as shown in fig. 4, the recording module further includes:
a first transmission submodule: the touch screen teaching terminal is used for transmitting the human-computer interface image displayed by the human-computer interface of the prototype to the touch screen teaching terminal which is provided with a touch screen and can record the position and the time sequence of an operation point;
a recording submodule: the teaching terminal is used for recording teaching actions finished on the touch screen teaching terminal and storing the teaching actions as a sequence of positions and time of a group of operation points;
a transformation submodule: the sequence of the positions and the time of the group of operation points is converted into a test action command, and the test action command is transmitted to the actuator;
a first acquisition submodule: the system comprises a sample man-machine interface, a test result image acquisition unit and a test result image acquisition unit, wherein the sample man-machine interface is used for acquiring a result image of a test action instruction executed by an actuator on the sample man-machine interface and taking the result image as a standard test result image;
saving the submodule: and the standard test result image and the corresponding test action instruction are stored.
The first transmission submodule may be a sensor mounted on the actuator. And transmitting the human-computer interface image displayed by the human-computer interface of the sample to a touch screen teaching terminal by using the sensor. The recording submodule exists in the touch screen teaching terminal. And the recording sub-module records the teaching action finished on the touch screen teaching terminal as a sequence of the positions and time of a group of operation points. The transformation submodule is present in the upper computer. The first acquisition sub-module may be an image acquisition device. And the first acquisition sub-module acquires an execution result image of the actuator after the actuator executes the test action instruction on the sample man-machine interface as a standard test result. The storage submodule is arranged in the upper computer and used for storing a standard test result. The five sub-modules constitute a recording module. In the five sub-modules, a data output port of the transmission sub-module is connected with a data input port in the touch screen teaching terminal, a data output port of the recording sub-module is connected with a data input port of a transformation sub-module in the upper computer, a data output port of the transformation sub-module is connected with a data input port of the actuator, and a data output port of the first acquisition sub-module is connected with a data input port of a storage sub-module in the upper computer.
As a preferred example, as shown in fig. 5, the transformation submodule further includes:
a conversion unit: the position sequence of the group of operation points is converted into an actuator terminal pose sequence under an actuator coordinate system through a touch screen coordinate system of a touch screen teaching terminal through an image coordinate system;
a first generation unit: the system is used for performing kinematic inversion on the end pose sequence of the actuator to generate the joint position of the actuator;
a second generation unit: the system is used for planning a track according to the joint position and the corresponding time sequence and generating a motion track of the actuator;
a third generation unit: and the test motion command is generated by converting the motion trail of the actuator.
The conversion submodule is used for converting the sequence of the positions and the time of the group of operation points into a test action command and transmitting the test action command to the actuator. And after receiving the test action command, the actuator executes corresponding operation on the sample. The transformation submodule may reside in the host computer. The conversion submodule generates the test action command through the conversion unit, the first generation unit, the second generation unit and the third generation unit.
Preferably, as shown in fig. 6, the playback module further includes:
a second transmission submodule: the test action command is transmitted to the executor;
a second acquisition submodule: the system comprises a test system, a test device and a control device, wherein the test system is used for acquiring a test result of a test action instruction received by an executor executing the same type of touch screen type product to be tested;
a comparison submodule: comparing the test result acquired by the second acquisition submodule with the standard test result, and if the test result is consistent with the standard test result, judging that the test comparison result is passed; if not, the comparison result of the test is failed.
The second transmission submodule is positioned in the upper computer and transmits the test action command to the actuator. And after the executor receives the test action command, executing the test action command on the same type of touch screen type product to be tested. The second acquisition sub-module may be an image acquisition device. And the second acquisition sub-module samples the test result of the actuator after executing the test action instruction. The comparison submodule is positioned in the upper computer and compares the sampled test result with the standard test result to obtain a comparison result. The playback module realizes the playback of the test action instruction through the transmission submodule, the second sampling submodule and the comparison submodule. And the performance of the same type of touch screen type product to be tested is tested through the playback module. The data output end of the transmission sub-module is connected with the data input end of the actuator, and the data output end of the second acquisition sub-module is connected with the data input end of the comparison sub-module.
In this embodiment, recording and playback may be performed for one test action. In other embodiments, multiple test actions may be combined into one test case, and the recording and playback method of the present invention may be performed on each test case.
In this embodiment, the upper computer is used to realize the verification function of converting the sequence of the positions and time of a group of operation points into a test action command and a result image, and in other embodiments, the functions can be realized by using equipment with a calculation function.
The present embodiment combines robotics and intelligent vision processing analysis techniques to perform test actions by machine instead of manually. The mode of manual teaching recording and playback is adopted, and any test case can be adopted regardless of tablet computers or mobile phones of different models. The actuators with more than three degrees of freedom execute the test action instead of special machine test, so that the universality is improved. For actuators with more than three degrees of freedom produced by different manufacturers or with different models, the functions can be reproduced and transplanted only by importing corresponding actuator parameters. The test program can be automatically generated, and the compiling of the test program and the script by professional personnel is not needed, so that the technical threshold is reduced.
In other embodiments, the transformation submodule may also be included in the touch screen teaching terminal. It is understood that any combination of functional blocks of the system, with no changes in connection or function, is encompassed by the spirit and scope of the present invention.
Claims (9)
1. A method for recording and playing back human-computer interface test actions of a touch screen type product is characterized by comprising the following steps:
s10 recording the man-machine interface test action;
s20 plays back the human-machine interface test action recorded in S10.
2. The method for recording and playing back human machine interface test actions of touch screen products as claimed in claim 1, wherein said S10 further comprises:
s101) taking a qualified touch screen type product as a prototype, collecting an image of a man-machine interface of the prototype, transmitting the image to a touch screen teaching terminal which is provided with a touch screen and can record the position and time sequence of an operation point in real time, and displaying the man-machine interface image in real time by the touch screen teaching terminal;
s102) saving a teaching action finished on the touch screen teaching terminal as a sequence of positions and time of a group of operation points;
s103) converting the sequence of the positions and the time of the group of operation points into a test action command, and transmitting the test action command to an actuator for execution;
s104) acquiring an execution result image of the actuator on the prototype, and taking the execution result image as a standard test result image;
s105) storing the standard test result image in the S104) and a corresponding test action instruction;
s106) returns to step S102 until the recording of all teaching actions for testing is completed.
3. The method for recording and playing back human-machine interface test actions of touch screen products according to claim 2, wherein in S103), the specific process of converting the position and time sequence of the group of operation points into a test action command is as follows:
and converting the position sequence of the group of operation points from a touch screen coordinate system of a touch screen teaching terminal to an actuator end pose sequence under the actuator coordinate system through an image coordinate system, performing kinematic inversion on the actuator end pose sequence to generate a joint position of the actuator, generating an interpolation track, namely a motion track of the actuator, from the joint position and a corresponding time sequence, and converting the motion track of the actuator into a corresponding actuator execution instruction to form a test action instruction.
4. The method for recording and playing back human machine interface test actions of touch screen products as claimed in claim 1, wherein said S20 further comprises:
s201) transmitting the nth test action command to the actuator for execution, where n is 1+ q, q is the number of times that step S204) returns to step S201), and the initial value of q is 0;
s202) acquiring an execution result image of an actuator on a touch screen type product of the same type as a prototype;
s203) comparing the acquired test result image with the standard test result image, if the acquired test result image is consistent with the standard test result image, the test comparison result is passed, and the step S204) is executed; if not, the comparison result of the test is failed, and the test is finished or the step S204 is entered);
s204) returns to step S201) until all test action instructions have been executed.
5. The method for recording and playing back human-machine interface test actions of touch-screen products according to claim 4, further comprising S205): replacing the products to be tested with the same model as the prototype, and returning to S201) until the test of all the products to be tested in the same batch is completed.
6. A system for recording and playing back human machine interface test actions of a touch screen product, the system comprising:
a recording module: the device is used for recording the human-computer interface test action;
a playback module: the device is used for playing back the human-computer interface test action recorded by the recording module;
an actuator: and the testing device is used for executing the testing action on the man-machine interface of the touch screen type product according to the acquired testing action instruction.
7. The system for touch screen product human machine interface test action recording and playback as recited in claim 6, wherein said recording module further comprises:
a first transmission submodule: the touch screen teaching terminal is used for transmitting the human-computer interface image displayed by the human-computer interface of the prototype to the touch screen teaching terminal which is provided with a touch screen and can record the position and the time sequence of an operation point;
a recording submodule: the teaching terminal is used for recording teaching actions finished on the touch screen teaching terminal and storing the teaching actions as a sequence of positions and time of a group of operation points;
a transformation submodule: the sequence of the positions and the time of the group of operation points is converted into a test action command, and the test action command is transmitted to the actuator;
a first acquisition submodule: the system comprises a test system, a test device and a control device, wherein the test system is used for acquiring a result image of a test action instruction executed by an actuator on a prototype human-computer interface and taking the result image as a standard test result image;
saving the submodule: and the standard test result image and the corresponding test action instruction are stored.
8. The touch-screen product human machine interface test action recording and playback system of claim 7, wherein said transformation submodule further comprises:
a conversion unit: the position sequence of the group of operation points is converted into an actuator terminal pose sequence under an actuator coordinate system through a touch screen coordinate system of a touch screen teaching terminal through an image coordinate system;
a first generation unit: the system is used for performing kinematic inversion on the end pose sequence of the actuator to generate the joint position of the actuator;
a second generation unit: the system is used for generating an interpolation track, namely a motion track of an actuator, according to the joint position and the corresponding time sequence;
a third generation unit: and the test motion command is generated by converting the motion trail of the actuator.
9. The touch-screen product human machine interface test action recording and playback system according to claim 6, wherein said playback module further comprises:
a second transmission submodule: the test action command is transmitted to the executor;
a second acquisition submodule: the test device is used for acquiring a test result of the executor executing the received test action instruction on the product to be tested with the same model as the prototype;
a comparison submodule: comparing the test result acquired by the second acquisition submodule with the standard test result image, and if the test result is consistent with the standard test result image, judging that the test comparison result is passed; if not, the comparison result of the test is failed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610114356.7A CN105653460A (en) | 2016-02-29 | 2016-02-29 | Method and system for recording and playback of man-machine interface testing actions of touch screen product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610114356.7A CN105653460A (en) | 2016-02-29 | 2016-02-29 | Method and system for recording and playback of man-machine interface testing actions of touch screen product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105653460A true CN105653460A (en) | 2016-06-08 |
Family
ID=56491946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610114356.7A Pending CN105653460A (en) | 2016-02-29 | 2016-02-29 | Method and system for recording and playback of man-machine interface testing actions of touch screen product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105653460A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107491391A (en) * | 2017-08-21 | 2017-12-19 | 深圳市天视通电子科技有限公司 | A kind of software testing system and method |
| WO2018099246A1 (en) * | 2016-11-30 | 2018-06-07 | 中兴通讯股份有限公司 | Touch screen debugging detection method and apparatus, and computer storage medium thereof |
| WO2019056333A1 (en) * | 2017-09-25 | 2019-03-28 | 深圳传音制造有限公司 | Screen recording method for mobile terminal testing and mobile terminal |
| CN110018952A (en) * | 2018-01-10 | 2019-07-16 | 武汉斗鱼网络科技有限公司 | A kind of test method, storage medium, equipment and the system of JS code |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100125834A1 (en) * | 2008-11-19 | 2010-05-20 | Sap Ag | Dynamic Tracing on Java Exceptions |
| CN103678105A (en) * | 2012-09-17 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | Page code testing method and device |
| CN103853658A (en) * | 2012-12-04 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Automated testing method for production and playback in Android equipment and Android equipment |
| CN104742127A (en) * | 2015-04-08 | 2015-07-01 | 深圳市山龙科技有限公司 | Robot control method and robot |
| CN105138436A (en) * | 2015-08-07 | 2015-12-09 | 天津市康凯特软件科技有限公司 | Terminal offline test method and apparatus |
-
2016
- 2016-02-29 CN CN201610114356.7A patent/CN105653460A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100125834A1 (en) * | 2008-11-19 | 2010-05-20 | Sap Ag | Dynamic Tracing on Java Exceptions |
| CN103678105A (en) * | 2012-09-17 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | Page code testing method and device |
| CN103853658A (en) * | 2012-12-04 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Automated testing method for production and playback in Android equipment and Android equipment |
| CN104742127A (en) * | 2015-04-08 | 2015-07-01 | 深圳市山龙科技有限公司 | Robot control method and robot |
| CN105138436A (en) * | 2015-08-07 | 2015-12-09 | 天津市康凯特软件科技有限公司 | Terminal offline test method and apparatus |
Non-Patent Citations (1)
| Title |
|---|
| 牛希望: "基于触摸示教与视觉引导的机器人控制方法研究", 《工程控制计算机》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018099246A1 (en) * | 2016-11-30 | 2018-06-07 | 中兴通讯股份有限公司 | Touch screen debugging detection method and apparatus, and computer storage medium thereof |
| CN107491391A (en) * | 2017-08-21 | 2017-12-19 | 深圳市天视通电子科技有限公司 | A kind of software testing system and method |
| WO2019056333A1 (en) * | 2017-09-25 | 2019-03-28 | 深圳传音制造有限公司 | Screen recording method for mobile terminal testing and mobile terminal |
| CN110018952A (en) * | 2018-01-10 | 2019-07-16 | 武汉斗鱼网络科技有限公司 | A kind of test method, storage medium, equipment and the system of JS code |
| CN110018952B (en) * | 2018-01-10 | 2022-12-27 | 武汉斗鱼网络科技有限公司 | JS code testing method, JS code storage medium, JS code testing equipment and JS code testing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107932504B (en) | PyQt-based mechanical arm operation control system | |
| CN107428004B (en) | Automatic collection and tagging of object data | |
| US10751877B2 (en) | Industrial robot training using mixed reality | |
| CN105653460A (en) | Method and system for recording and playback of man-machine interface testing actions of touch screen product | |
| GB2599831A (en) | Predictive virtual training systems, apparatuses, interfaces, and methods for implementing same | |
| JP6640060B2 (en) | Robot system | |
| She et al. | Teaching robots new actions through natural language instructions | |
| CN104679017A (en) | Automatic test device and test method for mobile terminal application program | |
| CN109079794B (en) | Robot control and teaching method based on human body posture following | |
| CN109732593B (en) | Remote control method and device for robot and terminal equipment | |
| CN107395844A (en) | Test equipment, method, system, medium, device and computing device | |
| CN201576207U (en) | Monitoring and motion simulating system for automatic stamping production line | |
| CN106909247B (en) | Touch simulation system, method and device and touch terminal test equipment and method | |
| CN114260893B (en) | Digital twin model construction method for industrial robot assembly pick-and-place process | |
| CN108132724A (en) | A kind of touch screen debugs detection method and device | |
| CN114474106A (en) | Method for controlling a robot device and robot control device | |
| Skoglund et al. | Programming by demonstration of pick-and-place tasks for industrial manipulators using task primitives | |
| WO2015126392A1 (en) | Emulating a user performing spatial gestures | |
| CN115741678A (en) | Robot motion simulation method and simulation system | |
| KR100956839B1 (en) | Jointed arm-robot simulation control program development tool | |
| CN116664355A (en) | Virtual teaching system and method based on remote experiment | |
| Sanches et al. | Scalable. Intuitive Human to Robot Skill Transfer with Wearable Human Machine Interfaces: On Complex, Dexterous Tasks | |
| Su et al. | A ROS based open source simulation environment for robotics beginners | |
| CN117140517A (en) | High-precision automatic hand-eye calibration method and system for mechanical arm | |
| Kang et al. | Automatic hand-over animation for free-hand motions from low resolution input |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160608 |