CN115454264A - Automatic track clicking and pressing device and method based on target detection - Google Patents
Automatic track clicking and pressing device and method based on target detection Download PDFInfo
- Publication number
- CN115454264A CN115454264A CN202211048826.6A CN202211048826A CN115454264A CN 115454264 A CN115454264 A CN 115454264A CN 202211048826 A CN202211048826 A CN 202211048826A CN 115454264 A CN115454264 A CN 115454264A
- Authority
- CN
- China
- Prior art keywords
- click
- key
- guide rail
- clicked
- pressing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Control Of Position Or Direction (AREA)
Abstract
The invention provides a track automatic click pressing device and method based on target detection, which relate to the technical field of automatic control and comprise the following steps: the device comprises a working panel, at least one longitudinal guide rail fixedly arranged with the working panel, and at least one transverse guide rail connected with the longitudinal guide rail in a sliding manner, wherein a click pressing device is connected on the transverse guide rail in a sliding manner; the longitudinal guide rail, the transverse guide rail and the click presser are all connected with stepping motors, and all the stepping motors are connected with a click pressure control calculation controller; the shooting and photographing device is also arranged above the working panel, and the click pressure control calculation controller and the shooting and photographing device are connected with the control device; click-press control software and target detection software are installed in the control equipment. The invention realizes the arbitrary movement of the click presser in the horizontal and vertical directions, realizes the continuous click pressing of single point or multiple points of arbitrary key equipment or devices, realizes the arbitrary placement of the equipment needing click pressing on the working panel through target detection, and improves the intelligent automation level.
Description
Technical Field
The invention relates to the technical field of automatic control, in particular to a track automatic clicking and pressing device and method based on target detection.
Background
Currently, automatic click-and-press has been applied in more and more automation scenarios during intelligent application landing. The automatic click and press mainly has two modes, one mode is a method for clicking and pressing buttons, texts, links and the like on a system, application or software and the like in a mode of simulating manual operation through a software program; one method is a single-point fixed position physical click pressing device (multiple physical devices are needed for multiple points, and multiple positions are fixed), and a method for controlling click pressing of a physical key is triggered and controlled through a software program.
The software program simulation manual clicking and pressing has a large number of applications in the fields of RPA process robots, software UI automatic testing and the like. The single-point fixed position physical click pressing device realizes physical click pressing under the condition that click pressing cannot be simulated through a software program or hardware click pressing is simulated by the software program and is shielded by a system, for example, in the application of an online bank RPA robot, automatic clicking and pressing of a U shield confirmation key, an OK key and the like can be involved, the position of the U shield confirmation key or the OK key is fixed, and automatic physical click pressing is realized through calling and triggering of the software program.
According to the existing method, device or product for physical click pressing based on single-point fixed position, before calling, triggering physical click and pressing, the click pressing position on the fixed device needs to be adjusted, the single-arm pressing contact needs to be aligned with a physical button to be clicked and pressed, the single arm (pressing arm or clicking arm) and the contact thereof are driven by the driving motor of the calculation control unit to move downwards, click and press or move upwards and bounce, and finally click and press on the button is achieved. The method is mainly applied to relevant scenes of automatic click pressing of keys on a keyboard and a U shield, and when the following scenes of multi-point click pressing are met:
keys such as mechanical keyboards require multiple key stroke presses, such as entering multiple characters on the keyboard.
And clicking numbers on a U-shield password keypad and other similar equipment to input the digital password.
Partial confirmation and key pressing of keys such as an OK key on the side face of equipment such as a U shield.
Click and press of keys of any other common equipment (mobile phone, remote controller, switch and the like).
For the above scenario, for the device with large area of the plane where the keys are located and large distance between the keys, when multi-point clicking is realized, each key needs to be configured with a corresponding device. There are problems in that: after each device is fixed in position, the device cannot be moved when in use, each device needs independent power supply, an independent trigger calling interface needs to be provided, a plurality of hardware interfaces are needed, the device cannot be reused and only can be special, the application range is narrow, the universality is not high, the flexibility and the expansibility are poor, and the utilization rate is low.
For equipment with small plane area of keys and small distance between keys, the existing method, device or product for physically clicking and pressing at a fixed position by a single point has the following problems: the problem that the limited key plane area and the key pitch are small is that multi-point clicking and pressing cannot be achieved through the number of stacked devices or products. For example, the function of automatically inputting the numerical code is realized by automatically clicking numbers on a U-shield password keypad. The device with the keys on the side cannot be pressed, for example, the confirmation key of a part of the U shield is pressed on the side of the U shield.
For clicking and pressing keys of any common general key and other equipment (mobile phones, remote controllers, switches and the like), the conventional method, device or product has no general solution and is limited by the influence of comprehensive factors such as the type, material, shape, height, size, key spacing, key position and the like of the key equipment, and one scheme or equipment cannot realize single-point automatic pressing and multi-point automatic continuous clicking and pressing.
Disclosure of Invention
Aiming at the problems, the invention provides a track automatic clicking pressing device and method based on target detection, which can support any key equipment or device to perform automatic single-point or multi-point continuous clicking pressing and long pressing under the scene of unfixed position.
In order to achieve the above object, the present invention provides an automatic track clicking and pressing device based on target detection, comprising: the device comprises a working panel and at least one longitudinal guide rail fixedly mounted with the working panel, wherein at least one transverse guide rail is arranged above the working panel and is in sliding connection with the longitudinal guide rail;
the longitudinal guide rail, the transverse guide rail and the click presser are respectively connected with a stepping motor, and all the stepping motors are connected with a click pressure control calculation controller;
the click pressure control calculation controller is connected with the control equipment;
the camera shooting and photographing device is also arranged above the working panel and is connected with the control device;
click-press control software and target detection software are installed in the control equipment.
The shooting and photographing equipment shoots the working panel and the equipment needing to be clicked and pressed placed on the working panel, and key coordinates of all keys in the equipment needing to be clicked and pressed are obtained through target detection software; the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment according to the received keys to be clicked, generates a mobile control instruction and sends the click control press calculation controller; and the click control and pressure calculation controller controls the stepping motors of the longitudinal guide rail and the transverse guide rail to sequentially move to the positions of the keys to be clicked based on the key coordinates of the keys to be clicked, and controls the stepping motors of the click depressors to realize click and pressure on the keys to be clicked.
As a further improvement of the present invention, the longitudinal guide rails include two longitudinal guide rails respectively installed at two ends of the working panel, and two ends of the transverse guide rail are respectively installed on the two longitudinal guide rails.
As a further improvement of the invention, the end part of the transverse guide rail is provided with a sliding roller, and the transverse guide rail is connected with the longitudinal guide rail in a sliding way through the sliding roller.
As a further improvement of the invention, the image pickup and photographing device comprises one or more than one image, the image of the press device needing clicking is currently on the working panel, and the size of the image is the same as that of the working panel.
As a further improvement of the present invention, the click presser and the stepping motor connected thereto are connected to the transverse guide rail together through a sliding roller.
The invention also provides a method for automatically clicking the pressing device on the track based on target detection, which comprises the following steps:
acquiring key coordinates of all keys on the equipment needing to be clicked through camera shooting and photographing equipment;
the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment according to the received keys to be clicked, generates a mobile control instruction and sends a click control and press calculation controller;
the click pressure control calculation controller controls each stepping motor connected with the transverse guide rail and the longitudinal guide rail to rotate according to the movement control instruction so as to drive the click presser to move in the transverse and longitudinal directions until the key coordinate position of the key to be clicked is reached;
the click pressure control calculation controller controls the stepping motor connected with the click presser to rotate, and clicks and presses the key to be clicked;
after clicking and pressing are finished, the clicking pressure control calculation controller controls each stepping motor connected with the transverse guide rail and the longitudinal guide rail to rotate, and drives the clicking presser to move in the transverse and longitudinal directions until the clicking presser resets or reaches the key coordinate position of the next key to be clicked.
As a further improvement of the invention, the key coordinates of each key on the equipment needing to be clicked and pressed are obtained through the camera shooting and photographing equipment; the method comprises the following steps:
placing pressing equipment needing to be clicked at any position on the working panel;
initializing the initial position of the click presser to be a coordinate origin;
adjusting the height of a camera on the camera shooting and photographing equipment, and shooting a picture of the pressing equipment needing to be clicked on the working panel, wherein the size of the shot picture is the same as that of the working panel;
and identifying and acquiring the key coordinates of each key in the picture through the target detection software.
As a further improvement of the present invention, the movement control instruction includes a click-press movement instruction, key coordinate X-axis data, key coordinate Y-axis data, a click-press instruction, and a click-press speed; the click pressure control calculation controller controls the stepping motors connected with the transverse tracks and the longitudinal tracks to rotate according to the click pressing movement instruction, the key coordinate X-axis data and the key coordinate Y-axis data, and controls the rotating speed of the stepping motors connected with the click presser according to the click pressing instruction and the click pressing speed.
As a further improvement of the present invention, when multi-point continuous click pressing is performed, the click pressing control software splices the movement control instructions one by one according to the coordinate of the plurality of keys to be clicked and the coordinate coefficient according to the ordered list, and sends the movement control instructions to the click pressure control calculation controller one by one.
As a further improvement of the present invention, after the click and the pressure are finished, a reset coordinate is obtained according to a key coordinate negation of the last key, and the click and pressure control calculation controller controls the rotation of each stepping motor connected to the transverse rail and the longitudinal rail according to the reset coordinate, so that the motor presser moves to an initial position.
Compared with the prior art, the invention has the beneficial effects that:
the multi-track mode is adopted to realize the arbitrary movement of the clicking presser in the horizontal and longitudinal directions, and the mode of controlling the clicking presser by the stepping motor is adopted, so that equipment needing to be clicked is not limited by types, materials, shapes, sizes, heights, key intervals, key positions and the like any more, and the multi-track type multi-point continuous clicking presser can be applied to common general key equipment such as keyboards, U shields, remote controllers, mobile phones, switches and the like, and realizes the automatic single-point or multi-point continuous clicking pressing of arbitrary key equipment or devices; furthermore, the camera shooting and photographing equipment is matched with the target detection software, so that the automatic identification of the coordinates of the keys on the equipment needing to be clicked is realized, the positions of the keys on the working panel needing to be clicked and pressed are not required to be fixed any more, the keys can be placed at will and moved at will, the flexibility is improved, and the intelligent automation level is improved.
The invention adopts a coordinate mode, and can realize accurate click pressing under the condition of fixed equipment position.
The invention adopts a mode of combining multi-track and target detection key coordinates, and can accurately click and press each case and realize the continuous click and pressing functions of a plurality of keys under the scene that the area of equipment to be clicked and pressed and the distance between keys are narrow (such as automatically clicking and inputting a password on a U shield).
According to the invention, when the click presser performs multi-point continuous click pressing, the click presser is controlled to move from the initial position to the first key coordinate and the next key coordinate in sequence, so that the moving distance of the click presser is saved, and the pressing efficiency is improved.
The invention can set the transverse guide rail and the longitudinal guide rail with corresponding lengths according to the size of the area of the key of the equipment needing to be clicked and pressed, and can also adjust the lengths of the transverse guide rail and the longitudinal guide rail according to the areas of the keys of a plurality of equipment needing to be clicked and pressed, so that the click pressing device can serve more equipment needing to be clicked and has stronger expansibility and flexibility.
Drawings
Fig. 1 is a schematic view of an automatic track clicking and pressing device based on target detection according to an embodiment of the disclosure;
fig. 2 is a flowchart of an automatic track click-and-press method based on target detection according to an embodiment of the present invention.
Description of the reference numerals:
1. a power line socket; 2. a USB-to-serial interface; 3. a guide rail holder; 41. a longitudinal guide rail; 42. a transverse guide rail; 5. a stepping motor; 6. clicking and pressing the contact; 7. a working panel; 8. clicking a pressure control calculation controller; 9. a sliding roller; 10. a control end stepping motor control line interface; 11. a wire arranging hole; 12. clicking a press controller; 13. a motor end stepping motor control line interface; 14. shooting and photographing equipment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The invention is described in further detail below with reference to the attached drawing figures:
as shown in fig. 1, the present invention provides an automatic track clicking and pressing device based on target detection, which includes: the working panel 7 is fixedly arranged on at least one longitudinal guide rail 41 on the longitudinal edge of the working panel 7 through the sliding guide rail fixer 3, the end part of the working panel is connected with at least one transverse guide rail 42 of the longitudinal guide rail 41 in a sliding way, and the working panel 7 can bear the contraction and expansion of the number of key equipment by changing the length of the longitudinal guide rail 41 and the length of the transverse guide rail 42 and clicking the area of the working panel 7;
the transverse guide rail 42 is connected with a click pressing device in a sliding mode, the click pressing device comprises a click pressing controller 12 and a click pressing contact 6, a click pressure control calculation controller 8 is fixedly installed on the lower side of the working panel 7, the longitudinal guide rail 41, the transverse guide rail 42 and the click pressing controller 12 are connected with stepping motors 5 respectively, and all the stepping motors 5 are connected with the click pressure control calculation controller 8;
the click pressure control calculation controller 8 is connected to a control device (e.g., a computer) in which click press control software is installed, and the click press control software sends an instruction to the click pressure control calculation controller 8 in a manner of operating a serial device.
And a camera shooting and photographing device 14 is further installed above the working panel 7, the camera shooting and photographing device is connected with the control device, and target detection software is installed in the control device.
A device needing to be clicked and pressed is placed on the working panel 7, the shooting and photographing device 14 shoots the working panel 7 and the device needing to be clicked and pressed placed on the working panel, and the key coordinates of all keys in the device needing to be clicked and pressed are obtained through target detection software; the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment 14 according to the received keys to be clicked, generates a movement control instruction and sends a click control and press calculation controller 8; the click pressure control calculation controller 8 controls the stepping motors 5 of the longitudinal guide rail 41 and the transverse guide rail 42 to move to the positions of the keys to be clicked in sequence based on the coordinates of the keys to be clicked, and controls the stepping motor 5 of the click presser to click and press the keys to be clicked.
Further, in the above-mentioned case,
the working panel 7 is provided with a coordinate system with an accuracy of at least millimeter;
the end part of the transverse guide rail 42 is provided with a sliding roller 9, and the transverse guide rail 42 is connected with the longitudinal guide rail 41 in a sliding way through the sliding roller 9;
the click presser and the stepping motor 5 connected with the click presser are connected on the transverse guide rail 42 in a sliding way through a sliding roller 9 arranged at the common bottom;
the image pickup and photographing device 12 can adopt one or more than one, the pixel is at least 200 ten thousand pixels, and when the current picture on the working panel 7 of the press device needing to be clicked is photographed, the size of the photographed picture is the same as that of the working panel 7.
Preferably, the first and second liquid crystal materials are,
the longitudinal guide rail 41 comprises two rails, so that the device is more stable; two longitudinal guide rails 41 are respectively installed at both ends of the working panel, and both ends of the transverse guide rail 42 are respectively installed on the two longitudinal guide rails 41.
In the concrete connection structure, the connecting structure is provided with a plurality of connecting rods,
the working panel 7 is provided with a power line socket 1 which is connected with a power supply through a 12V/5A power line to supply power for the click voltage control calculation controller 8 and the stepping motor 5; the working panel 7 is also provided with a USB (universal serial bus) to serial port interface 2, is connected with a control device through a USB to serial port line, is provided with a plurality of control end stepping motor control line interfaces 10 which are respectively used for connecting one ends of a plurality of motor control lines, and the other ends of the plurality of motor control lines are respectively connected with a plurality of motor end stepping motor control line interfaces 13 of the stepping motors 5; the camera shooting device 12 is connected to the control device through a USB connecting wire and is fixedly installed above the working panel 7, and the installation position is adjustable.
For convenient connection, the edge of the working panel 7 is provided with a wire arranging hole 11, and all motor control wires pass through the wire arranging hole 11 to be connected with the stepping motors 5 on the upper side and the lower side of the working panel 7 and the click pressure control calculation controller 8.
As shown in fig. 2, the present invention further provides a track automatic click pressing method based on target detection, including:
acquiring key coordinates of all keys on the equipment needing to be clicked through the camera shooting and photographing equipment 14;
the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment 14 according to the received keys to be clicked, generates a movement control instruction and sends a click control and press calculation controller 8;
the click pressure control calculation controller 8 controls each stepping motor 5 connected with the transverse guide rail 42 and the longitudinal guide rail 41 to rotate according to the movement control instruction so as to drive the click presser to move in the transverse and longitudinal directions until the key coordinate position of the key to be clicked is reached;
the click pressure control calculation controller 8 controls the stepping motor 5 connected with the click press controller 12 to rotate, and clicks and presses the key needing to be clicked;
after the click pressing is finished, the click pressure control calculation controller 8 controls the stepping motors 5 connected with the transverse guide rail 42 and the longitudinal guide rail 41 to rotate, and drives the click presser to move in the transverse and longitudinal directions until the click presser resets or reaches the key coordinate position of the next key to be clicked.
The key coordinates of each key on the equipment needing to be clicked and pressed are obtained through the camera shooting and photographing equipment 14; the method comprises the following steps:
a pressing device needing to be clicked is placed (does not need to be fixed) at any position on the working panel 7;
initializing the initial position of the click presser to be a coordinate origin;
adjusting the height of a camera on the camera shooting and photographing device 14, shooting a picture of the pressing device needing to be clicked on the working panel 7 at present, wherein the size of the shot picture is the same as that of the working panel 7;
and identifying and acquiring the key coordinates of each key in the picture through target detection software.
Further, in the above-mentioned case,
the movement control instruction comprises a click pressing movement instruction, key coordinate X-axis data, key coordinate Y-axis data, a click pressing instruction and a click pressing speed; the click pressure control calculation controller 8 controls the stepping motors 5 connected with the transverse track and the longitudinal track to rotate according to the click pressing movement instruction, the key coordinate X-axis data and the key coordinate Y-axis data, and controls the rotating speed of the stepping motors 5 connected with the click presser according to the click pressing instruction and the click pressing speed.
Adjusting the height of a camera on the camera shooting and photographing device 14, and shooting a picture of the current working panel 7 of the press device needing to be clicked, wherein the size of the shot picture is the same as that of the working panel 7; identifying and acquiring key coordinates of each key in the picture through target detection software; the target detection software uses a target detection model, the target detection model is trained by adopting a convolution cyclic neural network, and the training process comprises the following steps:
s1, acquiring and storing pictures of button equipment such as a U shield and the like in such scenes by adopting a camera shooting and photographing device 1414;
s2, manually marking position coordinates of the positions of keys, buttons and the like in the picture, which are transversely and longitudinally furthest apart by a marking tool (for example, a and c are respectively two points transversely furthest apart, and b and d are respectively two points longitudinally furthest apart) as training data;
s3, inputting training data into a convolution cyclic neural network training target detection model;
finally, the target detection software processes the target detection result obtained by the target detection model;
{"transcription":"xxx","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
X＝(xc-xa)/2，Y＝(yd-yb)/2
in a still further aspect of the present invention,
when multi-point continuous click pressing is carried out, the click pressing control software splices the mobile control instructions one by one according to the coordinate of a plurality of keys needing to be clicked and the coordinate coefficient according to the ordered list and sends the mobile control instructions to the click pressure control calculation controller 8 one by one.
After the click and the press are finished, the reset coordinates are obtained according to the key coordinate negation of the last key, and the click pressure control calculation controller 8 controls each stepping motor 5 connected with the transverse rail and the longitudinal rail to rotate according to the reset coordinates, so that the motor presser moves to the initial position.
The specific operation process is as follows:
firstly, inserting one end of a power line into a power port of a click press working panel 7, and inserting one end of the power line into a power supply end;
secondly, inserting a serial port end of the USB serial port line into a USB serial port interface 2 on the panel, and accessing the other end of the USB serial port line into a USB port of the control equipment;
thirdly, fixing a pressing device needing clicking on the working panel 7;
fourthly, adjusting the height of the click presser, and initializing the initial position of the click presser to be the origin of a coordinate system;
fifthly, adjusting the height of the equipment needing to be clicked and pressed (taking the height of the highest equipment as a reference, and heightening other equipment to the reference height by adding a gasket);
sixthly, adjusting the height of a camera in the camera shooting and photographing equipment 12 to enable the size of a shot picture to be the same as that of the working panel 7, namely the upper left corner of the image is the origin of coordinates, and the error is delta;
seventhly, acquiring the key coordinates of each key in the equipment needing to be clicked and pressed through target detection software;
eighthly, clicking the press control software to read or receive the key coordinate data;
ninth, the click press control software sends the click press movement instruction, coordinate X-axis data, coordinate Y-axis data, click press instruction, press speed and other parameters to the click press control calculation controller through a USB (universal serial bus) conversion serial port line;
tenth, all the key coordinate coefficient data are referenced to an origin (0, 0), if the continuous multi-point click is pressed, the coordinate of the first key click press is (x 1, y 1), the coordinate of the second key click press is (x 2, y 2), the first key click press needs to move from the origin to the target coordinate, the second key click press does not need to start from the origin, the distance of the position movement is (x 2-x1, y2-y 1) from the first coordinate position, and so on, so the position of the movement when the continuous multi-point click is (x + Δ x, y + Δ y), x and y are respectively the coordinate data of the last key click press, and Δ x = (x 2-x 1) ± δ x, Δ y = (y 2-y 1) ± δ y. (ii) a
Step ten, when the click press control software receives a plurality of key coordinate data, splicing the mobile control instructions one by one according to a coordinate coefficient data ordered list, and sending the mobile control instructions to the click press control calculation controller 8 one by one;
step ten, after the click-control press calculation controller receives the movement instruction, the stepping motor 5 of the transverse and longitudinal guide rail rotates and moves in the transverse and longitudinal directions of the guide rail according to the target coordinate position;
step thirteen, after the target coordinate position is reached, the click pressure control calculation controller 8 drives a stepping motor 5 connected with a click press controller 12 to rotate according to the received click press instruction and the click press speed, and drives a click press contact 6 to perform click press;
fourteenth, after the continuous multi-point clicking is finished, the control software can take the negative according to the coordinate data of the last key and send a movement control instruction (reset instruction) to enable the clicking presser to move and return to the origin of coordinates;
and fifteenth step, if the single-point click pressing is carried out, the control software moves according to the click key coordinate data and the moving instruction, then click pressing operation is carried out according to the click pressing instruction, and finally the click presser returns to the origin of coordinates according to the reset instruction.
The embodiment is as follows:
the invention is applied to the online bank flow downloading robot of the traffic bank, and the scene is as follows: when the robot automatically logs in the online banking system of the transportation bank, after an account number is automatically input, a password is required to be automatically input, a key on a password numeric keypad is automatically clicked on a U shield when the password is input, and an OK key is required to be clicked to confirm after the password is input. Assume that the transit bank has a password 246879 for the online bank account. The click control software is installed in the computer equipment (the system can be Windows or Linux, and the internet bank only supports Windows at present, so the example takes Windows10 as an example), and the multi-track click pressing equipment device is connected to the computer through the USB port after the power supply of the multi-track click pressing equipment device is switched on based on the target detection. The internet bank U shield of the transportation bank is already connected to the same computer device through USB, and the camera shooting device 12 is also already connected to the same computer device.
(1) Acquiring key coordinates of a U shield of a traffic bank through target detection software:
{"transcription":"2","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"4","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"6","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"8","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"7","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"9","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
{"transcription":"OK","points":[[xa,ya],[xb,yb],[xc,yc],[xd,yd]]}
processing each key correspondingly according to a formula to obtain key coordinates; the formula is as follows:
X＝(xc-xa)/2，Y＝(yd-yb)/2
(2) The password of the transportation bank online bank account is 246879, and the coordinate data of the OK confirmation key are configured as follows:
moving instructions: { MOV } { Δ X } { Δ Y } { SPEED }// movement command, movement distance in the X-axis direction and movement SPEED in the Y-axis direction; wherein, Δ x = (x 2-x 1) ± δ x, Δ y = (y 2-y 1) ± δ y, δ is the error of the target detection software detecting the same position relative to the coordinate system of the working panel
A pressing instruction: { PRESS } { SPEED }// PRESS command PRESS SPEED
The execution steps are as follows:
1. automatically inputting an online bank account of the traffic bank, and automatically switching to an online bank account password input box;
2. according to the error (delta is the relative origin position of the working panel, delta is the error of the same position detected by the target detection software relative to the working panel coordinate system, delta x =0 +/-delta x, and delta y =0 +/-delta y) of the corresponding coordinate of the number 2 on the E-bank shield of the transportation bank and the working panel coordinate system, the E-bank shield moves to a target position by triggering a moving instruction, and a click pressing instruction is called for clicking;
3. calculating a moving distance of the number 4, Δ x = (125-120) ± δ x, Δ y = (84-80) ± δ y; then in turn invoke move and press instructions// and so on, including a click confirmation of the OK key.
4. Calling a click pressing homing command to return the click presser to the origin of coordinates;
5. and automatically clicking a login button, and automatically logging in the online bank account of the transportation bank successfully.
Wherein, the steps 1 and 5 are independent of the device (mainly by means of RPA technology), and the steps 2, 3 and 4 are the specific steps realized by the device and the method of the invention.
The invention has the advantages that:
(1) The multi-track mode is adopted to realize the arbitrary movement of the click presser in the horizontal and vertical directions, and the mode of controlling the click presser by the stepping motor is adopted, so that the equipment needing to be clicked is not limited by types, materials, shapes, sizes, heights, key intervals, key positions and the like any more, and the multi-track click presser can be applied to common general key equipment such as keyboards, U shields, remote controllers, mobile phones, switches and the like, and realizes the automatic single-point or multi-point continuous click pressing of arbitrary key equipment or devices; furthermore, the invention realizes the automatic identification of the key coordinates on the equipment needing to be clicked and pressed by arranging the camera shooting and photographing equipment and matching with the target detection software, so that the position of the key of the equipment needing to be clicked and pressed on the click and press working panel is not required to be fixed any more, the key can be placed at will and moved at will, the flexibility is improved, and the intelligent automation level is improved.
(2) The deep learning based on the neural network is adopted, the target detection models of the detection keyboard, the U shield, the remote controller, the mobile phone, the switch and other equipment and the keys on the equipment are accurately trained, then the mode that the target detection software acquires the coordinates is adopted, the keys on the equipment at any position on the working panel can be accurately acquired, the accurate motor pressing is realized, and the comprehensive automation of the automatic track clicking pressing device is facilitated.
(3) By adopting a mode of combining multi-track and target detection, each case can be accurately clicked and pressed under the scene that the area of the equipment to be clicked and pressed and the distance between keys are narrow (for example, a password is automatically clicked and input on a U shield), and the continuous click pressing function of a plurality of keys is realized.
(4) When clicking the pressing device and carrying out the multiple spot and click in succession and press, control and click the pressing device and move to first button coordinate, next button coordinate by initial position in proper order, save the displacement who clicks the pressing device, improve and press efficiency.
(5) The transverse guide rails and the longitudinal guide rails 41 with corresponding lengths can be arranged according to the size of the area of the keys of the equipment needing to be clicked and pressed, and the lengths of the transverse guide rails and the longitudinal guide rails 41 can be adjusted according to the areas of the keys of a plurality of equipment needing to be clicked and pressed, so that the click pressing device can serve a plurality of equipment needing to be clicked and pressed simultaneously, and has expansibility and flexibility.
(6) The complexity of application is reduced, the utilization rate of equipment is improved, and only one hardware interface, one set of control software and one set of power supply interface are needed. The same working panel can support click pressing of more equipment device keys.
(7) The problem of narrow and small button panel can not the multiple spot automatic click press is solved, for example the automatic click inputs the password on the U shield.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A track automatic click pressing device based on target detection is characterized by comprising: the device comprises a working panel and at least one longitudinal guide rail fixedly mounted with the working panel, wherein at least one transverse guide rail is arranged above the working panel and is in sliding connection with the longitudinal guide rail;
the longitudinal guide rail, the transverse guide rail and the click presser are respectively connected with a stepping motor, and all the stepping motors are connected with a click pressure control calculation controller;
the click pressure control calculation controller is connected with the control equipment;
the camera shooting and photographing device is also arranged above the working panel and is connected with the control device;
click-press control software and target detection software are installed in the control equipment;
the shooting and photographing equipment shoots the working panel and the equipment to be clicked and pressed placed on the working panel, and key coordinates of all keys in the equipment to be clicked and pressed are obtained through target detection software; the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment according to the received keys to be clicked, generates a mobile control instruction and sends the click control press calculation controller; and the click control and pressure calculation controller controls the stepping motors of the longitudinal guide rail and the transverse guide rail to sequentially move to the positions of the keys to be clicked based on the key coordinates of the keys to be clicked, and controls the stepping motors of the click depressors to realize click and pressure on the keys to be clicked.
2. The target detection-based track automatic click pressing device according to claim 1, characterized in that: the two longitudinal guide rails are respectively arranged at two ends of the working panel, and two ends of the transverse guide rail are respectively arranged on the two longitudinal guide rails.
3. The target detection-based track automatic click pressing device according to claim 1 or 2, characterized in that: the end part of the transverse guide rail is provided with a sliding roller, and the transverse guide rail is connected with the longitudinal guide rail in a sliding way through the sliding roller.
4. The target detection-based track automatic click pressing device according to claim 1, characterized in that: the shooting and photographing equipment comprises one or more than one shooting equipment, the current picture on the working panel of the pressing equipment needing to be clicked is shot, and the size of the shot picture is the same as that of the working panel.
5. The object detection-based track automatic click pressing device according to claim 1, wherein: the click presser and the stepping motor connected with the click presser are connected to the transverse guide rail through a sliding roller.
6. A method for applying the automatic track clicking and pressing device based on target detection as claimed in any one of claims 1 to 5, comprising:
acquiring key coordinates of all keys on the equipment needing to be clicked through camera shooting and photographing equipment;
the click press control software acquires the key coordinates of the keys to be clicked based on the key coordinates of all the keys acquired by the camera shooting and photographing equipment according to the received keys to be clicked, generates a mobile control instruction and sends a click control and press calculation controller;
the click pressure control calculation controller controls each stepping motor connected with the transverse guide rail and the longitudinal guide rail to rotate according to the movement control instruction so as to drive the click presser to move in the transverse and longitudinal directions until the key coordinate position of the key to be clicked is reached;
the click pressure control calculation controller controls the stepping motor connected with the click presser to rotate, and clicks and presses the key to be clicked;
after clicking and pressing are finished, the clicking pressure control calculation controller controls each stepping motor connected with the transverse guide rail and the longitudinal guide rail to rotate, and drives the clicking presser to move in the transverse and longitudinal directions until the clicking presser resets or reaches the key coordinate position of a next key needing to be clicked.
7. The method of claim 6, wherein: acquiring key coordinates of each key on the equipment needing to be clicked through the camera shooting and photographing equipment; the method comprises the following steps:
placing pressing equipment needing to be clicked at any position on the working panel;
initializing the initial position of the click presser to be a coordinate origin;
adjusting the height of a camera on the camera shooting and photographing equipment, and shooting a picture of the pressing equipment needing to be clicked on the working panel, wherein the size of the shot picture is the same as that of the working panel;
and identifying and acquiring the key coordinates of each key in the picture through the target detection software.
8. The method of claim 6, wherein: the mobile control instruction comprises a click pressing mobile instruction, key coordinate X-axis data, key coordinate Y-axis data, a click pressing instruction and a click pressing speed; the click pressure control calculation controller controls the rotation of each stepping motor connected with the transverse track and the longitudinal track according to the click pressure moving instruction, the key coordinate X-axis data and the key coordinate Y-axis data, and controls the rotation speed of the stepping motor connected with the click presser according to the click pressure instruction and the click pressure speed.
9. The method of claim 6, wherein: when multi-point continuous click pressing is carried out, the click pressing control software splices the mobile control instructions one by one according to the key coordinates of a plurality of keys needing to be clicked and the coordinate coefficient according to the ordered list and sends the mobile control instructions to the click pressure control calculation controller one by one.
10. The method of claim 6, wherein: after clicking and pressing are finished, obtaining a reset coordinate according to the key coordinate negation of the last key, and controlling the rotation of each stepping motor connected with the transverse rail and the longitudinal rail by the clicking pressure control calculation controller according to the reset coordinate to enable the motor presser to move to an initial position.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211048826.6A CN115454264A (en) | 2022-08-29 | 2022-08-29 | Automatic track clicking and pressing device and method based on target detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211048826.6A CN115454264A (en) | 2022-08-29 | 2022-08-29 | Automatic track clicking and pressing device and method based on target detection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115454264A true CN115454264A (en) | 2022-12-09 |
Family
ID=84300948
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211048826.6A Pending CN115454264A (en) | 2022-08-29 | 2022-08-29 | Automatic track clicking and pressing device and method based on target detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115454264A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117111772A (en) * | 2023-10-23 | 2023-11-24 | 浙江保融科技股份有限公司 | High-speed positioning automatic key-press method and device |
-
2022
- 2022-08-29 CN CN202211048826.6A patent/CN115454264A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117111772A (en) * | 2023-10-23 | 2023-11-24 | 浙江保融科技股份有限公司 | High-speed positioning automatic key-press method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10525597B2 (en) | Robot and robot system | |
| CN100438746C (en) | Method of moving a device provided with a camera to a desired position by means of a control system, and such a system | |
| CN115454264A (en) | Automatic track clicking and pressing device and method based on target detection | |
| KR20050003402A (en) | Remotely-operated robot, and robot self position identifying method | |
| CN111604942A (en) | Object detection device, control device, and computer program for object detection | |
| CN113664835B (en) | Automatic hand-eye calibration method and system for robot | |
| CN111083376B (en) | Method, system and device for determining installation position of target object and electronic equipment | |
| CN114089735A (en) | Method and device for adjusting shelf pose of movable robot | |
| CN105259684B (en) | A kind of double-station liquid crystal display panel off-line checking system and method | |
| CN109773756A (en) | A kind of Multi-freedom-degreemanipulator manipulator and control method | |
| CN115042175A (en) | Method for adjusting tail end posture of mechanical arm of robot | |
| CN112947458B (en) | Robot accurate grabbing method based on multi-mode information and computer readable medium | |
| CN102402361A (en) | Method and device for controlling on computer based on movement track of mouse | |
| CN210723699U (en) | Pressing plate operating device and power equipment | |
| CN104822029A (en) | Method and device for controlling rotary camera to rotate and mobile terminal | |
| CN115421600A (en) | Automatic track clicking and pressing device and method based on coordinates | |
| CN108115697A (en) | The control method of service robot and service robot | |
| EP4207741A1 (en) | Image processing method and related device | |
| CN111098306A (en) | Calibration method and device of robot, robot and storage medium | |
| CN110815208A (en) | Control system, analysis device, and control method | |
| CN115401696A (en) | Data/model hybrid-driven robot remote driving method | |
| CN114610155A (en) | Gesture control method and device, display terminal and storage medium | |
| KR101332382B1 (en) | Tester for gui | |
| CN112306341A (en) | Display area moving method and device, storage medium and electronic whiteboard | |
| CN111258427A (en) | Blackboard control method and control system based on binocular camera gesture interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |