CN109871249A - A kind of remote desktop operation method, apparatus, readable storage medium storing program for executing and terminal device - Google Patents
A kind of remote desktop operation method, apparatus, readable storage medium storing program for executing and terminal device Download PDFInfo
- Publication number
- CN109871249A CN109871249A CN201910032791.9A CN201910032791A CN109871249A CN 109871249 A CN109871249 A CN 109871249A CN 201910032791 A CN201910032791 A CN 201910032791A CN 109871249 A CN109871249 A CN 109871249A
- Authority
- CN
- China
- Prior art keywords
- coordinate
- remote desktop
- subgraph
- feature vector
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention belongs to field of computer technology more particularly to a kind of remote desktop operation method, apparatus, computer readable storage medium and terminal device.The method receives the remote desktop operation request that user issues, and extracts preset template image and operational order from remote desktop operation request；The video in window of remote desktop is obtained, and is searched and the matched target subgraph of the template image in the video in window of the remote desktop；First coordinate of the target subgraph in the video in window is obtained, and obtains second coordinate of the video in window of the remote desktop in local desktop；Third coordinate of the target subgraph in the local desktop is calculated according to first coordinate and second coordinate；The operational order is executed at the third coordinate in the local desktop.Operation object is searched using the mode of images match, and determines the specific location of operation object by coordinate transform, to substantially increase operation success rate.
Description
Technical field
The invention belongs to field of computer technology more particularly to a kind of remote desktop operation method, apparatus, computer-readable
Storage medium and terminal device.
Background technique
With the development of technology, more and more enterprises start using robot process automation (Robotic Process
Automation, RPA) some routine operations are executed, still, existing RPA technology is often confined to local some
Operation object etc. does shirtsleeve operation, and faces the operation object on remote desktop, due to that can not directly acquire long-range table
The window handle of operation object on face, it is difficult to determine that the location of operation object, operation success rate are extremely low.
Summary of the invention
In view of this, the embodiment of the invention provides a kind of remote desktop operation method, apparatus, computer-readable storage medium
Matter and terminal device, to solve the problems, such as that the prior art is extremely low to the operation success rate of the operation object in remote desktop.
The first aspect of the embodiment of the present invention provides a kind of remote desktop operation method, may include:
The remote desktop operation request that user issues is received, and extracts preset mould from remote desktop operation request
Plate image and operational order；
The video in window of remote desktop is obtained, and is searched and the template image in the video in window of the remote desktop
Matched target subgraph；
First coordinate of the target subgraph in the video in window is obtained, and obtains the window of the remote desktop
Second coordinate of the image in local desktop；
The of the target subgraph in the local desktop is calculated according to first coordinate and second coordinate
Three coordinates；
The operational order is executed at the third coordinate in the local desktop.
The second aspect of the embodiment of the present invention provides a kind of remote desktop operation device, may include:
Operation requests receiving module, for receiving the remote desktop operation request that user issues, and from the remote desktop
Preset template image and operational order are extracted in operation requests；
Target image search module, for obtaining the video in window of remote desktop, and in the window figure of the remote desktop
It is searched and the matched target subgraph of the template image as in；
Coordinate obtaining module for obtaining first coordinate of the target subgraph in the video in window, and obtains
Second coordinate of the video in window of the remote desktop in local desktop；
Coordinate calculation module, for calculating the target subgraph in institute according to first coordinate and second coordinate
State the third coordinate in local desktop；
Operation executing module, for executing the operational order at the third coordinate in the local desktop.
The third aspect of the embodiment of the present invention provides a kind of computer readable storage medium, the computer-readable storage
Media storage has computer-readable instruction, and the computer-readable instruction realizes following steps when being executed by processor:
The remote desktop operation request that user issues is received, and extracts preset mould from remote desktop operation request
Plate image and operational order；
The video in window of remote desktop is obtained, and is searched and the template image in the video in window of the remote desktop
Matched target subgraph；
First coordinate of the target subgraph in the video in window is obtained, and obtains the window of the remote desktop
Second coordinate of the image in local desktop；
The of the target subgraph in the local desktop is calculated according to first coordinate and second coordinate
Three coordinates；
The operational order is executed at the third coordinate in the local desktop.
The fourth aspect of the embodiment of the present invention provides a kind of terminal device, including memory, processor and is stored in
In the memory and the computer-readable instruction that can run on the processor, the processor executes the computer can
Following steps are realized when reading instruction:
The remote desktop operation request that user issues is received, and extracts preset mould from remote desktop operation request
Plate image and operational order；
The video in window of remote desktop is obtained, and is searched and the template image in the video in window of the remote desktop
Matched target subgraph；
First coordinate of the target subgraph in the video in window is obtained, and obtains the window of the remote desktop
Second coordinate of the image in local desktop；
The of the target subgraph in the local desktop is calculated according to first coordinate and second coordinate
Three coordinates；
The operational order is executed at the third coordinate in the local desktop.
Existing beneficial effect is the embodiment of the present invention compared with prior art: the embodiment of the present invention is in the case where receiving user
After the remote desktop operation request of hair, preset template image is extracted from remote desktop operation request first and operation refers to
It enables, then obtains the video in window of remote desktop, and searched and the template image in the video in window of the remote desktop
Matched target subgraph then obtains first coordinate of the target subgraph in the video in window, and described in acquisition
Second coordinate of the video in window of remote desktop in local desktop, finally according to first coordinate and the second coordinate meter
Third coordinate of the target subgraph in the local desktop is calculated, and is executed at the third coordinate in the local desktop
The operational order.Through the embodiment of the present invention, traditional method that operation object is determined by window handle is no longer used, and
It is to search operation object in remote desktop using the mode of images match, and pass through between local desktop and remote desktop
Coordinate transform determines the specific location of operation object, to substantially increase operation success rate.
Detailed description of the invention
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to embodiment or description of the prior art
Needed in attached drawing be briefly described, it should be apparent that, the accompanying drawings in the following description is only of the invention some
Embodiment for those of ordinary skill in the art without any creative labor, can also be according to these
Attached drawing obtains other attached drawings.
Fig. 1 is a kind of one embodiment flow chart of remote desktop operation method in the embodiment of the present invention；
Fig. 2 is the exemplary flow searched in the video in window of remote desktop with the matched target subgraph of template image
Figure；
Fig. 3 is the schematic flow diagram of the feature vector set of calculation template image；
Fig. 4 is a kind of one embodiment structure chart of remote desktop operation device in the embodiment of the present invention；
Fig. 5 is a kind of schematic block diagram of terminal device in the embodiment of the present invention.
Specific embodiment
In order to make the invention's purpose, features and advantages of the invention more obvious and easy to understand, below in conjunction with the present invention
Attached drawing in embodiment, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that disclosed below
Embodiment be only a part of the embodiment of the present invention, and not all embodiment.Based on the embodiments of the present invention, this field
Those of ordinary skill's all other embodiment obtained without making creative work, belongs to protection of the present invention
Range.
Referring to Fig. 1, a kind of one embodiment of remote desktop operation method may include: in the embodiment of the present invention
Step S101, the remote desktop operation request that user issues is received, and is mentioned from remote desktop operation request
Take preset template image and operational order.
The template image is the image for the object that user wants operation in remote desktop, which can be by
User selects previously according to actual demand, for example, if it is a document in remote desktop that user, which wants the object operated,
The image of the document can be then intercepted in remote desktop as template image.When user carries out screenshot, screenshot can recorde
The affiliated handle information in region, here, handle is remote desktop.The operational order includes but is not limited to: mouse-click, mouse
Mark the specific operational orders such as double-click, mouse-over, input text.
Step S102, the video in window of remote desktop is obtained, and search and institute in the video in window of the remote desktop
State the matched target subgraph of template image.
When executing operation, pre-recorded handle information namely remote desktop, the window of the top set remote desktop are found first
Mouthful, it is therefore an objective to guarantee that mouse can click on the window.Then the present image namely the remote desktop of the window are intercepted
Video in window, and searched in the video in window of the remote desktop by process as shown in Figure 2 and the template image
Matched target subgraph:
Step S1021, each subgraph is successively intercepted from the video in window of the remote desktop.
Also it regard the image of each icon area in the video in window of the remote desktop as a subgraph
Picture, so as to intercept outstanding more subgraph from the video in window of the remote desktop, herein, by the remote desktop
Number of sub-images in video in window is SN, the serial number of each subgraph is successively labeled as 1,2,3 ..., s ..., SN,
In, 1≤s≤SN.
Step S1022, the feature vector set of the template image and the feature vector of each subgraph are calculated
Set.
It include the feature vector of each characteristic point in described eigenvector set.
As shown in figure 3, the process for calculating the feature vector set of the template image can specifically include following steps:
Step S10221, respectively using each smooth function in preset smooth function set to the template image into
Row smoothing processing obtains smoothed image sequence corresponding with the template image.
Preferably, before being smoothed, it can use first and gray scale is carried out to color image with weighted mean method
Change handles to obtain gray level image, specifically, can carry out according to the following formula gray processing processing:
Gray=0.30 × R+0.59 × G+0.11 × B
Wherein, R, G, B respectively represent the red component, green component and blue component of pixel, and Gray represents pixel
Gray value.
Each smooth function in the smooth function set is as follows:
Wherein, f is the serial number of each smooth function, and 1≤f≤FN, FN are the smooth function in the smooth function set
Number, exp is natural exponential function, σfFor the standard deviation of f-th of smooth function, and σf<σf+1, Gf(x, y) is smooth f-th
Function.
Image is smoothly obscured with the smooth function of different scale (standard deviation), so, the image volume of script
The image under multiple different fog-levels is increased after product.
Step S10222, difference image sequence corresponding with the smoothed image sequence is constructed.
It altogether include FN difference image in the difference image sequence, wherein f-th in the difference image sequence is poor
Partial image is the difference image between f-th of smoothed image and the f-1 smoothed image in the smoothed image sequence, namely
The gray value of f-th of smoothed image and the f-1 smoothed image corresponding pixel points is subtracted each other into obtained image.
Distinguishingly, the 1st difference image in the difference image sequence is the 1st in the smoothed image sequence flat
Difference image between sliding image and the 0th smoothed image, the original that the 0th smoothed image herein is not smoothed as
Beginning image.
Step S10223, each characteristic point is chosen from the difference image sequence, and calculates separately each characteristic point
Feature vector.
Difference image reflection is the smooth difference for obscuring rear image, and the big pixel of difference is exactly characteristic point.Firstly, with every
A pixel, can be as extreme point when it is more than or less than all consecutive points compared with surrounding pixel.
But this extreme point is decided after difference of Gaussian, then it is the point belonged on discrete space, is not necessarily
Extreme point truly.It also needs to be fitted using a curve, following Taylor expansion is used in the present embodiment
It is converted to discrete continuously:
Then corresponding extreme point truly are as follows:As characteristic point.
By above step, required accurate feature points are substantially obtained, next according to the following formula to difference image
Characteristic point carries out finite difference, acquires the argument and amplitude of image gradient in region:
Wherein, (x, y) is characterized coordinate a little, and L (x+1, y) is the gray value for the pixel that coordinate is (x+1, y), L (x-
1, y) be the pixel that coordinate is (x-1, y) gray value, L (x, y+1) be coordinate for the gray value of the pixel of (x, y+1), L
(x, y-1) is the gray value for the pixel that coordinate is (x, y-1), and m (x, y) is the amplitude of image gradient, and θ (x, y) is image ladder
The argument of degree.
When each characteristic point is described, 4 × 4 totally 16 sub-regions delimited in characteristic point contiguous range, each
0~360 ° of omnidirection is divided into 8 sub- directions (every sub- direction width is 45 °), it is strong to count each sub- direction gradient by subregion
Degree, 8 dimension subregion description vectors of building.For 16 sub-regions, then 4 × 4 × 8=128 dimension description vectors are ultimately generated, namely
The feature vector of characteristic point.
Step S10224, the feature vector set of each characteristic point is configured to the set of eigenvectors of the template image
It closes.
The above is the process that calculates the feature vector set of the template image, the feature of each subgraph to
The calculating process of duration set is similar therewith, specifically can refer to above-mentioned specific narration content, details are not described herein again.
Step S1023, calculate separately the template image feature vector set and each subgraph feature to
The degree of correlation between duration set.
Search second feature point corresponding with each fisrt feature point respectively first in s-th of subgraph.
The fisrt feature point is the characteristic point in the template image, and the second feature point is in s-th of subgraph
Characteristic point.
Then, the feature vector set of the template image and the set of eigenvectors of s-th of subgraph are calculated according to the following formula
The degree of correlation between conjunction:
Wherein, p is characterized serial number a little, and 1≤p≤PN, PN are the number of fisrt feature point, and d is the dimension of feature vector
Serial number, 1≤d≤DN, DN are the dimension number of feature vector, MtDegs,pFor p-th of fisrt feature point and corresponding second feature
Matching degree between point, andFtValp,dFor p-th first spies
Levy value of the feature vector of point in d-th of dimension, SvFtVals,p,dFor second feature corresponding with p-th of fisrt feature point
Value of the feature vector of point in d-th of dimension, distinguishingly, if be not present in s-th of subgraph and p-th first spies
The corresponding second feature point of sign point, then have MtDegs,p=0, MatchDegsFor the feature vector set and the of the template image
The degree of correlation between the feature vector set of s subgraph.
Step S1024, the subgraph work that the degree of correlation is greater than preset relevance threshold is chosen from each subgraph
For the target subgraph.
The relevance threshold can be configured according to the actual situation, for example, can be set to 90%, 95%,
98% or other values.
Step S103, first coordinate of the target subgraph in the video in window is obtained, and is obtained described long-range
Second coordinate of the video in window of desktop in local desktop.
It, can be using the pixel in the video in window most upper left corner as seat for first coordinate
Origin is marked, direction from left to right is as X-axis forward direction, and direction from top to bottom is as Y-axis forward direction.Certainly, the above is only one kind
Feasible coordinate definition mode can also choose other coordinate definition modes according to the actual situation.Due to the target subgraph
Seem a region, can therefrom choose the coordinate of any one pixel as first coordinate, it is preferable that can be by institute
It states the coordinate of the central point of target subgraph and is denoted as (AxisX as first coordinate, and by first coordinate1,
AxisY1)。
For second coordinate, can using the pixel in the local desktop most upper left corner as coordinate origin,
Direction from left to right is as X-axis forward direction, and direction from top to bottom is as Y-axis forward direction.Certainly, the above is only a kind of feasible seats
Definition mode is marked, other coordinate definition modes can also be chosen according to the actual situation.From the video in window of the remote desktop
The coordinate for choosing the pixel in the most upper left corner is denoted as (AxisX as second coordinate, and by second coordinate2,
AxisY2)。
Step S104, the target subgraph is calculated in the local table according to first coordinate and second coordinate
Third coordinate in face.
Specifically, the third coordinate can be calculated according to the following formula:
Wherein, (AxisX3, AxisY3) it is the third coordinate.
Step S105, the operational order is executed at the third coordinate in the local desktop.
After the coordinate for determining the target subgraph (namely operation object), can in the local desktop the
The operational order is executed to operation object at three coordinates, for example, operation object can be carried out mouse-click, double click,
The specific operations such as mouse-over, input text, to realize the precise manipulation to operation object in remote desktop.
In conclusion the embodiment of the present invention is after receiving the remote desktop operation request that user issues, first from described
Preset template image and operational order are extracted in remote desktop operation request, then obtains the video in window of remote desktop, and
Search and the matched target subgraph of the template image, then obtain the target in the video in window of the remote desktop
First coordinate of the subgraph in the video in window, and obtain of the video in window of the remote desktop in local desktop
Two coordinates finally calculate the target subgraph in the local desktop according to first coordinate and second coordinate
Third coordinate, and the operational order is executed at the third coordinate in the local desktop.Through the embodiment of the present invention, no longer
Using traditional method for determining operation object by window handle, but searched in remote desktop using the mode of images match
Operation object is sought, and determines the specific location of operation object by the coordinate transform between local desktop and remote desktop, from
And substantially increase operation success rate.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
Corresponding to a kind of remote desktop operation method described in foregoing embodiments, Fig. 4 shows offer of the embodiment of the present invention
A kind of remote desktop operation device one embodiment structure chart.
In the present embodiment, a kind of remote desktop operation device may include:
Operation requests receiving module 401, for receiving the remote desktop operation request that user issues, and from the long-range table
Preset template image and operational order are extracted in the operation requests of face；
Target image search module 402, for obtaining the video in window of remote desktop, and in the window of the remote desktop
It is searched and the matched target subgraph of the template image in image；
Coordinate obtaining module 403 for obtaining first coordinate of the target subgraph in the video in window, and obtains
Take second coordinate of the video in window of the remote desktop in local desktop；
Coordinate calculation module 404, for calculating the target subgraph according to first coordinate and second coordinate
Third coordinate in the local desktop；
Operation executing module 405, for executing the operational order at the third coordinate in the local desktop.
Further, the target image search module may include:
Subgraph interception unit, for successively intercepting each subgraph from the video in window of the remote desktop；
Set of eigenvectors close computing unit, for calculate the template image feature vector set and each height
The feature vector set of image includes the feature vector of each characteristic point in described eigenvector set；
Correlation calculating unit, for calculate separately the template image feature vector set and each subgraph
Feature vector set between the degree of correlation；
Target image selection unit is greater than preset relevance threshold for choosing the degree of correlation from each subgraph
Subgraph as the target subgraph.
Further, described eigenvector set computing unit may include:
Smoothing processing subelement, for using each smooth function in preset smooth function set to the mould respectively
Plate image is smoothed, and obtains smoothed image sequence corresponding with the template image, in the smooth function set
Each smooth function is as follows:
Wherein, f is the serial number of each smooth function, and 1≤f≤FN, FN are the smooth function in the smooth function set
Number, exp is natural exponential function, σfFor the standard deviation of f-th of smooth function, and σf<σf+1, Gf(x, y) is smooth f-th
Function；
Difference image sequence structure subelement, for constructing difference image sequence corresponding with the smoothed image sequence,
Wherein, f-th of difference image in the difference image sequence is f-th of smoothed image in the smoothed image sequence and the
Difference image between f-1 smoothed image；
Characteristic point chooses subelement, for choosing each characteristic point from the difference image sequence, and calculates separately each
The feature vector of a characteristic point；
Feature vector set constructor unit, for the feature vector set of each characteristic point to be configured to the Prototype drawing
The feature vector set of picture.
Further, the correlation calculating unit may include:
Characteristic point searches subelement, for searching corresponding with each fisrt feature point the respectively in s-th of subgraph
Two characteristic points, the fisrt feature point are the characteristic point in the template image, and the second feature point is in s-th of subgraph
Characteristic point, 1≤s≤SN, SN be subgraph number；
Relatedness computation subelement, for calculating the feature vector set and s-th of son of the template image according to the following formula
The degree of correlation between the feature vector set of image:
Wherein, p is characterized serial number a little, and 1≤p≤PN, PN are the number of fisrt feature point, and d is the dimension of feature vector
Serial number, 1≤d≤DN, DN are the dimension number of feature vector, MtDegs,pFor p-th of fisrt feature point and corresponding second feature
Matching degree between point, andFtValp,dFor p-th first spies
Levy value of the feature vector of point in d-th of dimension, SvFtVals,p,dFor second feature corresponding with p-th of fisrt feature point
Value of the feature vector of point in d-th of dimension, MatchDegsFor the feature vector set of the template image and s-th
The degree of correlation between the feature vector set of subgraph.
Further, the coordinate calculation module is specifically used for calculating the third coordinate according to the following formula:
Wherein, (AxisX1, AxisY1) it is first coordinate, (AxisX2, AxisY2) it is second coordinate,
(AxisX3, AxisY3) it is the third coordinate.
It is apparent to those skilled in the art that for convenience and simplicity of description, the device of foregoing description,
The specific work process of module and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, is not described in detail or remembers in some embodiment
The part of load may refer to the associated description of other embodiments.
The schematic block diagram that Fig. 5 shows a kind of terminal device provided in an embodiment of the present invention is only shown for ease of description
Part related to the embodiment of the present invention.
In the present embodiment, the terminal device 5 can be desktop PC, notebook, palm PC and cloud clothes
Business device etc. calculates equipment.The terminal device 5 can include: processor 50, memory 51 and be stored in the memory 51 simultaneously
The computer-readable instruction 52 that can be run on the processor 50, such as execute the calculating of above-mentioned remote desktop operation method
Machine readable instruction.The processor 50 realizes above-mentioned each remote desktop operation method when executing the computer-readable instruction 52
Step in embodiment, such as step S101 to S105 shown in FIG. 1.Alternatively, the processor 50 execute the computer can
The function of each module/unit in above-mentioned each Installation practice, such as the function of module 401 to 405 shown in Fig. 4 are realized when reading instruction 52
Energy.
Illustratively, the computer-readable instruction 52 can be divided into one or more module/units, one
Or multiple module/units are stored in the memory 51, and are executed by the processor 50, to complete the present invention.Institute
Stating one or more module/units can be the series of computation machine readable instruction section that can complete specific function, the instruction segment
For describing implementation procedure of the computer-readable instruction 52 in the terminal device 5.
The processor 50 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
The memory 51 can be the internal storage unit of the terminal device 5, such as the hard disk or interior of terminal device 5
It deposits.The memory 51 is also possible to the External memory equipment of the terminal device 5, such as be equipped on the terminal device 5
Plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card dodge
Deposit card (Flash Card) etc..Further, the memory 51 can also both include the storage inside list of the terminal device 5
Member also includes External memory equipment.The memory 51 is for storing the computer-readable instruction and the terminal device 5
Required other instruction and datas.The memory 51 can be also used for temporarily storing the number that has exported or will export
According to.
The functional units in various embodiments of the present invention may be integrated into one processing unit, is also possible to each
Unit physically exists alone, and can also be integrated in one unit with two or more units.Above-mentioned integrated unit both may be used
To use formal implementation of hardware, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product
When, it can store in a computer readable storage medium.Based on this understanding, technical solution of the present invention substantially or
Person says that all or part of the part that contributes to existing technology or the technical solution can body in the form of software products
Reveal and, which is stored in a storage medium, including several computer-readable instructions are used so that one
Platform computer equipment (can be personal computer, server or the network equipment etc.) executes described in each embodiment of the present invention
The all or part of the steps of method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-
Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. are various can be with
Store the medium of computer-readable instruction.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution.
Claims (10)
1. a kind of remote desktop operation method characterized by comprising
The remote desktop operation request that user issues is received, and extracts preset Prototype drawing from remote desktop operation request
Picture and operational order；
The video in window of remote desktop is obtained, and searches in the video in window of the remote desktop and is matched with the template image
Target subgraph；
First coordinate of the target subgraph in the video in window is obtained, and obtains the video in window of the remote desktop
The second coordinate in local desktop；
Third of the target subgraph in the local desktop is calculated according to first coordinate and second coordinate to sit
Mark；
The operational order is executed at the third coordinate in the local desktop.
2. remote desktop operation method according to claim 1, which is characterized in that the window in the remote desktop
It searches in image with the matched target subgraph of the template image and includes:
Each subgraph is successively intercepted from the video in window of the remote desktop；
Calculate the feature vector set of the template image and the feature vector set of each subgraph, the feature to
It include the feature vector of each characteristic point in duration set；
Calculate separately the phase between the feature vector set of the template image and the feature vector set of each subgraph
Guan Du；
The degree of correlation is chosen from each subgraph is greater than the subgraph of preset relevance threshold as the target subgraph
Picture.
3. remote desktop operation method according to claim 2, which is characterized in that the spy for calculating the template image
Levying vector set includes:
The template image is smoothed using each smooth function in preset smooth function set respectively, is obtained
Smoothed image sequence corresponding with the template image, each smooth function in the smooth function set are as follows:
Wherein, f is the serial number of each smooth function, and 1≤f≤FN, FN are the number of the smooth function in the smooth function set
Mesh, exp are natural exponential function, σfFor the standard deviation of f-th of smooth function, and σf<σf+1, Gf(x, y) is f-th of smooth letter
Number；
Construct difference image sequence corresponding with the smoothed image sequence, wherein f-th in the difference image sequence is poor
Partial image is the difference image between f-th of smoothed image and the f-1 smoothed image in the smoothed image sequence；
Each characteristic point is chosen from the difference image sequence, and calculates separately the feature vector of each characteristic point；
The feature vector set of each characteristic point is configured to the feature vector set of the template image.
4. remote desktop operation method according to claim 2, which is characterized in that described to calculate separately the template image
Feature vector set and each subgraph feature vector set between the degree of correlation include:
Search second feature point corresponding with each fisrt feature point respectively in s-th of subgraph, the fisrt feature point is
Characteristic point in the template image, the second feature point are the characteristic point in s-th of subgraph, and 1≤s≤SN, SN are son
The number of image；
The phase between the feature vector set of the template image and the feature vector set of s-th of subgraph is calculated according to the following formula
Guan Du:
Wherein, p is characterized serial number a little, and 1≤p≤PN, PN are the number of fisrt feature point, and d is the dimension serial number of feature vector,
1≤d≤DN, DN are the dimension number of feature vector, MtDegs,pFor p-th of fisrt feature point and corresponding second feature point it
Between matching degree, andFtValp,dFor p-th of fisrt feature point
Value of the feature vector in d-th of dimension, SvFtVals,p,dFor second feature point corresponding with p-th of fisrt feature point
Value of the feature vector in d-th of dimension, MatchDegsFor the feature vector set and s-th of subgraph of the template image
The degree of correlation between the feature vector set of picture.
5. remote desktop operation method according to any one of claim 1 to 4, which is characterized in that described according to
First coordinate and second coordinate calculate third coordinate of the target subgraph in the local desktop
The third coordinate is calculated according to the following formula:
Wherein, (AxisX1, AxisY1) it is first coordinate, (AxisX2, AxisY2) it is second coordinate, (AxisX3,
AxisY3) it is the third coordinate.
6. a kind of remote desktop operation device characterized by comprising
Operation requests receiving module, for receiving the remote desktop operation request that user issues, and from the remote desktop operation
Preset template image and operational order are extracted in request；
Target image search module, for obtaining the video in window of remote desktop, and in the video in window of the remote desktop
It searches and the matched target subgraph of the template image；
Coordinate obtaining module, for obtaining first coordinate of the target subgraph in the video in window, and described in acquisition
Second coordinate of the video in window of remote desktop in local desktop；
Coordinate calculation module, for calculating the target subgraph at described according to first coordinate and second coordinate
Third coordinate in ground desktop；
Operation executing module, for executing the operational order at the third coordinate in the local desktop.
7. remote desktop operation device according to claim 6, which is characterized in that the target image search module packet
It includes:
Subgraph interception unit, for successively intercepting each subgraph from the video in window of the remote desktop；
Set of eigenvectors close computing unit, for calculate the template image feature vector set and each subgraph
Feature vector set, include the feature vector of each characteristic point in described eigenvector set；
Correlation calculating unit, for calculating separately the feature vector set of the template image and the spy of each subgraph
Levy the degree of correlation between vector set；
Target image selection unit, the son for being greater than preset relevance threshold for choosing the degree of correlation from each subgraph
Image is as the target subgraph.
8. remote desktop operation device according to claim 7, which is characterized in that described eigenvector set computing unit
Include:
Smoothing processing subelement, for using each smooth function in preset smooth function set to the Prototype drawing respectively
Picture is smoothed, and obtains smoothed image sequence corresponding with the template image, each in the smooth function set
Smooth function is as follows:
Wherein, f is the serial number of each smooth function, and 1≤f≤FN, FN are the number of the smooth function in the smooth function set
Mesh, exp are natural exponential function, σfFor the standard deviation of f-th of smooth function, and σf<σf+1, Gf(x, y) is f-th of smooth letter
Number；
Difference image sequence structure subelement, for constructing difference image sequence corresponding with the smoothed image sequence, wherein
F-th of difference image in the difference image sequence is f-th of smoothed image and f-1 in the smoothed image sequence
Difference image between smoothed image；
Characteristic point chooses subelement, for choosing each characteristic point from the difference image sequence, and calculates separately each spy
Levy the feature vector of point；
Feature vector set constructor unit, for the feature vector set of each characteristic point to be configured to the template image
Feature vector set.
9. a kind of computer readable storage medium, the computer-readable recording medium storage has computer-readable instruction, special
Sign is, the long-range table as described in any one of claims 1 to 5 is realized when the computer-readable instruction is executed by processor
The step of face operating method.
10. a kind of terminal device, including memory, processor and storage are in the memory and can be on the processor
The computer-readable instruction of operation, which is characterized in that the processor realizes such as right when executing the computer-readable instruction
It is required that described in any one of 1 to 5 the step of remote desktop operation method.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910032791.9A CN109871249B (en) | 2019-01-14 | 2019-01-14 | Remote desktop operation method and device, readable storage medium and terminal equipment |
| PCT/CN2019/091528 WO2020147258A1 (en) | 2019-01-14 | 2019-06-17 | Remote desktop operation method and apparatus, readable storage medium, and terminal device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910032791.9A CN109871249B (en) | 2019-01-14 | 2019-01-14 | Remote desktop operation method and device, readable storage medium and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109871249A true CN109871249A (en) | 2019-06-11 |
| CN109871249B CN109871249B (en) | 2021-08-24 |
Family
ID=66917584
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910032791.9A Active CN109871249B (en) | 2019-01-14 | 2019-01-14 | Remote desktop operation method and device, readable storage medium and terminal equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109871249B (en) |
| WO (1) | WO2020147258A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110046658A (en) * | 2019-03-30 | 2019-07-23 | 华为技术服务有限公司 | The method and relevant apparatus of remote operation apparatus |
| WO2020147258A1 (en) * | 2019-01-14 | 2020-07-23 | 平安科技（深圳）有限公司 | Remote desktop operation method and apparatus, readable storage medium, and terminal device |
| CN114706642A (en) * | 2022-03-25 | 2022-07-05 | 曲阜师范大学 | Remote desktop operation method based on rapid matching of webpage command line and subgraph |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101132571A (en) * | 2006-08-22 | 2008-02-27 | 薛飞 | Method and system for displaying and operating remote computer on mobile phone |
| CN103092518A (en) * | 2013-01-24 | 2013-05-08 | 福建升腾资讯有限公司 | Moving cloud desktop accurate touch method based on remote desktop protocol (RDP) |
| CN103631551A (en) * | 2012-08-27 | 2014-03-12 | 深圳市蓝韵网络有限公司 | Local display method for remote desktop in medical consultation system |
| CN104348878A (en) * | 2013-08-06 | 2015-02-11 | 好看科技（深圳）有限公司 | Method, device and system for remotely controlling mobile terminal as well as mobile terminal |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
| US20170169227A1 (en) * | 2007-03-06 | 2017-06-15 | Unisys Corporation | Methods and systems for providing and controlling cryptographic secure communications terminal providing a remote desktop accessible in secured and unsecured environments |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5987299B2 (en) * | 2011-11-16 | 2016-09-07 | ソニー株式会社 | Display control apparatus, display control method, and program |
| CN105989029A (en) * | 2015-02-02 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Image searching method and image searching system |
| JP2018176387A (en) * | 2017-04-19 | 2018-11-15 | 富士ゼロックス株式会社 | Robot device and program |
| CN109118347B (en) * | 2018-07-20 | 2022-12-27 | 苏宁易购集团股份有限公司 | Automatic cooperation method and system |
| CN109871249B (en) * | 2019-01-14 | 2021-08-24 | 平安科技（深圳）有限公司 | Remote desktop operation method and device, readable storage medium and terminal equipment |
-
2019
- 2019-01-14 CN CN201910032791.9A patent/CN109871249B/en active Active
- 2019-06-17 WO PCT/CN2019/091528 patent/WO2020147258A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101132571A (en) * | 2006-08-22 | 2008-02-27 | 薛飞 | Method and system for displaying and operating remote computer on mobile phone |
| US20170169227A1 (en) * | 2007-03-06 | 2017-06-15 | Unisys Corporation | Methods and systems for providing and controlling cryptographic secure communications terminal providing a remote desktop accessible in secured and unsecured environments |
| CN103631551A (en) * | 2012-08-27 | 2014-03-12 | 深圳市蓝韵网络有限公司 | Local display method for remote desktop in medical consultation system |
| CN103092518A (en) * | 2013-01-24 | 2013-05-08 | 福建升腾资讯有限公司 | Moving cloud desktop accurate touch method based on remote desktop protocol (RDP) |
| CN104348878A (en) * | 2013-08-06 | 2015-02-11 | 好看科技（深圳）有限公司 | Method, device and system for remotely controlling mobile terminal as well as mobile terminal |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020147258A1 (en) * | 2019-01-14 | 2020-07-23 | 平安科技（深圳）有限公司 | Remote desktop operation method and apparatus, readable storage medium, and terminal device |
| CN110046658A (en) * | 2019-03-30 | 2019-07-23 | 华为技术服务有限公司 | The method and relevant apparatus of remote operation apparatus |
| CN110046658B (en) * | 2019-03-30 | 2021-08-20 | 华为技术服务有限公司 | Method for remote operation of a device and related apparatus |
| CN114706642A (en) * | 2022-03-25 | 2022-07-05 | 曲阜师范大学 | Remote desktop operation method based on rapid matching of webpage command line and subgraph |
| CN114706642B (en) * | 2022-03-25 | 2023-07-25 | 曲阜师范大学 | Remote desktop operation method based on quick matching of webpage command line and subgraph |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020147258A1 (en) | 2020-07-23 |
| CN109871249B (en) | 2021-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Poulenard et al. | Effective rotation-invariant point cnn with spherical harmonics kernels | |
| Liu et al. | Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images | |
| Jin et al. | A deep 3D residual CNN for false‐positive reduction in pulmonary nodule detection | |
| Battle et al. | Dynamic prefetching of data tiles for interactive visualization | |
| US8571328B2 (en) | Determining correspondence between image regions | |
| WO2018021942A2 (en) | Facial recognition using an artificial neural network | |
| Donati et al. | Deep orientation-aware functional maps: Tackling symmetry issues in shape matching | |
| CN108694055A (en) | Vectorial reduction processor | |
| Sui et al. | Goal-directed robot manipulation through axiomatic scene estimation | |
| CN109871249A (en) | A kind of remote desktop operation method, apparatus, readable storage medium storing program for executing and terminal device | |
| EP2668618A1 (en) | Method and system for comparing images | |
| CN107886516A (en) | The method and computing device that hair moves towards in a kind of calculating portrait | |
| Ravari et al. | Reconstruction of B-spline curves and surfaces by adaptive group testing | |
| US12002260B2 (en) | Automatic topology mapping processing method and system based on omnidirectional image information | |
| Bourmaud | Online variational bayesian motion averaging | |
| Sui et al. | Tracking completion | |
| Hegde et al. | Learning manifolds in the wild | |
| Slimani et al. | RoCNet++: Triangle-based descriptor for accurate and robust point cloud registration | |
| CN111488479A (en) | Hypergraph construction method, hypergraph construction device, computer system and medium | |
| Tang et al. | Image dataset creation and networks improvement method based on CAD model and edge operator for object detection in the manufacturing industry | |
| Zhang et al. | Cross-cascading regression for simultaneous head pose estimation and facial landmark detection | |
| Jiang et al. | Progressive filtering for feature matching | |
| Lin et al. | SiamLight: lightweight networks for object tracking via attention mechanisms and pixel-level cross-correlation | |
| CN108629798A (en) | Rapid Image Registration method based on GPU | |
| Yuan et al. | Anisotropic neighborhood searching for point cloud with sharp feature |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |