CN105027054A

CN105027054A - Gesture operation conversion method and apparatus

Info

Publication number: CN105027054A
Application number: CN201380002623.6A
Authority: CN
Inventors: 陈普; 许天锡
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2013-11-30
Filing date: 2013-11-30
Publication date: 2015-11-04
Also published as: WO2015078020A1

Abstract

The embodiments of the present invention are a gesture operation conversion method, apparatus and system, comprising: receiving gesture operation information of a user; obtaining focus application information of the current operation of the user; converting the gesture operation information to mouse and keyboard information according to the focus application information. The embodiments of the present invention can identify the gesture operation of the user according to the user operation scene, so as to enable the user to adopt the adaptive gesture for the operation scene in different operation scenes without presetting gesture operation solution, and enable the system to be converted to the adaptive keyboard and mouse information.

Description

Gesture operation conversion method and apparatus

The conversion method and engineering device technique field of a kind of gesture operation

The present embodiments relate to computer technology, particularly a kind of conversion method of gesture operation, system and device.Background technology

SmartClient of today, such as smart mobile phone, tablet personal computer, the standard peripherals of mouse-keyboard class is not had generally, user is only capable of by operation touch-screen（Touch pad）Change client-side to or position（Client etc. is such as rocked up and down）To operate client, the various operations for the user that client is captured can be referred to as the gesture operation of user.The gesture operation of the user captured is converted to mouse-keyboard message by the identification software of client, and by the mouse-keyboard messaging to operating system, the processing to user's operation is completed with this.Client preserves the gesture operation of user and the conversion corresponding relation of mouse-keyboard message, and client identification software is changed or recognized accordingly.

Because different users has different operating habits, different application programs（Such as Word, PPT, PDF etc.）Also there are different requirements to the operation of user, if a set of gesture operation scheme is separately configured, Consumer's Experience can be deteriorated, therefore, in the prior art, it is maximized lifting Consumer's Experience, client is prefixed the scheme of various gestures operation, user can be in operation scheme of the client by certain gesture of menu setecting, and user have selected the corresponding relation that gesture operation scheme have selected the gesture operation and key mouse message produced on certain touch-screen.Touch screen operation, according to the specific corresponding relation under the certain gestures operation scheme, is converted into mouse-keyboard message by client after the gesture operation of user is received.

In the prior art, user needs that the scheme of gesture operation is pre-selected before gesture operation, and in practical application, user is frequently necessary to be switched to another gesture operation scheme the need for according to current operation scene, so the business of user needs to interrupt and operated into the selection mode of gesture operation scheme in one's hands, if the comparison that this switching occurs is frequently, the impression of Consumer's Experience will be greatly reduced, and reduce system availability.The content of the invention

The embodiment of the present invention proposes a kind of conversion method of gesture operation, device and system, being capable of basis The operation scenario of user recognizes the gesture operation of user.

In a first aspect, the embodiment of the present invention proposes a kind of conversion method of gesture operation, including：Receive the information of the gesture operation of user；

Obtain the information of the focus application of user's current operation；

According to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

With reference in a first aspect, in the first possible implementation, in addition to：

Obtain switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；

Then the information according to the focus application, mouse-keyboard information is converted to by the gesture operation information, including：

Determine the type of the gesture operation；

The switching strategy set according to the type queries of the information of the focus application and the gesture operation, determines the type of current mouse keyboard operation；

According to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

With reference in a first aspect, in second of possible implementation, in addition to：

Obtain multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding then information according to the focus application of type of each switching strategy set comprising gesture operation and type that mouse-keyboard is operated, the gesture operation information is converted into mouse-keyboard information, including：

According to the information of the focus application, the switching strategy set of current home is determined；Determine the type of the gesture operation；

The switching strategy set of current home according to the type queries of the gesture operation, it is determined that currently The type of mouse-keyboard operation；

With reference to first aspect or the first of first aspect or two kind of possible implementation, in the third possible implementation, before the gesture operation information for receiving user, in addition to：

Foundation is connected with the remote session of server；

The information by the gesture operation is converted to after mouse-keyboard information, in addition to：The mouse-keyboard information is passed through into the remote session linkup transmit to the server.

With reference to first aspect or the first of first aspect or two kind of possible implementation, in the 4th kind of possible implementation, in addition to：

Monitor the information of the focus application of user's current operation；

The information of the focus application for obtaining user's current operation includes：

Obtain the information of the focus application for user's current operation that monitoring is obtained.

With reference to the 4th kind of possible implementation of first aspect, in the 5th kind of possible implementation, the information of the focus application of monitoring user's current operation, including：

The information of the preposition window of user's current operation described in timing probe, the process title of the focus application is determined according to the information of the preposition window；Or,

Installed in advance in each application program of operating system and apply Monitoring instruction, when the focus application by it is preposition when, so that the operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, the process title of the focus application is obtained and preserved from the operating system.

Second aspect, the embodiment of the present invention proposes a kind of gesture operation conversion method, including：After the remote session connection of foundation and client, the information of the gesture operation for the user that the client is sent is received；

Determine the information of the focus application of user's current operation；

According to the preset switching strategy of the information matches of the focus application, by the gesture operation Information is converted to mouse-keyboard information, and the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

With reference to second aspect, in the first possible implementation, in addition to：

Determine the type of the gesture operation；

With reference to second aspect, in second of possible implementation, in addition to：

The switching strategy set of current home, determines the type of current mouse keyboard operation according to the type queries of the gesture operation；

With reference to second aspect or the first of first aspect or two kind of possible implementation, in the third possible implementation, in addition to： The mouse-keyboard information is passed through into the remote session linkup transmit to the server.

With reference to second aspect or the first of first aspect or two kind of possible implementation, in the 4th kind of possible implementation, in addition to：

Monitor the information of the focus application of user's current operation；

With reference to the 4th kind of possible implementation of second aspect, in the 5th kind of possible implementation, the information of the focus application of monitoring user's current operation, including：

The third aspect, the embodiment of the present invention proposes a kind of client device, the conversion of the gesture operation for realizing user, including：

User operates trapping module, the information of the gesture operation for capturing user, and the information of the gesture operation is sent into gesture translation request gesture conversion；

Gesture translation, the information of the focus application for obtaining user's current operation；Gesture translation, it is additionally operable to according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

Fourth aspect, the embodiment of the present invention proposes a kind of server, for realizing the conversion that the user gesture of client is operated, including：

Key mouse module, for after the connection of the remote session of foundation and client, receiving the information of the gesture operation for the user that the client is sent； Gesture translation, the information of the focus application for determining user's current operation；Gesture translation, it is additionally operable to according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

5th aspect, the embodiment of the present invention proposes a kind of remote session system, the conversion of the gesture operation for realizing user, including：

Client device, it is connected for setting up with the remote session of server, receive the information of the gesture operation of user, obtain the information of the focus application of user's current operation, according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program, and the mouse-keyboard information is passed through into the remote session linkup transmit to the server；

Server, the information of the focus application for monitoring user's current operation, and the information for monitoring the obtained focus application is sent to the client device.

6th aspect, the embodiment of the present invention proposes a kind of remote session system, the conversion of the gesture operation for realizing user, including：

Client device, for set up be connected with the remote session of server, capture user gesture operation information, and by the information of the gesture operation by the remote session linkup transmit to server；Server, for the information for the gesture operation for receiving the user that the client is sent, determine the information of the focus application of user's current operation, according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.Methods, devices and systems provided in an embodiment of the present invention, after the gesture operation of user is got, it is determined that current focus application, and gone to match switching strategy according to the type of focus application and gesture operation, so as to the automatic mouse-keyboard information that the gesture operation of user is converted to adaptation current application scene, this method causes user in different operation scenarios without specifying gesture operation scheme, you can adopt The gesture adapted to the operation scenario, improves Consumer's Experience and system effectiveness.

Brief description of the drawings

Technical scheme in order to illustrate the embodiments of the present invention more clearly, the required accompanying drawing used in embodiment will be briefly described below, apparently, drawings in the following description are only some embodiments of the present invention, for those of ordinary skill in the art, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.

Fig. 1 is the system architecture diagram of Remote desk process；

Fig. 2 is a kind of Organization Chart of the system provided according to embodiments of the present invention；

Fig. 3 is the Organization Chart of another system provided according to embodiments of the present invention；

Fig. 4 be according to embodiments of the present invention in switching strategy set configuration diagram；

Fig. 5 is a kind of method flow diagram provided according to embodiments of the present invention；

Fig. 6 is another method flow diagram provided according to embodiments of the present invention；

Fig. 7 is a kind of composition figure of the computer provided according to embodiments of the present invention.Embodiment

The embodiment of the present invention proposes a kind of conversion method of gesture operation, device and system, the gesture operation of user can be recognized according to the operation scenario of user, so that user need not preassign gesture operation scheme in different operation scenarios, you can use the gesture of operation scenario adaptation and enable the system to be transformed into suitable keyboard and mouse information.

Conversion method, the device and system for the gesture operation that the embodiment of the present invention is proposed, go for smart mobile phone, the usage scenario of the ordinary meaning of tablet personal computer these clients, such as network browsing or experience game, it is readily applicable to the mobile office scene of client, for example, user logs in remote computer by client, its desktop contents is sent to client, and the desktop contents for the local device display remote computer for passing through client by remote computer by remote desktop technology.Remote desktop can include the technologies such as the display of the desktop picture of remote computer, remote sound transmission or the mapping of pluggable hardware.Remote desktop is properly termed as virtual desktop again.At least there is a server and a client under virtual desktop scene End, is logged on far-end server by the virtual desktop client software installed in client.

Remote desktop architecture scene has been merely given as the part related to the embodiment of the present invention as shown in figure 1, this implement scene is only a kind of exemplary illustration, but Fig. 1 is not the specific restriction applied to the present invention.Left side is the client of remote desktop in Fig. 1, and the apparatus-form of these clients can be tablet personal computer 101a, smart mobile phone 101b etc. in embodiments of the present invention.They access remote desktop service by network 102 using RDP 103.The η of server 104a 104 provide remote desktop, and the remote desktop of user is present on server in the form of the η of virtual machine 105a, 105b-" 105.Long-distance desktop management system 106, for providing the function such as the client of user and the mapping of virtual machine.Client is firstly connected to long-distance desktop management system 106, the virtual machine address of user is obtained, and then is connected to virtual machine, long-distance desktop management system 106 can be server, can also be ordinary personal computers etc., the present invention is not especially limited to this.User accesses the virtual machine that the user is distributed on server by client, and the client of the content transmission that server accesses user to user is shown.The embodiment of the present invention in order to realize the user gesture under different application scene operation identification or conversion, switching strategy can be pre-configured with, the switching strategy contains the type of the corresponding keyboard and mouse operation of type of gesture operations different under different application programs；Client or server, which are provided with, applies monitoring modular, and the module can monitor the focus application of user's current operation；Client or server are also equipped with gesture translation, the gesture translation can obtain the information of the focus application of user's current operation and the gesture operation of user, and the information of the current gesture operation of user is converted into corresponding keyboard and mouse information according to switching strategy.

As shown in Fig. 2 the system composition figure under applying to the application scenarios of remote desktop for the embodiment of the present invention, the remote session system includes client 21, server 22 and long-distance desktop management system 23.The switching strategy set that long-distance desktop management system 23 is used for gesture operation is managed, switching strategy management module 231 can for example be included, the long-distance desktop management system 23 can be to system manager, or directly provide switching strategy input interface to user, keeper or user is allowd to formulate switching strategy according to hobby or actual demand, and the switching strategy is input to long-distance desktop management system, form the merging of switching strategy collection to be preserved in switching strategy management module 231, switching strategy management Switching strategy set can be handed down to client 21 by module 231, and the opportunity issued can, according to the request of client 21 or when accessing server by customer end 22, specifically be issued the opportunity embodiment of the present invention and be not especially limited.The resource of server 22 has been abstracted into many virtual machines, every virtual machine can distribute to one or many clients 21 are used, operating system 221 can be installed on virtual machine, the operating system 221 can include applying monitoring modular 2211 and key mouse module 2212, it can be set up using monitoring modular 221 1 in client 21 and virtual machine after remote desktop session connection, monitor the focus application of user's current operation, and the information of the focus application monitored is sent to client 21, wherein, the focus application refers to the application program that user is currently being operated or prepares operation, or refer to application program corresponding to preposition window in system；Key mouse module 2212 is used for receiving the keyboard and mouse information that client 21 is sended over.Client 21 can be tablet personal computer or intelligent movable mobile phone, client 21 includes gesture translation 211, user's operation trapping module 212, user's operation trapping module 212 asks gesture to be changed for capturing the gesture operation information of user, and the information of the gesture operation being sent into gesture translation 211；Gesture translation 211 is used for the gesture operation information for operating trapping module 212 to obtain user from user, obtain the information of the focus application of user's current operation, switching strategy is obtained from long-distance desktop management system 23, according to the information matches switching strategy of the gesture operation information and the focus application, the information of the gesture operation of user is converted into keyboard and mouse information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.In above-mentioned Fig. 2, gesture conversion is completed by client, as shown in Figure 3, the system composition figure applied to for further embodiment of this invention under the application scenarios of remote desktop, the gesture operation converting system includes client 31, server 32 and long-distance desktop management system 33, from unlike embodiment illustrated in fig. 2, the conversion of gesture in the present embodiment is completed by server side.In the system as shown in fig. 3, long-distance desktop management system 33 is still used to be managed the switching strategy of gesture operation, for example, can include switching strategy management module 331, unlike, change plan Slightly switching strategy set can be handed down to server 32 by management module 331, and the opportunity issued can be issued according to the request or periodicity of server 32, specifically issue the opportunity embodiment of the present invention and be not especially limited.Client 31 can be tablet personal computer or intelligent movable mobile phone, client 31 can set up remote session connection with the virtual machine 1 in server 32, trapping module 311 is operated including user, user's operation trapping module 311 is used for the gesture operation information for capturing user, and gesture operation information is passed through into remote session linkup transmit to virtual machine 1.The resource of server 32 has been abstracted into many virtual machines, and every virtual machine can distribute to one or many clients 31 are used.Operating system 321 can be installed on virtual machine 1, the operating system 321 can include application monitoring modular 3211, key mouse module 3212 and gesture translation 3213, it can be set up using monitoring modular 3211 in client 31 and virtual machine 1 after remote desktop session connection, the focus application of user's current operation is monitored, and is preserved；Key mouse module 3212 is used for receiving gesture operation information of the client 31 by the user of remote session linkup transmit, and the gesture operation information is sent to gesture translation 3213 changed, gesture translation 3213 is used for after receiving convert task, according to the information of the focus application, the information of the gesture operation is converted into mouse-keyboard information.Specifically, the gesture translation 3213 obtains the information of current focus application from application monitoring modular, and switching strategy is obtained from long-distance desktop management system 33（Specifically, gesture translation 3213 can obtain the switching strategy set that the long-distance desktop management system 33 of own cache is issued in advance, can also ask to obtain switching strategy set to long-distance desktop management system 33 temporarily）And according to the switching strategy and the information of focus application, the information of the gesture operation of user is converted into keyboard and mouse information, and the keyboard and mouse information being converted to is returned into key mouse module 3212, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.Key mouse module 3212 is got after keyboard and mouse information, and keyboard and mouse information can also be converted into keyboard and mouse event, and keyboard and mouse event is notified into operating system 321, to cause operating system 321 to handle the affairs of client 31. In the system shown in Fig. 2 and Fig. 3, due to being prefixed in advance comprising application program, the switching strategy set of corresponding relation between the type three of type and the keyboard and mouse operation of gesture operation, and the information for the focus application for monitoring user's current operation in real time using monitoring modular is also set up in virtual machine, allow after gesture translation gets the gesture operation of user, it is determined that current focus application, and gone to match switching strategy set according to the type of focus application and gesture operation, so as to the automatic mouse-keyboard information that the gesture operation of user is converted to adaptation current application scene, this method causes user in different operation scenarios without specifying gesture operation scheme, the gesture that can be adapted to using the operation scenario, improve Consumer's Experience and system effectiveness.Expansion introduction is carried out to the embodiment of the present invention below.

First, the preset switching strategy of the configuration of switching strategy set indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.It can be represented by the form of switching strategy set.The corresponding relation of the type of information of the one of which switching strategy set set comprising application program, the type of gesture operation and mouse-keyboard operation, the corresponding relation of three can be preserved by data list, each inquiry keyboard mouse message be required for and meanwhile use application program information and gesture operation type as search index；Another form of expression, it can also be preserved by multiple data lists, for example, multiple corresponding relations are set up respectively, first set up multiple switching strategy set of application program and switching strategy set, the corresponding relation for the type that the type that each switching strategy set includes gesture operation again is operated with mouse-keyboard, as shown in Figure 4.Switching strategy set shown in Fig. 4 includes multiple switching strategy set, the one or more application programs of each switching strategy set correspondence, the corresponding relation of the type and the type of keyboard and mouse operation of gesture operation can be included inside each switching strategy set, for example in the switching strategy set 1 of Fig. 4, the corresponding gesture operation of virtual key layout is relative keyboard key assignments, user touches which button of dummy keyboard, just convert thereof into corresponding keyboard key assignments, the gesture operation correspondence mouse roller of the downward screen sliding of user is scrolled down through, user tilts the corresponding relations such as the left button of client device correspondence keyboard to the left.It can be turned in switching strategy set with the gesture switching strategy of allocating default in inquiry less than corresponding When changing strategy, the gesture operation of user can be changed according to the gesture switching strategy of acquiescence.

Specifically, system as shown in Figure 2, the gesture translation 211 of described client 21 can also include the first tactful submodule, for obtaining switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；Or, the gesture translation 211 of described client 21 can also include the second tactful submodule, for obtaining multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding relation for the type that type of each switching strategy set comprising gesture operation is operated with mouse-keyboard.System as shown in Figure 3, the gesture translation 3213 of the operating system of virtual machine in described server 32 can also include the first tactful submodule, for obtaining switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；Or, the gesture translation 3213 of described server can also include the second tactful submodule, for obtaining multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding relation for the type that type of each switching strategy set comprising gesture operation is operated with mouse-keyboard.

2nd, the monitoring of focus application

It can be arranged at using monitoring modular in the operating system of virtual machine, it can monitor the application program in focus condition that the focus application in virtual machine, i.e. user operate or prepare operation, or by preposition application program.

There can be kinds of schemes using monitoring module monitors focus application.The first scheme provided in an embodiment of the present invention is：Using monitoring modular timing or the information for the preposition window for periodically detecting user's current operation, and determine according to the information of the preposition window process title of the focus application.For example, Windows operating system API functions GetForegroundWindow is called using monitoring modular timing, current preposition window is obtained, then by calling Windows operating system API function GetWindowThreadProcessId^ PsGetCurrentProcess standing grain mouthful PsGetProcessImageFi leName to obtain the process title of focus application successively.Second scheme provided in an embodiment of the present invention is：The application monitoring modular is pre- in each application program of operating system First install and apply Monitoring instruction, when the focus application by it is preposition when, operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, and the process title of the focus application is notified into the application monitoring modular.For example, use Hook technologies, call Windows operating system api function SetWindowsHookEx that one hook is installed using monitoring modular, Hook Function is injected into each application program in system, when some application program is selected as focus, being installed to the Hook Function of the application program can be called, Windows operating system API function GetCurrentProcessId^ PsGetCurrentProcess and PsGetProcessImageFi leName can be called to finally give the process title of focus application successively in Hook Function.Above-described embodiment is by taking Windows operating system as an example, when running other operating systems outside Windows in server, such as Linux/UNIX, during Mac OS, associated process information can also be got by corresponding api function in system, and the embodiment of the present invention is repeated no more.

Using monitoring modular after current focus application is monitored, the information of the current focus application monitored can be recorded, or gesture translation can also be supplied to be preserved the information of focus application, for example if gesture translation is arranged at client, the process name of focus application is referred to as message and is sent to client, gesture conversion is carried out for it.3rd, the capture of user gesture operation information

The gesture operation for the user that user's operation trapping module can be recognized and captured.For example：

1) user clicks on some position of screen；

2) user slides on screen；

3) user uses peripheral hardware keyboard and mouse, or equivalent；

4) user uses gravity sensing and gyroscope associative operation, and such as move up and down equipment, to all directions reclining device etc..

The information of the gesture operation of the user captured is sent to gesture translation by user's operation trapping module, and gesture conversion is carried out for it.

Meanwhile, user's operation trapping module can also be responsible for detecting the peripheral hardware that client device has currently been inserted and activated, such as USB handles, external keyboard and mouse.When discovery client device has activated such peripheral hardware When, then gesture conversion can be disabled, it is automatic when extracting peripheral hardware to enable gesture conversion.4th, gesture is changed

Gesture translation is responsible for the gesture operation of user being converted into corresponding keyboard mouse message, and the module can be positioned over client（As shown in Figure 2）, can also be positioned over server（As shown in Figure 3）.Gesture translation, when remote session connection can be set up in client and server, obtains switching strategy collection from long-distance desktop management system and is merged into capable preservation.After a connection setup, gesture translation can constantly receive the information of the gesture operation for the user that user's operation trapping module is sent, being also possible to can be constantly from the process title that current focus application is received using monitoring modular, and the process title of certain focus application is also possible to not real-time reception but has and go to obtain again when demand.Once receiving after the gesture operation information of user, gesture translation determines the type of the gesture operation, according to the process title of current focus application and the type of the gesture operation, the type operated from switching strategy collection query to corresponding mouse-keyboard, and the information of the gesture operation of user is converted to corresponding keyboard and mouse information by the type operated according to the mouse-keyboard.The flow example one of the embodiment of the present invention

System with reference to shown in Fig. 2, a flow example of the embodiment of the present invention is as shown in figure 5, in the present embodiment, gesture conversion is completed by client, and the flow path switch of gesture operation includes：

S501 :Subscription client sets up Remote desk process by long-distance desktop management system and user virtual machine.

S502:Switching strategy set is sent to the gesture translation of client by long-distance desktop management system, so that it preserves switching strategy set.

The corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；Or, the switching strategy set includes multiple switching strategy set, the different application program of each switching strategy set correspondence, the corresponding relation for the type that type of each switching strategy set comprising gesture operation is operated with mouse-keyboard.

S503 :Some application program module in user virtual machine have invoked operating system API (Appl ication Programming Interface, application programming interface）Its window is preposition, The corresponding application records of the preposition window can be current focus application by the window manager of operating system, or due to the operation of user, operating system have selected some application program as focus application, and the application records are focus application by window manager.

S504:Using monitoring module monitors operating system, to obtain the information of the focus application in window manager；

If using having have recorded focus application before monitoring modular, can also contrast and finding whether current focus application is altered, if altered, using the information of the focus application of monitoring modular more new record.

The rapid S504 of Walk can be realized using two kinds of specific implementations：

S504-a, detects using monitoring modular timing or periodically the information of the preposition window that the user is currently being operated, and determines according to the information of the preposition window process title of the focus application.

S504-b, the application monitoring modular is installed and applies Monitoring instruction in advance in each application program of operating system, when the focus application by it is preposition when, operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, and the process title of the focus application is notified into the application monitoring modular.

S505:Using monitoring modular by the information of the focus application of current record（Such as process title）Message is packaged into, and is sent to the gesture translation of client, gesture translation preserves the information of the focus application.

It is rapid that certain Ci Walk are alternatively Ke Xuan Walk suddenly, using monitoring modular after newest focus application is got, the information of current focus application can simply be preserved, not performing S505, that the information of the focus application is sent into gesture translation Walk is rapid, but it is sent to gesture translation after the inquiry request of subsequently received gesture translation, then by the information of focus application.

S506:After the information of the focus application is received, gesture translation can also first determine the switching strategy collection of current home according to the information inquiry switching strategy set of the focus application After the switching strategy set of current home is determined, current translative mode can be changed so that be successfully received after the gesture operation of user, directly be changed under current translative mode, saved conversion time delay, improve efficiency.The rapid S506 of Walk are similarly Ke and select Zhi Hang Walk rapid, if switching strategy contains multiple switching strategy set, a kind of translative mode of each switching strategy set correspondence, S506 can then be performed, if what switching strategy was included is the corresponding relation of the type three of the information of application program, the type of gesture operation and mouse-keyboard operation, S506 is without performing.

S507:User's operation trapping module captures the gesture operation of user customer equipment, and the information of the gesture operation of user is transmitted into gesture translation.

S508:Gesture translation is received after the information of the gesture operation of user, determine the type of the gesture operation, the switching strategy set that current home is determined in the rapid S506 of the gesture operation Xin breath Cha Xun Walk is inquired about again, determine the type of the corresponding current mouse keyboard operation of type of the gesture operation, further according to the type of the current mouse keyboard operation of determination, the information of the gesture operation is converted to the information of keyboard and mouse.If not finding the type of corresponding mouse-keyboard operation, the type of selection acquiescence mouse-keyboard operation.

If Shang rapid S506 of Shu Walk are not carried out, then gesture translation is received after the gesture operation message of user, determine the type of the gesture operation, according to the type queries switching strategy set of the focus application and the gesture operation, determine the type of the corresponding current mouse keyboard operation of type of gesture operation, further according to the type of the current mouse keyboard operation of determination, the information of the gesture operation is converted to the information of keyboard and mouse.If not finding corresponding switching strategy, selection acquiescence switching strategy.

S509:The mouse-keyboard information being converted to is returned to user's operation trapping module by gesture translation.

S510:User's operation trapping module is taken from gesture translation after the mouse-keyboard message changed, and message is transmitted to the key mouse module of the virtual machine accessed to user in server.

S511 :Mouse-keyboard message is converted into the operating system that mouse-keyboard event feeds back to virtual machine by key mouse module.The flow example two of the embodiment of the present invention System with reference to shown in Fig. 3, a flow example of the embodiment of the present invention is as shown in fig. 6, in the present embodiment, gesture is changed to be completed by the virtual machine of server, and the flow path switch of gesture operation includes：

S601 :Subscription client sets up Remote desk process by long-distance desktop management system and user virtual machine.

S602 :Switching strategy is sent to the gesture translation of virtual machine by long-distance desktop management system, so that it preserves switching strategy.

S603 :Some application program module in user virtual machine have invoked operating system API (Appl ication Programming Interface, application programming interface）Its window is preposition, the corresponding application records of the preposition window can be current focus application by the window manager of operating system, or due to the operation of user, operating system have selected some application program as focus application, and the application records are focus application by window manager.

S604:Using monitoring module monitors operating system, the focus application in window manager is obtained；If using having have recorded focus application before monitoring modular, can contrast and finding whether current focus application is altered, if altered, using the information of the focus application of monitoring modular more new record.

The rapid S604 of Walk can with specific implementation it is similar with the S504 in above-mentioned Fig. 5, will not be repeated here.

S605:Using monitoring modular by the information of the focus application of current record（Such as process title）Message is packaged into, gesture translation is sent to, gesture translation preserves the information of the focus application.

Certain Ci Walk are alternatively Ke Xuan Walk suddenly suddenly, using monitoring modular after newest focus application is got, and can simply preserve the information of current focus application, do not perform S605 by Jiao The information of point application program is sent to gesture translation, but is sent to gesture translation after the inquiry request of subsequently received gesture translation, then by the information of focus application.

S606:After the information of the focus application is received, gesture translation can also first determine the corresponding switching strategy set of the focus application according to the information inquiry switching strategy of the focus application.

After corresponding switching strategy set is determined, translative mode can be changed so that be successfully received after the gesture operation of user, directly be changed under corresponding translative mode, saved conversion time delay, improve efficiency.The rapid S606 of Walk are similarly Ke and select Zhi Hang Walk rapid, if switching strategy contains multiple switching strategy set, a kind of translative mode of each switching strategy set correspondence, S606 can then be performed, if what switching strategy was included is the corresponding relation of the type three of the information of application program, the type of gesture operation and mouse-keyboard operation, S606 is without performing.

S607:User's operation trapping module of client captures the gesture operation of user customer equipment, and the information of the gesture operation of user is connected to the key mouse module for being transmitted to virtual machine by remote session.

S608:The information of gesture operation is sent to gesture translation by key mouse module, asks the conversion of gesture operation.

S609:Gesture translation is received after the information of the gesture operation of user, determines the type of the gesture operation, then inquires about the switching strategy set that current home is determined in the rapid S606 of the gesture operation Xin breath Cha Xun Walk（Or determine current translative mode）, the type of the corresponding current mouse keyboard operation of type of the gesture operation is determined, further according to the type of the current mouse keyboard operation of determination, the information of the gesture operation is converted to the information of keyboard and mouse.If not finding the type of corresponding mouse-keyboard operation, the type of selection acquiescence mouse-keyboard operation.

If Shang rapid S606 of Shu Walk are not carried out, then gesture translation is received after the gesture operation message of user, determine the type of the gesture operation, according to the type queries switching strategy set of the focus application and the gesture operation, determine the type of the corresponding current mouse keyboard operation of type of gesture operation, further according to the type of the current mouse keyboard operation of determination, the information of the gesture operation is converted to the information of keyboard and mouse.If not finding corresponding switching strategy, selection acquiescence switching strategy. S610:The information for the mouse-keyboard being converted to is returned to key mouse module by gesture translation. S611 :Mouse-keyboard message is converted into the operating system that mouse-keyboard event feeds back to virtual machine by key mouse module.Above-mentioned two flow embodiment, due to being prefixed in advance comprising application program, the switching strategy set of corresponding relation between the type three of type and the keyboard and mouse operation of gesture operation, and the information for the focus application for monitoring user's current operation in real time using monitoring modular is also set up in virtual machine, allow after gesture translation gets the gesture operation of user, it is determined that current focus application, and gone to match switching strategy set according to the type of focus application and gesture operation, so as to the automatic mouse-keyboard information that the gesture operation of user is converted to adaptation current application scene, this method causes user in different operation scenarios without specifying gesture operation scheme, the gesture that can be adapted to using the operation scenario, improve Consumer's Experience and system effectiveness.With reference to the flow embodiment that above-mentioned two is complete, the method provided in an embodiment of the present invention for realizing gesture conversion may be summarized to be：

(1) realized by client：

Receive the information of the gesture operation of user；

Obtain the information of the focus application of user's current operation；According to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

(2) realized by server：

After the remote session connection of foundation and client, the information of the gesture operation for the user that the client is sent is received；

According to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the different gestures behaviour under every kind of application program Make the type of the corresponding keyboard and mouse operation of type.

In addition, above-mentioned Fig. 2 and Fig. 3, application scenarios shown in Fig. 5 and Fig. 6 are all the scene of remote desktop application, and in fact, the present invention can also be applied to the scene that not remote is applied, such as, client unit performs business, now, the user described in above-described embodiment operates trapping module, is all arranged on client with using monitoring modular and gesture translation, and those skilled in the art can realize the method that the embodiment of the present invention is provided on the client.

It is the structure composition schematic diagram of the computer of the embodiment of the present invention such as Fig. 7.The computer of the embodiment of the present invention may include：

Processor 701, memory 702, system bus 704 and communication interface 705.Connected between CPU70K memories 702 and communication interface 705 by system bus 704 and complete mutual communication.

Processor 701 may be monokaryon or multinuclear CPU, be either specific collection into circuit or assemble circuit to be configured to implement the one or more of the embodiment of the present invention.

Memory 702 can be high-speed RAM memory, or nonvolatile memory

(non-volati le memory), for example, at least one magnetic disk storage.

Memory 702 is used for computer executed instructions 703.Specifically, program code can be included in computer executed instructions 703.

When computer is run, the operation computer executed instructions 703 of processor 701 can perform the method that any one embodiment of the embodiment of the present invention is provided.

It will be recognized by those of ordinary skill in the art that the possibility implementation of various aspects of the invention or various aspects can be embodied as system, method or computer program product.Therefore, the possibility implementation of each aspect of the present invention or various aspects can use complete hardware embodiment, complete software embodiment（Including firmware, resident software etc.）, or the embodiment of integration software and hardware aspect form, collectively referred to herein as " circuit ", " module " or " system ".In addition, the possibility implementation of each aspect of the present invention or various aspects can use the form of computer program product, computer program product refers to be stored in the computer readable program code in computer-readable medium.

Computer-readable medium can be computer-readable signal media or computer-readable recording medium. Computer-readable recording medium is including but not limited to electronics, magnetic, optics, electromagnetism, infrared or semiconductor system, equipment either device or foregoing any appropriately combined, such as random access memory（RAM), read-only storage（R0M), Erasable Programmable Read Only Memory EPROM (EPR0M or flash memory）, optical fiber, portable read-only storage (CD-ROM).

Processor in computer reads the computer readable program code being stored in computer-readable medium so that processor is able to carry out that Mei Walk in flow charts be rapid, function action specified in the combinations of the Ge Walk of Huo suddenly；The device in function action specified in each piece of block diagram or each piece of combination is implemented in generation.

Computer readable program code can perform completely on the computer of user, partly perform on the computer of user, as single software kit, partly on the computer of user and part is performed on remote computer or server on the remote computer, or completely.It is also noted that in some alternate embodiments, the function that Ge Walk is suddenly in flow charts or each piece indicates in block diagram may not be occurred by the order indicated in figure.For example, dependent on involved function, the Liang Walk shown in succession is rapid or two blocks may be actually executed substantially concurrently, or these blocks may sometimes be performed with reverse order.

Those of ordinary skill in the art are it is to be appreciated that the unit and Suan Fa Walk of each example described with reference to the embodiments described herein suddenly, can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Professional and technical personnel can realize described function to each specific application using distinct methods, but this realization is it is not considered that beyond the scope of this invention.

It is described above; only embodiment of the invention, but protection scope of the present invention is not limited thereto, any one skilled in the art the invention discloses technical scope in; change or replacement can be readily occurred in, should be all included within the scope of the present invention.Therefore, protection scope of the present invention described should be defined by scope of the claims.

Claims

Claim

1st, a kind of conversion method of gesture operation, it is characterised in that including：

Receive the information of the gesture operation of user；

Obtain the information of the focus application of user's current operation；

According to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

2nd, according to the method described in claim 1, it is characterised in that also include：

Obtain switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；

Then the information according to the focus application, mouse-keyboard information is converted to by the gesture operation information, including：

Determine the type of the gesture operation；

The switching strategy set according to the type queries of the information of the focus application and the gesture operation, determines the type of current mouse keyboard operation；

According to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

3rd, according to the method described in claim 1, it is characterised in that also include：

Obtain multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding then information according to the focus application of type of each switching strategy set comprising gesture operation and type that mouse-keyboard is operated, the gesture operation information is converted into mouse-keyboard information, including：

According to the information of the focus application, the switching strategy set of current home is determined；Determine the type of the gesture operation；

The switching strategy set of current home according to the type queries of the gesture operation, it is determined that currently The type of mouse-keyboard operation；

According to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

4th, the method according to claim any one of 1-3, it is characterised in that before the gesture operation information of the reception user, in addition to：

Foundation is connected with the remote session of server；

The information by the gesture operation is converted to after mouse-keyboard information, in addition to：The mouse-keyboard information is passed through into the remote session linkup transmit to the server.

5th, method according to claim 4, it is characterised in that the information of the focus application of the acquisition user current operation includes：

Inquiry request is sent to the server, and obtains the information that the server passes through the focus application of the remote session linkup transmit；Or,

Obtain the information for the focus application that the server itself preserved passes through remote session linkup transmit.

6th, the method according to claim any one of 1-3, it is characterised in that also include：Monitor the information of the focus application of user's current operation；

The information of the focus application for obtaining user's current operation includes：

Obtain the information of the focus application for user's current operation that monitoring is obtained.

7th, method according to claim 6, it is characterised in that the information of the focus application of monitoring user's current operation, including：

The information of the preposition window of user's current operation described in timing probe, the process title of the focus application is determined according to the information of the preposition window；Or,

Installed in advance in each application program of operating system and apply Monitoring instruction, when the focus application by it is preposition when, so that the operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, the process title of the focus application is obtained and preserved from the operating system. 8th, a kind of conversion method of gesture operation, it is characterised in that including：

After the remote session connection of foundation and client, the information of the gesture operation for the user that the client is sent is received；

Determine the information of the focus application of user's current operation；

According to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

9th, method according to claim 8, it is characterised in that also include：

Obtain switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；

Then the information according to the focus application, mouse-keyboard information is converted to by the gesture operation information, including：

Determine the type of the gesture operation；

The switching strategy set according to the type queries of the information of the focus application and the gesture operation, determines the type of current mouse keyboard operation；

According to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

10th, method according to claim 8, it is characterised in that also include：

Obtain multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding then information according to the focus application of type of each switching strategy set comprising gesture operation and type that mouse-keyboard is operated, the gesture operation information is converted into mouse-keyboard information, including：

According to the information of the focus application, the switching strategy set of current home is determined；Determine the type of the gesture operation；

The switching strategy set of current home according to the type queries of the gesture operation, it is determined that currently The type of mouse-keyboard operation；

According to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

11st, the method according to claim any one of 8-10, it is characterised in that the information by the gesture operation is converted to after mouse-keyboard information, in addition to：

The mouse-keyboard information is converted into keyboard and mouse event, the keyboard and mouse event is notified to operating system.

12nd, the method according to claim any one of 8-11, it is characterised in that also include：Monitor the information of the focus application of user's current operation；

The information of the focus application for obtaining user's current operation includes：.

Obtain the information of the focus application for user's current operation that monitoring is obtained.

13rd, method according to claim 12, it is characterised in that the information of the focus application of monitoring user's current operation, including：

The information of the preposition window of user's current operation described in timing probe, the process title of the focus application is determined according to the information of the preposition window；Or,

Installed in advance in each application program of operating system and apply Monitoring instruction, when the focus application by it is preposition when, so that the operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, the process title of the focus application is obtained and preserved from the operating system.

14th, a kind of client device, it is characterised in that for the conversion for the gesture operation for realizing user, including：

User operates trapping module, the information of the gesture operation for capturing user, and the information of the gesture operation is sent into gesture translation request gesture conversion；

Gesture translation, the information of the focus application for obtaining user's current operation；Gesture translation, is additionally operable to according to the preset switching strategy of the information matches of the focus application, and the information of the gesture operation is converted into mouse-keyboard information, and the switching strategy indicates every kind of answer The type operated with the corresponding keyboard and mouse of different gesture operation types under program.

15th, client device according to claim 14, it is characterized in that, the gesture operation modular converter is specifically for obtaining switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；Determine the type of the gesture operation, the switching strategy set that the first policy module according to the user operates the information of the focus application and the type queries of the gesture operation of trapping module acquisition is obtained, determine the type of current mouse keyboard operation, according to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

16th, client device according to claim 14, it is characterized in that, the gesture translation, specifically for obtaining multiple switching strategy set, wherein, the different application program of each switching strategy set correspondence, the corresponding relation for the type that type of each switching strategy set comprising gesture operation is operated with mouse-keyboard；According to the information of the focus application, determine the switching strategy set of current home, determine the type of the gesture operation, the switching strategy set of current home described in the type queries of the gesture operation obtained according to user operation trapping module, determine the type of current mouse keyboard operation, according to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

17th, the client device according to claim any one of 14-16, it is characterized in that, user's operation trapping module is additionally operable to set up to be connected with the remote session of server, the information for the mouse-keyboard being converted to is obtained from the gesture translation, and the mouse-keyboard information is passed through into the remote session linkup transmit to the server.

18th, client device according to claim 17, it is characterized in that, the gesture translation to the server specifically for sending inquiry request, and the information that the acquisition server passes through the focus application of the remote session linkup transmit；Or, the information for the focus application that the server for obtaining itself preservation passes through remote session linkup transmit.

19th, the client device according to claim any one of 14-16, it is characterised in that also include： Using monitoring modular, the information of the focus application for monitoring user's current operation；The information of the focus application for user's current operation that the gesture translation is obtained specifically for acquisition monitoring.

20th, client device according to claim 19, it is characterized in that, the application monitoring modular, specifically for the information of the preposition window of user's current operation described in timing probe, the process title of the focus application is determined according to the information of the preposition window；Or, installed in advance in each application program of operating system and apply Monitoring instruction, when the focus application by it is preposition when, so that the operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, the process title of the focus application is obtained and preserved from the operating system.

21st, a kind of server, it is characterised in that for realizing the conversion that the user gesture of client is operated, including：

Key mouse module, for after the connection of the remote session of foundation and client, receiving the information of the gesture operation for the user that the client is sent；

Gesture translation, the information of the focus application for determining user's current operation；Gesture translation, it is additionally operable to according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

22nd, server according to claim 21, it is characterized in that, the gesture translation, specifically for obtaining switching strategy set, the corresponding relation of the type of information of the switching strategy set comprising application program, the type of gesture operation and mouse-keyboard operation；Determine the type of the gesture operation, the switching strategy set that the first policy module according to the user operates the information of the focus application and the type queries of the gesture operation of trapping module acquisition is obtained, determine the type of current mouse keyboard operation, according to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

23rd, server according to claim 21, it is characterised in that the gesture translation, specifically for obtaining multiple switching strategy set, wherein, the different application of each switching strategy set correspondence Program, the corresponding relation for the type that type of each switching strategy set comprising gesture operation is operated with mouse-keyboard；According to the information of the focus application, determine the switching strategy set of current home, determine the type of the gesture operation, the switching strategy set of current home described in the type queries of the gesture operation obtained according to user operation trapping module, determine the type of current mouse keyboard operation, according to the type of the current mouse keyboard operation, the information of the gesture operation is converted to the information of keyboard and mouse.

24th, the server according to claim any one of 21-23, it is characterised in that the mouse-keyboard information that the gesture translation is additionally operable to be converted to is sent to the key mouse module；

The key mouse module, is additionally operable to the mouse-keyboard information being converted to keyboard and mouse event, the keyboard and mouse event is notified to operating system.

25th, the server according to claim any one of 21-24, it is characterised in that also including applying monitoring modular, the information of the focus application for monitoring user's current operation；The information of the focus application for user's current operation that the gesture translation is obtained specifically for acquisition monitoring.

26th, server according to claim 25, it is characterized in that, the application monitoring modular, specifically for the information of the preposition window of user's current operation described in timing probe, the process title of the focus application is determined according to the information of the preposition window；Or, installed in advance in each application program of operating system and apply Monitoring instruction, when the focus application by it is preposition when, so that the operating system obtains the process title for the focus application being triggered according to the application Monitoring instruction, the process title of the focus application is obtained and preserved from the operating system.

27th, a kind of remote session system, it is characterised in that for the conversion for the gesture operation for realizing user, including：

Client device, it is connected for setting up with the remote session of server, receive the information of the gesture operation of user, obtain the information of the focus application of user's current operation, according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the corresponding keyboard and mouse behaviour of different gesture operation types under every kind of application program The type of work, and the mouse-keyboard information is passed through into the remote session linkup transmit to the server；

Server, the information of the focus application for monitoring user's current operation, and the information for monitoring the obtained focus application is sent to the client device.

28th, a kind of remote session system, it is characterised in that for the conversion for the gesture operation for realizing user, including：

Client device, for set up be connected with the remote session of server, capture user gesture operation information, and by the information of the gesture operation by the remote session linkup transmit to server；Server, for the information for the gesture operation for receiving the user that the client is sent, determine the information of the focus application of user's current operation, according to the preset switching strategy of the information matches of the focus application, the information of the gesture operation is converted into mouse-keyboard information, the switching strategy indicates the type of the corresponding keyboard and mouse operation of different gesture operation types under every kind of application program.

29th, a kind of computer, it is characterised in that including：Processor, memory, bus and communication connect

P ;

The memory is used to store computer executed instructions, the processor is connected with the memory by the bus, when the computer is run, the computer executed instructions of memory storage described in the computing device, so that the computer performs the method as described in claim 1-13 any one.

30th, a kind of computer-readable medium, it is characterised in that including computer executed instructions, when computer executed instructions described in the computing device of computer, the computer performs the method as described in claim 1-13 any one.