JP6607876B2 - Coordinating input method editor (IME) activity between virtual application client and server - Google Patents

Description

  The present invention relates to a method for coordinating input method editor (IME) activity between a virtual application client and a server.

  Operating systems typically support IME (Input Method Editor) to assist in inputting human language characters into software applications. For example, IME is provided for inputting traditional Chinese, Japanese, and Korean characters, for which no corresponding keyboard button is provided. In a typical configuration, an IME uses IME elements such as composite windows and candidate windows to support character input into a running application, such as a word processing document, spreadsheet, web browser, etc. Runs as a helper application to present.

  For example, the user's keyboard layout can support Simplified Chinese (eg, Pinyin), Japanese, or Korean. In response to the user typing a keyboard button, the IME generates a composite window. The composite window includes composite characters that are displayed according to display attributes. A composite character is a temporary character that may or may not be entered into the application. The display attribute of the composite character specifies how the composite character is displayed in order to emphasize the composite character, distinguish the composite character from the already determined characters, and identify the status of the composite character. In some examples, the composite character simply reflects direct keyboard input (eg, as in Microsoft Pinyin IME). In other examples, the composite character is a complex character that the IME has converted from keyboard input to the target language (eg, as in Google Pinyin IME).

  At the same time that the IME generates a composite window, the IME also generates a candidate window. The candidate window includes a candidate list, ie, a list of predictions for one or more characters in the target language corresponding to keyboard input. For example, the user can enter a keyboard symbol that is pronounced in a manner similar to how it is pronounced when characters in the target language are spoken. As the user types keyboard input to the application, the IME dynamically updates and refines the composite character and candidate list based on further user input. The user can, for example, click on an item on the candidate list, use the arrow keys to click “Enter” or “Spacebar”, or enter a number that identifies the item on the candidate list Can select one or more characters from the candidate list. When the user selects a candidate from the candidate list, the IME inserts the selected character or characters into the application (eg, into a document, cell, input field, etc.).

  For so-called “partially IME-aware” applications such as Microsoft Word, Microsoft Notepad, and Microsoft Wordpad, the application is responsible for displaying composite elements, but not for displaying candidate windows. The IME generates composite elements based on user input and provides these elements to the application. The application then displays the composite element at the current cursor position. At the same time, the IME displays a candidate window next to the composite element. For so-called “fully IME aware” applications, such as Microsoft Command, the application is responsible for displaying both composite and candidate list elements. In these cases, the IME generates these elements and provides these elements to the application, which displays these elements according to its own programming.

  A user may wish to adopt IME not only for inputting characters to an application running locally, but also for inputting characters to a virtual application. As is known, a “virtual application” is an application that is hosted from a remote server but presents a user interface locally on the client machine. Virtual applications can be provided in the context of a virtual desktop or in an application window without a desktop context. Virtual applications can have the same look and feel as applications that run locally on the client machine, whether or not they are served in a virtual desktop. However, behind the scenes, data is exchanged between the client and server to maintain a local look and feel.

  Unfortunately, it has proven difficult to retain the impression of local execution if the user wishes to adopt IME to enter language characters into a virtual application. For example, the user may wish to invoke the local client's IME to enter characters into the virtual application, but there is no structure that allows communication of IME operations between the client and server. There is. As a result, some IME functions may not work for virtual applications in the same way that they work for locally running applications, resulting in a loss of user experience. Some servers of the virtual application provide their own IMEs that can be operated remotely by the user. However, such an IME may be different from the user's preferred IME. Therefore, the user experience is impaired even in these situations.

  In contrast to traditional approaches, an improved technique for inputting human language characters into a virtual application is between a client IME running on a client machine and a server IME running on a server. Establishing a communication channel and allowing the client IME to interact with a virtual application running on the server using the server IME as a proxy. The server IME relays information from the client IME to the virtual application. The server IME also relays information from the virtual application to the client IME. Thus, the server IME allows the client IME to interact with the virtual application as if interacting with the local application, facilitating a similar user experience for both the virtual application and the local application. Users can continue to adopt their preferred client IMEs, and such IMEs behave in the same way as when used with local applications when used with virtual applications.

  In one example, the server IME relays the composite element generated by the client IME to the virtual application so that the virtual application can display the composite window in the same manner that the local application displays.

  In another example, the server IME obtains the composition window coordinates or cursor coordinates from the virtual application and provides these coordinates to the client IME, so that the client IME can place a candidate list next to the composition window or cursor. Can be displayed.

  In another example, the server IME obtains information from the virtual application indicating whether the candidate list should be displayed by the virtual application or the client IME. The server IME sends this information to the client IME so that the client IME can display a candidate list on its own or send a candidate window element to the server so that the virtual application can Can be displayed.

  Certain embodiments are directed to a method of inputting human language characters into a virtual application. This method is a step of establishing a communication channel between a client IME (Input Method Editor) running on a client machine and a server IME running on a server, wherein the communication channel is a server IME. Enabling a user to interact with the virtual application via The virtual application runs on the server and provides screen output, which is rendered on the client machine. The method relays IME data received from a client IME via a communication channel to a virtual application by a server IME and provides the virtual application with the IME data, wherein the IME data is sent to the client machine. It further includes a step generated by the client IME in response to user input. The method also includes relaying application information received from the virtual application by the server IME to the client IME via the communication channel by the server IME to provide the application information to the client IME.

  Other embodiments are directed to computer program products. The computer program product is a non-transitory computer readable medium having instructions that, when executed by a control circuit, cause the control circuit to execute a method of inputting human language characters into a virtual application, such as the method described above. Includes set.

  Still further embodiments are directed to server devices that are constructed and configured to perform a method of inputting human language characters into a virtual application, such as the method described above. Some embodiments include activities performed at a single location, while other embodiments include activities distributed over a computerized environment (eg, over a network). .

  The foregoing summary has been presented for purposes of illustration and is intended to be limiting in order to assist the reader in easily understanding the illustrative features presented herein. is not.

  The foregoing and other features and advantages will become apparent from the following description of specific embodiments of the invention as illustrated in the accompanying drawings. In the drawings, like reference characters refer to the same or similar parts throughout the different views.

1 is a block diagram of an exemplary environment in which embodiments of the improved techniques herein can be practiced. FIG. FIG. 2 is a screenshot showing exemplary composite elements generated by the client IME of FIG. 1, which follow the current cursor position in the virtual application. FIG. 4 is a block diagram illustrating example communications for enabling a client IME to interact with a virtual application. FIG. 6 is a screenshot showing an exemplary synchronization of IME compartments for a client IME and a server IME. 2 is a flow chart illustrating an exemplary sequence of activities for initializing and operating a client IME in the environment of FIG. 2 is a flowchart of an exemplary method for inputting human language symbols into a virtual application.

  An embodiment of the present invention will be described. Such embodiments are provided by way of example to illustrate various features and principles of the invention, and the invention herein is broader than the specific exemplary embodiments disclosed. .

  An improved technique for inputting human language characters into a virtual application is to establish a communication channel between a client IME running on a client machine and a server IME running on a server, and the client IME Includes interacting with a virtual application running on the server using the server IME as a proxy. Thus, the server IME allows the client IME to interact with the virtual application as if interacting with the local application, facilitating a similar user experience for both the virtual application and the local application.

  FIG. 1 illustrates an exemplary environment 100 in which the improved technique embodiments herein may be practiced. Here, a client machine (“client”) 110 is connected to a server device (“server”) 120 via a network 130. Network 130 may be any type of network or combination of networks, such as a local area network (LAN), a wide area network (WAN), the Internet, and / or some other type of network or combination of networks. It is possible that Server 120 comprises virtualization software, which allows server 120 to run application instances (“applications”) internally, while screen sharing and / or other, for example, Via means, the user interface of those applications can be rendered on a client machine, such as client machine 110. The virtualization software on the server 120 can support application virtualization and / or full desktop virtualization, and the improved techniques disclosed herein are intended to apply in both cases. ing.

  As is known, “application virtualization” refers to a configuration in which a server runs an application instance while exporting the application's user interface to a client. For example, a virtual application running on the server appears in an application window on the client desktop, and the user can use the virtual application on the client in the application window, for example, using the user's keyboard, mouse, etc. You can interact with.

  In contrast, “desktop virtualization” refers to a configuration in which a server completes its own desktop and applications and runs the entire user session on behalf of the user. The server exports the virtual desktop to the user's machine, which displays the virtual desktop on the entire screen or in a window on the machine's own desktop. A user can start applications in the virtual desktop and interact with these applications, and can use the virtual desktop in the same way as using the native desktop.

  FIG. 1 shows an exemplary implementation of client 110 and server 120 in more detail. These examples are simplified and are intended to be illustrative rather than limiting.

  As shown, server 120 includes one or more network interfaces 160, a set of processors 162, and memory 170. The network interface 160 is, for example, one or more network interface adapters for converting electronic and / or optical signals received via the network 130 into an electronic form for use by the server 120. including. The set of processors 162 includes one or more processing chips, cores, and / or assemblies, and associated coprocessors and chipsets. Memory 170 includes both volatile memory (eg, RAM) and non-volatile memory (eg, one or more ROMs, disk drives, solid state drives, etc.). The set of processors 162 and memory 170 together form a control circuit, which is constructed and configured to perform the various methods and functions described herein. The memory 170 also includes various software structures implemented in the form of executable instructions. When executable instructions are executed by the set of processors 162, the set of processors 162 is caused to perform the operations of the software structure. Although specific software structures are specifically shown and described, memory 170 typically includes many other software structures not shown, such as an operating system, various applications, processes, and daemons. Understand. Although the server 120 is shown as a single machine, the server 120 may alternatively be a plurality of, eg, a data center, server farm, or bank of machines anywhere in the “cloud”. It may be implemented as a machine and / or as one or more virtual machines. Server 120 may be implemented as a mobile device, such as a smartphone, tablet, etc., but this configuration is not expected to be very common.

  Within memory 170, server 120 “includes” server virtualization component 172, ie, is implemented by execution of software instructions. Server virtualization component 172 includes server agent 174, server IME 176, and virtual application 178. Server IME 176 and virtual application 178 may alternatively be provided external to server virtualization component 172, for example, as an independent component in memory 170 or as part of other components. The server agent 174 is arranged in an operable communication state with the server IME 176 via the interface 176a. The server IME 176 is arranged in an operable communication state with the virtual application 178 via the interface 178a. Interfaces 176a and 178a may be implemented using, for example, an application program interface (API), a system message, or some other form of one or more forms.

  Server agent 144 on server 120 is configured to communicate with client agent 144 on client 110. Server agent 174 and client agent 144 establish a communication channel 180 between server 120 and client 110. Channel 180 is physically implemented via network 130 and provides a real-time (or near real-time) means for exchanging user input related information between server 120 and client 110. The channel 180 can be established based on actions by either the server 120 or the client 110 and is a “virtual session”, ie, an application virtualization session or a desktop virtualization session (regardless of which is true). ) For a duration of).

  Although FIG. 1 shows a single virtual application 178, the server 120 can host any number of virtual applications. Also, although the virtual desktop is not shown, the virtual application 178 shown can be run within the virtual desktop. Multiple virtual desktops can be provided for each different user, for example, and any number of virtual applications can be run within each virtual desktop. In one example, one server agent 174 is provided for each virtual session. However, one server IME 176 can be provided for each virtual application, and each virtual application 178 has its own server IME 176. Thus, for example, multiple virtual applications 178 in a single virtual session can share the same server agent 174 instance while each having its own server IME instance.

  In the illustrated example, client machine 110 includes a network interface 130, a processor 132, and memory 140. The set of processors 132 and memory 140 together form a control circuit, which is constructed and configured to perform the various methods and functions described. Client machine 110 also includes a display 150 (eg, a monitor, screen, touch screen, television, etc.). Memory 140 includes various software structures implemented in the form of executable instructions. When these executable instructions are executed by the set of processors 142, the set of processors 142 is caused to perform the operations of these software structures. Although client machine 110 is shown as a computer, client machine 110 can be a wide range of machines, devices, and appliances such as desktop computers, laptop computers, smartphones, tablets, personal readers, set-top devices It can be implemented using boxes, video streaming devices, smart TVs, gaming systems, smart appliances, etc., and combinations thereof.

  Client 110 includes a client virtualization component 142 and a client IME 146. Client virtualization component 142 includes a client agent 144. The client agent 144 is arranged in an operable communication state with the client IME 146 via the interface 146a. Interface 146a may be implemented using, for example, an API, system message, or some other form of interface.

  The client virtualization component 142 establishes and maintains a communication session with the server 120, displays virtual application and / or virtualized desktop screen content received from the server 120, and Client support for application virtualization and / or desktop virtualization is provided, such as by directing user input to the server 120 from, for example, a keyboard, mouse, etc. (or equivalent software-implemented function).

  The client IME 146 running on the client 110 facilitates user input of human language symbols such as non-phonetic characters that are not suitable for direct keyboard input. The user can install the client IME 146 based on the user's own preferences. Although only a single client IME 146 is shown for brevity, client machine 110 may run any number of client IMEs, eg, for different target languages and / or from different vendors. And the user can switch between different client IMEs during a virtual session.

  In an exemplary operation, a user 112 of a client machine 110 initiates a virtual session, for example, by launching a program or navigating to a specific web address using a browser. In response to this action by the user 112, the server 120 instantiates a virtual session, for example, by opening a virtual application 178 in the container and / or creating a new desktop session. The client virtualization component 142 coordinates with the server virtualization component 172 to establish a channel 180 between the client agent 144 and the server agent 174. The server 120 exports the virtual application 178 or the entire virtual desktop user interface to the client 110, which displays the virtual output of the virtual application 178 or virtual desktop on the display 150.

  With the channel 180 established and the virtual application 178 running, the user 112 can subsequently launch the client IME 146. For example, the user 112 may wish to use a word processor or email program to enter traditional Chinese characters into a document using a regular QUERYTY-style keyboard.

  When the user operates the client IME 146, the client IME 146 generates IME data 182. The IME data 182 flows to the client agent 144 via the interface 146a and is transferred to the server agent 174 via the channel 180. Within server 120, IME data 182 flows to server IME 176, which provides IME data 182 to virtual application 178. Similarly, when virtual application 178 operates, virtual application 178 generates application data 184 related to the IME operation. Server IME 176 receives application data 184 and returns application data 184 to client IME 146 via channel 180. Thus, server IME 176 operates as a proxy or relay for client IME 146 to transfer IME data 182 to virtual application 178 and return application data 184 to client IME 146. For example, when the client IME 146 operates to generate UI elements, such as composite elements and candidate lists, in response to user input, the server IME 176 interacts with the virtual application 178 on behalf of the client IME 146 to Data 182 is provided to virtual application 178 and application data 184 is returned to client IME 146. Thus, agents 144 and 174, server IME 176, and channel 180 allow client IME 146 to interact remotely with virtual application 178 as if interacting with a local application.

  FIG. 2 is a screenshot 200 illustrating an exemplary operation of the client IME 146 for entering characters into the virtual application 178 in accordance with the configuration of FIG. Here, the client IME 146 is a Google Pinyin IME, but any IME may be used. A virtual desktop 210 is shown in a window 212 on the native desktop 250 of the client machine 110. Virtual desktop 210 includes a virtual system tray 214 and native desktop 250 includes a native system tray 254.

  The virtual desktop 210 operates an instance of Microsoft Notepad as a virtual application 178. Notepad appears to be running in the application window 220 on the virtual desktop 210. Although only Notepad appears to be running, any number of virtual applications can run on the virtual desktop 210.

  Multiple IME elements can be seen in FIG. For example, composite window 154 includes composite character 154a being rendered with display attributes 154b (eg, underlined as shown, but may also include bold, color, and other settings). . The candidate window 156 includes a candidate list 156a. A user provides keyboard input (shown below the compositing window 154), and the client IME 146 responds by generating a composite character 154a and a candidate list 156a. Candidate list 156a provides a best match prediction (numbered in the figure) for keyboard input in the target language. The user can then operate the client IME 146 to select a desired candidate, i.e., a candidate that the user intends to input. When the user makes a selection, for example by entering a number, the selected character becomes a fixed character and is entered into the application document. A number of confirmed characters 152 are shown, and these characters 152 have already been entered by the user using the client IME 146. In the example shown, the composite character 154 matches the current selection in the candidate list 156a.

  FIG. 2 also shows a cursor 158. The cursor 158 appears immediately after the compositing window 154 and moves forward as new characters are entered or as different width characters are selected.

  The configuration of FIG. 1, agent 144, agent 174, channel 180, client IME 146, and server IME 176, inputs characters to the local application when client IME 146 inputs characters to virtual application 178. Allows you to work exactly as you do. For example, the configuration of FIG. 1 allows the virtual application 178 to render the composite window 154 in the application window 220, i.e., by sending the IME data 182 to the server IME 176, where the server IME 176 178 to communicate. Virtual application 178 displays composite window 154 based on input from client IME 146 just as a local application displays. This configuration also allows the client IME 146 to place the candidate window 156 next to the composite window 154 or next to the cursor 158 if the composite window is empty. For example, the client IME 146 may request screen coordinates for the composite window 154 or cursor 158 and receive these screen coordinates from the virtual application 178 as application data 184. The client IME 146 can then apply the received coordinates in placing the candidate window 156. Furthermore, the client IME 146 determines whether the virtual application 178 can display the candidate list on its own, that is, whether the virtual application 178 is partially IME aware or fully IME aware. The application data 184 indicating that can be obtained from the virtual application 178. Based on the received application data 184, the client IME 146 may subsequently display the candidate window 156 on its own (for applications that partially recognize IME, such as Notepad), or (such as Command). In addition, for applications that fully recognize IME), send candidate list 156a and associated settings (via channel 180) to virtual application 178 so that virtual application 178 displays the candidate list on its own. Can be made possible.

  FIG. 3 shows exemplary communication between the client IME 146 and the virtual application 178 via the channel 180. This communication allows the client IME 146 to operate with the virtual application 178 as if it operates with a local application.

  Here, in response to a user action, for example, in response to a user entering a first keyboard symbol into the client IME 146, a composite start message 310 is generated as IME data 182 by the client IME 146. Composite start message 310 propagates from client IME 146 to client agent 144, to server agent 174 via channel 180, and to server IME 176. Server IME 176 provides a composite start message (or a processed version of the composite start message) to virtual application 178. In response, the virtual application 178 prepares to start the synthesis procedure.

  The composite element 312 includes a composite character 154a and a display attribute 154b. The client IME 146 generates these composite elements 312 in response to user input and sends these composite elements 312 to the server IME 176 via the channel 180, and the server IME 176 sends these composite elements 312 to the virtual application 178. To provide. Virtual application 178 then displays a composite window 154 using the received composite element.

  The start request 320 is a request to start a candidate window for identifying character predictions based on the set of composite characters in the composite window. Startup request 320 is received by server IME 176, which forwards the request to virtual application 178.

  Application 178 answers start request 320 using start response 322, and application 178 returns start response 322 to client IME 146 via channel 180. Startup response 322 indicates whether virtual application 178 displays candidate window 156 or whether client IME 146 displays candidate window 156. For example, the start response 322 is “FALSE” when the virtual application 178 is in charge of displaying the candidate window 156, and “TRUE” when the client IME 146 is in charge of displaying the candidate window 156. It is possible.

  If the start response 322 is “FALSE”, the client IME 146 sends a candidate list element 330 (eg, candidate list character, page index, current page, etc.) to the server IME 176, and the server IME 176 Element 330 is provided to virtual application 178. Virtual application 178 then displays candidate window 156 in the screen output of virtual application 178 using candidate list element 330. Note that this is typically the case for applications that are fully IME aware.

  However, if the start response 322 is “TRUE”, the client IME 146 then displays the candidate window 156 locally on the display 150. Note that this is typically the case for applications that are partially IME aware.

  In the situation where the client IME 146 displays the candidate window 156, the configuration of FIG. 1 places the client IME 146 in the appropriate location, that is, in the same location where the candidate window 156 is located for the local application. Make it possible to do. For example, the client IME 146 sends a coordinate request 340 to the virtual application 178 via the channel 180. Virtual application 178 receives coordinate request 340 and provides coordinate response 342. Coordinate response 342 is returned to client IME 146 via channel 180. In one example, the coordinate response 342 provides a set of coordinates that indicate the extent area of the composite window 154 displayed by the virtual application 178. These coordinates can be represented as, for example, the vertices of a rectangle that includes the composite window 154. The client IME 146 then places the candidate window 156 relative to the composition window 154 in the same manner as it is for the local application, eg, just below the composition window 154 (as shown in FIG. 2). be able to. In another example, coordinate response 342 provides the coordinates of cursor 158. In such a case, the client IME 146 places the candidate window 156 relative to the location of the cursor 158. In some examples, the coordinate response 342 provides the extent area of the composite window 154 if the composite window 154 includes the composite character 154a, but the coordinates of the cursor 158 if the composite window 154 is empty. I will provide a.

  The synthesis end message 350 triggers the end of synthesis. For example, the client IME 146 transmits the composition end message 350 after the user selects a candidate from the candidate list 156a. In response to the composition end message 350, the virtual application 178 places the confirmed character 152 in the application window 220 and clears the composition window 154. The synthesis procedure is then complete.

  The communication described above can be repeated any number of times as the user provides new keyboard input. For example, the client IME 146 may repeatedly send a new composite element 312 in response to the user continuing to type new keyboard input. Also, coordinate request 340 and response 342 may be repeated as composition window 154 changes size and / or as the user moves or resizes application window 220 and / or desktop window 212. is there. For example, client IME 146 can continually update the position of candidate window 156 so that candidate window 156 always follows composition window 154 or cursor 158.

  FIG. 4 shows an exemplary screenshot 400. This is similar to the screen shot 200 of FIG. 2, but provides a closer view of the system trays 214 and 254. FIG. 4 also shows the status window 402 of the client IME 146. In the native desktop 250, the system tray 254 includes a keyboard language setting 440 that identifies the human language that the client machine 110 applies when interpreting keyboard data. In the virtual desktop 210, the system tray 214 includes a keyboard language setting 430. The keyboard language 430 in the system tray 214 identifies the keyboard language selection for the server 120, i.e., the human language that the server 120 applies in interpreting keyboard input.

  According to a further refinement of the specification, a change in the keyboard language setting 440 triggers the automatic synchronization of the keyboard language setting from the client 110 to the server 120. Synchronization can include activating a particular server IME 176 for input in a new keyboard language. Thus, for example, changing the keyboard setting 440 in the native desktop 450 from Japanese (JP) to simplified Chinese (zh-CN) will change the keyboard setting 430 and server IME 176 in the virtual desktop 210 from Japanese to simplified Chinese. Trigger automatic switching to Such synchronization ensures that IME data exchanged between client 110 and server 120 is interpreted in the same way by both client 110 and server 120.

  The system tray 214 in the virtual desktop 210 also includes options for changing the server compartment settings 410. The server compartment settings 410 are related to the settings of the server IME 176, and are related to the target language settings 410a used to specify the human language of the symbols input by the server IME 176, and the symbols in a particular language. And a width setting 410b used to specify whether these symbols are entered using half-width spacing or full-width spacing. Additional or other server compartment settings can be provided.

  The status window 402 includes options for changing the client compartment settings 420. The client compartment settings 420 relate to the settings of the client IME 146 and may also include a target language setting (420a) and a symbol interval setting (420b). Additional or other client compartment settings may be provided. Thus, it can be seen that the client compartment settings 420 control the same settings on the client side as the server compartment settings 410 control on the server side.

  According to a further improvement, a change in any of the client compartment settings 420 of the client IME 146 automatically synchronizes its settings with the corresponding server compartment settings of the server IME 176, and vice versa. Trigger automatic synchronization. Thus, for example, changing the client target language setting 420a in the status window 402 triggers an automatic change in the server target language setting 410a. Similarly, changing the server interval setting 410b in the system tray 214 triggers an automatic change in the client interval setting 420b in the status window 402. This synchronization ensures that the client IME 146 and the server IME 176 operate in the same target language with the same settings so that the data sent between the client IME 146 and the server IME 176 over the channel 180 is correctly interpreted. Make sure.

  FIG. 5 illustrates an exemplary phase of operations 500 involved in coordinating IME activity between client 110 and server 120 to support virtual applications. These phases include an opening phase 510, a combining phase 520, an activation phase 530, and a synthesis phase 540. These phases are typically performed in the order shown, but the activation phase 530 can be repeated if another client IME 146 is selected.

  During the opening phase 510, a virtual session is created. Client 110 instantiates client agent 144, and server 120 instantiates server agent 174. Client agent 144 and server agent 174 establish channel 180 to carry IME related information between client 110 and server 120.

  During the binding phase 520, the server 120 contacts the client 110 via the channel 180 and informs the client of its own capabilities, such as the languages that the server 120 supports. In response, the client 110 provides the functionality of the client 110, such as the language that the client 110 supports. In one example, the server 120 then prepares a language profile for each language that both the client and server support. The server then sends a join commit message to the client confirming the supported languages.

  The activation phase 530 starts when the user selects the client IME 146 and the target language. Client agent 144 communicates the selected target language to server agent 174 via channel 180. In response, server agent 174 activates server IME 176 (assuming that server IME 176 has not yet been activated) and causes server IME 176 to operate using the profile for the selected target language. Configure. If the current virtual session is running desktop virtualization and multiple virtual applications 178 are running, the server agent 174 is running all newly activated server IMEs 176. Applies to applications as a system default IME for virtual sessions. During this phase, the client agent 144 also sends the current client keyboard language (ie, as indicated by settings 440; see FIG. 4) to the server agent 174, and the server agent 174 sets the server keyboard language (ie, as indicated by setting 430) to be the same as the client keyboard language.

  The composition phase 540 follows the activation phase and is between the client agent 144 and the server agent 174 when the user of the client machine 110 is using the client IME 146 to enter language symbols. Holding communications over channel 180. The virtual session can remain in the synthesis phase 540 for the remaining duration of the virtual session or until the user changes the client IME 146, the target language, and / or the keyboard language, at which point the operation is It is possible to return to the activation phase 530 and then proceed to the synthesis phase 540 again when activation is complete.

  FIG. 6 shows an exemplary process 600 for inputting human language symbols into a virtual application. Process 600 may be performed, for example, in connection with environment 100, for example, by the software structures described in connection with FIG. 1, which are in memory 140 of client machine 110. And / or run by the set of processors 132 and / or run in the memory 170 of the server 120 and run by the set of processors 162. The various acts of process 600 can be ordered in any suitable manner. Thus, embodiments may be constructed in which actions are performed in a different order than the illustrated order, and these embodiments may perform several actions simultaneously (even if these actions are In the illustrated embodiment, even though they are shown as occurring sequentially).

  At 610, the method 600 includes establishing a communication channel 180 between a client IME (Input Method Editor) 146 running on the client machine 110 and a server IME 176 running on the server 120. Communication channel 180 allows client IME 146 to interact with virtual application 178 via server IME 176. Virtual application 178 runs on server 120 and provides screen output 220 that is rendered on client machine 110.

  At 620, method 600 includes relaying IME data 182 received from client IME 146 via communication channel 180 to virtual application 178 by server IME 176 and providing IME data 182 to virtual application 178. The IME data 182 is generated by the client IME 146 in response to user input to the client machine 110.

  At 630, method 600 further includes relaying application information 184 received from virtual application 178 by server IME 176 to client IME 146 via communication channel 180 by server IME 176 to provide application information 184 to client IME 146. .

  An improved technique for inputting human language characters into a virtual application has been described. This technique establishes a communication channel 180 between a client IME 146 running on a client machine 110 and a server IME 176 running on a server 120, which uses the server IME 176 as a proxy and the server 120. Including allowing to interact with a virtual application 178 running on top. The server IME 176 relays the information 182 from the client IME 146 to the virtual application 178. Server IME 176 also relays information 184 from virtual application 178 to client IME 146. Thus, the server IME 176 allows the client IME 146 to interact with the virtual application 178 as if it were interacting with the local application, facilitating a similar user experience for both the virtual application and the local application. Users can continue to adopt their preferred client IMEs, and such IMEs behave in the same way as when used with local applications when used with virtual applications.

  While specific embodiments have been described, many alternative embodiments or variations can be made. For example, although IMEs have been described in connection with input of non-phonetic language symbols such as Chinese characters, the improved techniques herein are not limited to IMEs that perform these functions. For example, the improved techniques herein can also be used with IMEs that input phonetic characters, for example, to support autocomplete, handwritten character recognition, or speech recognition.

  Furthermore, although features have been shown and described with reference to particular embodiments herein, such features may be included in any of the disclosed embodiments and variations thereof. Yes, and is included herein. Thus, it can be seen that features disclosed in connection with any embodiment are included as variants of any other embodiment.

  Furthermore, this improvement, or part of this improvement, includes: magnetic disk, magnetic tape, compact disk, DVD, optical disk, flash drive, SD (Secure Digital) chip or device, application specific integrated circuit (ASIC), field Embodied as a computer program product including one or more non-transitory computer readable storage media (eg, shown as media 650 in FIG. 6), such as a programmable gate array (FPGA) It is possible. Any number of computer readable media can be used. The media can be encoded with instructions that, when executed on one or more computers or other processors, are one or more processes described herein. Execute. Such media can be considered a product or a machine and can be transported from one machine to another.

  As used throughout this document, the terms “comprising”, “including”, “containing”, and “having” refer to any particular item, step , Elements or aspects are intended to be presented in an open-ended manner. Also, as used herein, the word “set” means one or more of something, unless specifically stated otherwise. This is true regardless of whether the phrase “set of” is followed by the singular or plural object, and the phrase “set of” is joined to the singular or plural verb. Whether or not it is true. Furthermore, in this specification, expressions indicating order, for example, “first”, “second”, “third”, etc. may be used as adjectives, but such order The expression is used for identification purposes and is not intended to imply any ordering or sequence unless specifically indicated. Thus, for example, a second event can occur before or after the first event, or even if the first event does not occur. In addition, identifying a particular element, feature, or act as a “first” such element, feature, or act is referred to herein as a “second” or other such Should not be construed as requiring that any element, feature, or act be present. Rather, the “first” item may be the only item. Although specific embodiments are disclosed herein, it will be understood that these are provided by way of example only and that the present invention is not limited to these specific embodiments.

  Accordingly, one of ordinary skill in the art appreciates that various changes in form and detail can be made to the embodiments disclosed herein without departing from the scope of the invention. Let's go.

Claims (17)

Establishing a communication channel between a client IME (Input Method Editor) operating on a client machine and a server IME operating on a server, the communication channel including the server IME The virtual application runs on the server and provides screen output, the screen output is rendered on the client machine, and the client IME An IME installed on a client machine and selected by a user operating the client machine, and the communication channel is a virtual application using the server IME as a proxy by the client IME. And a step which makes it possible to interact with the Deployment,
Relaying IME data received from the client IME via the communication channel to the virtual application by the server IME and providing the IME data to the virtual application, wherein the IME data is the client (I) a composite element generated by the client IME in response to user input to a machine and including a set of composite characters and display attributes; and (ii) character prediction based on the set of composite characters. A start request that provides a request to start a candidate window of:
Wherein the application information received from the virtual application by the server IME through the communication channel by the server IME relays to the client IME, and a step of providing the application information to the client IME, the application information but the generated by the virtual application in response to a start request, wherein the application information comprises the steps including a starting response which the virtual application indicating whether the candidate window,
In response to returning the startup response, and as the startup response indicates that the virtual application displays the candidate window,
Receiving a candidate list element from the client IME via the communication channel by the server IME;
Transferring the candidate list element to the virtual application by the server IME;
Rendering the candidate window into the screen output of the virtual application using the candidate list element by the virtual application, and inputting human language characters into the virtual application.   Establishing a communication channel between a client IME (Input Method Editor) operating on a client machine and a server IME operating on a server, the communication channel including the server IME The virtual application runs on the server and provides screen output, the screen output is rendered on the client machine, and the client IME An IME installed on a client machine and selected by a user operating the client machine, and the communication channel is a virtual application using the server IME as a proxy by the client IME. Enabling the user to interact with the application,
  Relaying IME data received from the client IME via the communication channel to the virtual application by the server IME and providing the IME data to the virtual application, wherein the IME data is the client (I) a composite element generated by the client IME in response to user input to a machine and including a set of composite characters and display attributes; and (ii) character prediction based on the set of composite characters. A start request that provides a request to start a candidate window of:
  Relaying application information received from the virtual application by the server IME to the client IME via the communication channel by the server IME, and providing the application information to the client IME, the application information Generated by the virtual application in response to the start request, the application information including a start response indicating whether the client IME displays a candidate window;
  In response to returning the start response, and as the start response indicates that the client IME displays the candidate window,
  Receiving a coordinate request from the client IME via the communication channel by the server IME;
  Transferring the coordinate request to the virtual application by the server IME;
  Receiving a coordinate response from the virtual application by the server IME, the coordinate response comprising:
  (I) the cursor position of the virtual application cursor, or
  (Ii) providing one of a set of coordinates indicating an extent area of the composite window displayed by the virtual application;
  The coordinate response is returned to the client IME via the communication channel, allowing the client IME to display the candidate window at a location relative to the screen output based on the coordinate response, thereby The candidate window is
  (I) the cursor position, or
  (Ii) a method of inputting human language characters into a virtual application comprising: displaying next to one of the composition windows. The IME data received via the communication channel is
Further comprising a composite start message generated by the client IME in response to a user input action;
The method comprises
In response to receiving the composition start message and the composition element, further comprising displaying a composition window in the screen output of the virtual application by the virtual application;
The method of claim 1 or 2 , wherein the composite window renders the set of composite characters using the display attributes. Receiving the startup request from the client IME via the communication channel by the server IME;
Transferring the startup request to the virtual application by the server IME;
Receiving the startup response from the virtual application by the server IME;
The starting response, further comprising the steps of: returning to the client IME via the communication channel, The method of claim 3. Responsive to a change in the composite element, further comprising: sending an updated coordinate response by the server IME to the client IME via the communication channel;
The updated coordinate response is
3. The method of claim 2 , indicating one of (i) an updated cursor position, or (ii) an updated set of coordinates indicating the extent area of the composite window. If the user moves the display window on the client machine, a further updated coordinate response is sent by the server IME via the communication channel to the client IME;
The client IME
6. The method of claim 5 , further comprising the step of: (i) allowing the candidate window to continue to be displayed next to the cursor position, or (ii) one of the composite windows. The client IME and the server IME each have a respective compartment identifying an IME configuration;
The method synchronizes the compartment settings of the client IME with the compartment settings of the server IME so that any changes made to the compartment settings of either IME are performed in the compartment settings of both IMEs. further comprising the method of claim 1 or 2 the steps. Further comprising performing a combining operation after establishing the communication channel;
The combining operation is
(I) sending a join request to the client machine by the server, the join request specifying a list of human languages supported by the server;
(Ii) receiving a combined response from the client machine by the server, the combined response identifying a list of human languages supported by the server that is also supported by the client machine; ,
And (iii) a step of configuring the server to support each of the human language on the list of received human language from the client machine, the method according to claim 1 or 2. A computer program product comprising a set of non-transitory computer readable media having instructions that when executed by a control circuit cause the control circuit to execute a method for inputting human language characters into a virtual application,
By said method
Establishing a communication channel between a client IME (Input Method Editor) running on a client machine and a server IME running on a server,
The communication channel allows the client IME to interact with a virtual application via the server IME;
The virtual application runs on the server and provides screen output, the screen output is rendered on the client machine, the client IME is installed on the client machine, and the client machine is An IME selected by an operating user, wherein the communication channel allows the client IME to interact with a virtual application using the server IME as a proxy;
Relaying IME data received from the client IME via the communication channel to the virtual application by the server IME to provide the IME data to the virtual application;
The IME data is generated by the client IME in response to user input to the client machine, and is based on (i) a composite element including a set of composite characters and display attributes; and (ii) based on the set of composite characters. A start request that provides a request to start a candidate window for identifying character predictions;
Said relays to the client IME via the communication channel by the server IME application information received from the virtual application by the server IME, providing the application information to the client IME, the application information, the starting the generated by the virtual application in response to the request, the application information, see contains an initiation response which the client IME indicating whether to display the candidate window,
In response to returning the startup response, and the startup response indicates that the client IME displays the candidate window;
Receiving a coordinate request from the client IME via the communication channel by the server IME;
Transferring the coordinate request to the virtual application by the server IME;
Configured to receive a coordinate response from the virtual application by the server IME;
The coordinate response is
(I) the cursor position of the virtual application cursor, or
(Ii) providing one of a set of coordinates indicating an extent area of the composite window displayed by the virtual application;
The coordinate response is returned to the client IME via the communication channel, allowing the client IME to display the candidate window at a location relative to the screen output based on the coordinate response, thereby The candidate window is
(I) the cursor position or
(Ii) A computer program product configured to be displayed next to one of the composition windows . The IME data received via the communication channel is
Further comprising a composite start message generated by the client IME in response to a user input action;
In response to receiving the composition start message and the composition element by the method, the virtual application displays a composition window in the screen output of the virtual application;
The computer program product of claim 9 , wherein the composite window is configured to render the set of composite characters using the display attributes. By said method
Receiving the startup request from the client IME via the communication channel by the server IME;
Forwarding the startup request to the virtual application by the server IME;
Configured to receive the startup response from the virtual application by the server IME;
The computer program product of claim 10 , wherein the computer program product is configured to return the startup response to the client IME via the communication channel. By said method
Responsive to changes in the composite element, configured to send an updated coordinate response by the server IME to the client IME via the communication channel;
The updated coordinate response is
(I) updated cursor position, or (ii) is configured to indicate one of a set of the updated coordinates indicating the extent area of the synthesis window, the computer according to claim 9 Program product. By said method
If the user moves the display window on the client machine, a further updated coordinate response is sent by the server IME via the communication channel to the client IME;
The client IME
The computer program product of claim 12 , configured to allow the candidate window to continue to be displayed next to (i) the cursor position, or (ii) one of the composite windows. A server device including a control circuit including a set of processors coupled to the memory for executing instructions stored in the memory,
The control circuit comprises:
A client IME (Input Method Editor) running on the client machine;
Configured to establish a communication channel with a server IME running on the server;
The communication channel allows the client IME to interact with a virtual application via the server IME;
The virtual application runs on the server and provides screen output;
The screen output is rendered on the client machine, the client IME is an IME installed on the client machine and selected by a user operating the client machine, and the communication channel is the client Enabling an IME to interact with a virtual application using the server IME as a proxy;
IME data received from the client IME via the communication channel is relayed to the virtual application by the server IME to provide the IME data to the virtual application;
The IME data is generated by the client IME in response to user input to the client machine, and is based on (i) a composite element including a set of composite characters and display attributes; and (ii) based on the set of composite characters. A start request that provides a request to start a candidate window for identifying character predictions;
Said relays to the client IME via the communication channel by the server IME application information received from the virtual application by the server IME, providing the application information to the client IME, the application information, the starting the generated by the virtual application in response to the request, the application information, see contains an initiation response which the virtual application indicating whether the candidate window,
In response to returning the startup response, and as the startup response indicates that the virtual application displays the candidate window,
The control circuit comprises:
Receiving a candidate list element from the client IME via the communication channel by the server IME;
Transferring the candidate list element to the virtual application by the server IME;
A server device constructed and configured to render the candidate window in the screen output of the virtual application using the candidate list element by the virtual application . The IME data received via the communication channel is
Further comprising a composite start message generated by the client IME in response to a user input action;
In response to receiving the composition start message and the composition element, the control circuit causes the virtual application to display a composition window in the screen output of the virtual application and to use the display attribute. 15. The server device of claim 14 , further constructed and configured to render the set of composite characters. The control circuit comprises:
Receiving the startup request from the client IME via the communication channel by the server IME;
Forwarding the startup request to the virtual application by the server IME;
Configured to receive the startup response from the virtual application by the server IME;
16. The server device according to claim 15 , further constructed and configured to return the startup response to the client IME via the communication channel. The client IME and the server IME each have a respective compartment identifying an IME configuration;
The control circuit synchronizes the compartment settings of the server IME with the compartment settings of the client IME, so that any changes made to the compartment settings of either IME are in the compartment settings of both IMEs. 15. The server device according to claim 14 , further constructed and configured to be implemented.