CN115079887A - Method for displaying auxiliary prompt for mapping android application keyboard on desktop system - Google Patents
Method for displaying auxiliary prompt for mapping android application keyboard on desktop system Download PDFInfo
- Publication number
- CN115079887A CN115079887A CN202210992711.6A CN202210992711A CN115079887A CN 115079887 A CN115079887 A CN 115079887A CN 202210992711 A CN202210992711 A CN 202210992711A CN 115079887 A CN115079887 A CN 115079887A
- Authority
- CN
- China
- Prior art keywords
- frame body
- text
- target frame
- window
- display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/146—Aligning or centring of the image pick-up or image-field
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/048—Indexing scheme relating to G06F3/048
- G06F2203/04804—Transparency, e.g. transparent or translucent windows
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a method for displaying keyboard mapping auxiliary prompts of android applications on a desktop system, which is characterized in that when a user configures and starts a keyboard mapping function and converts key values in a configuration file into corresponding auxiliary prompt information to be displayed in an android application display interface appointed by key positions in the configuration file in an android compatible environment, auxiliary prompt information display modes in various forms are provided according to the characteristics of the android application display interface, so that the display of the auxiliary prompt information can achieve the effect that the user can obviously identify and the normal visual experience of the user is not influenced.
Description
Technical Field
The invention belongs to the technical field of development of running mobile applications on a desktop operating system, and particularly relates to a display method of an auxiliary mapping prompt of an android application keyboard on a desktop system.
Background
The Android running environment of the desktop operating system such as Linux or Windows is a technology for running Android application (App) on the Linux or Windows operating system, and the technology has very important significance for enriching the application ecology of the operating system. Under the general condition, a typical android compatible environment mainly includes an android application display unit and an android application compatible service, the android application display unit is used for realizing a user interface of an android application on a desktop operating system, the android application compatible service is used for providing the android compatible environment running on the desktop operating system, and the android application is effectively run on the desktop operating system through the mutual cooperation of the android application display unit and the android application compatible service, for example, an implementation scheme of the existing android compatible environment has xDroid, which includes xDroid ui and xDroid server, the xDroid ui is the android application display unit, and the xDroid server is the android application compatible service.
When an Android application is run on a desktop operating system in an Android running environment, a mouse is usually used to click and simulate a touch event because a display of a PC computer often does not support touch. However, for complex Android application window operations, such as multi-point quick touch in a game application, different keys of a keyboard are required to correspond to different positions of an application window, that is, pressing a certain key is equivalent to clicking a certain position of the application window, and a keyboard event is converted into a touch event in an Android system according to a setting preset by a user, so that a function of experiencing an Android system game by using the keyboard is realized. However, for some large-scale game applications, many keys are often required to be set to meet the operation requirements of the user, and the increased keys may cause difficulty in memorizing the corresponding relationship between the keys and the display interface area by the user, so that displaying the keyboard mapping auxiliary prompt information for the android application configured with the keyboard mapping function is very necessary for the user to operate conveniently.
In the existing keyboard mapping implementation scheme, the keyboard mapping auxiliary prompt information cannot intelligently select a displayed area, and the situation of covering texts in an android application display interface may be generated, so that the operation of a user is inconvenient and even misoperation is caused, and further the user experience is influenced.
Disclosure of Invention
In view of this, the invention provides a method for displaying keyboard mapping auxiliary prompts of an android application on a desktop system, provides various display modes of the keyboard mapping auxiliary prompts and realizes that the specific display modes of the keyboard mapping auxiliary prompts are adjusted according to the actual situation of a display interface of the android application.
The invention provides a display method of an android application keyboard mapping auxiliary prompt on a desktop system, which specifically comprises the following steps:
step 1, when a user opens an android application in an android compatible environment and starts a keyboard mapping, saving a current display interface of the android application; acquiring a window handle, a window size and a window left vertex position of a current display window of the android application; establishing a window with top display at the beginning and end and transparent background in a desktop system as an auxiliary prompt window, wherein the size and the position of the auxiliary prompt window are the same as those of the current display window, setting a parent window of the auxiliary prompt window by adopting the window handle, and setting the auxiliary prompt window into a mouse and keyboard event penetration mode;
step 2, converting key values of keys mapped by the keyboard into corresponding auxiliary prompt texts, and acquiring text display ranges of the auxiliary prompt texts; if the auxiliary prompt display mode is a frame mode, executing the step 3, and if the auxiliary prompt display mode is a text mode, executing the step 4;
step 3, constructing a target frame body according to the text display range and the key positions mapped by the keyboard, wherein the left vertex position or the center position of the target frame body is the key position, the size of the target frame body is a set value increased on the basis of the text display range, the filling color of the target frame body is a set value with transparency, and an auxiliary prompt text is drawn in the target frame body; executing the step 5;
step 4, determining a corresponding target area in the current display interface of the android application according to the text display range and the key positions mapped by the keyboard, calculating the mean value of RGB values of all pixel points in the target area, and taking a reflection color value for the mean value; constructing a target frame body with a completely transparent background, wherein the left vertex position or the center position of the target frame body is a key position, the size of the target frame body is a text display range, an auxiliary prompt text is drawn in the target frame body, and the color of the auxiliary prompt text is set as a counter color value; executing the step 5;
and 5, drawing a target frame body in the auxiliary prompt window to finish the display of the keyboard mapping auxiliary prompt information.
Further, when the user opens the android application in the android compatible environment to start the keyboard mapping in step 1, if the keyboard mapping configuration information of the android application exists in the keyboard mapping configuration file and the keyboard mapping is configured to be opened, saving the current display interface of the android application; and otherwise, storing the current display interface of the android application after the user configures and stores the keyboard mapping configuration file.
Further, the auxiliary prompt window is a frameless window.
Further, after the target frame is drawn in the auxiliary prompt window in the step 5, if the android application in the desktop system is multi-window display, the focus event of the current display window is monitored, and if the current display window loses focus, the auxiliary prompt window is hidden.
Further, after the auxiliary prompt text is drawn in the target frame in step 3, the following steps are performed:
step 5.1, acquiring the positions of texts in all text boxes in a current display window in an android compatible environment as display interface text positions, and sending the display interface text positions to an android application display unit;
step 5.2, if the left vertex position of the target frame body is the same as the display interface text position or the difference value is smaller than the threshold value, calculating a new region which is the same as the target frame body in area and different from the display interface text position or the difference value is not smaller than the threshold value by the android application display unit by taking the target frame body as the center, if the new region exists, taking the new region as a new target frame body and executing the step 5, and if the new region does not exist, executing the step 5.4; if the left vertex position of the target frame body is different from the text position of the display interface or the difference value is not smaller than the threshold value, executing the step 5.3;
step 5.3, identifying whether a text is contained in the region with the same position as the target frame body in the current display window, if so, taking the text region with the same position as the target frame body as a display interface text region, making the value of the test frequency N zero, and executing step 5.4; if the text is not contained, the target frame body is reserved as a new target frame body, and then the step 5 is executed;
step 5.4, if N is not greater than the threshold, moving the target frame body on a circle which takes the center point of the frame body as the center and takes the set pixel as the radius, acquiring the new position of the target frame body according to the set step in the moving process, taking the frame body at the new position as the target frame body, adding 1 to N, adding the set pixel with the set value, and executing step 5.3; and if N is larger than the threshold value, selecting the display interface text area with the minimum text area as the target frame body and then executing the step 5.
Further, in step 4, after the color of the auxiliary prompt text is set to the color reflection value, the following steps are performed:
6.1, acquiring the positions of texts in all text boxes in a current display window in an android compatible environment as display interface text positions, and sending the display interface text positions to an android application display unit;
6.2, if the left vertex position of the target frame body is the same as the text position of the display interface or the difference value is smaller than the threshold value, calculating a new region which has the same area as the target frame body and is different from the text position of the display interface or the difference value is not smaller than the threshold value by the android application display unit by taking the target frame body as the center, if the new region exists, taking the new region as a new target frame body and executing the step 5, and if the new region does not exist, executing the step 6.4; if the left vertex position of the target frame body is different from the text position of the display interface or the difference value is not smaller than the threshold value, executing a step 6.3;
step 6.3, identifying whether a text is contained in the region with the same position as the target frame body in the current display window, if so, taking the text region with the same position as the target frame body as a display interface text region, making the value of the test frequency N zero, and executing step 6.4; if the text is not contained, the target frame body is reserved as a new target frame body, and then the step 5 is executed;
step 6.4, if N is not greater than the threshold, moving the target frame body on a circle which takes the center point of the frame body as the center and takes the set pixel as the radius, acquiring the new position of the target frame body according to the set step in the moving process, taking the frame body at the new position as the target frame body, adding 1 to N, adding the set pixel with the set value, and executing step 6.3; and if N is larger than the threshold value, selecting the display interface text area with the minimum text area as the target frame body and then executing the step 5.
Has the advantages that:
1. according to the method and the device, when a user configures and starts a keyboard mapping function and converts key values in the configuration file into corresponding auxiliary prompt information to be displayed in the android application display interface appointed by the key positions in the configuration file in the android compatible environment, the auxiliary prompt information display modes in various forms are provided according to the characteristics of the android application display interface, so that the display of the auxiliary prompt information can achieve the effect that the user can obviously identify and the normal visual experience of the user is not influenced.
2. According to the method and the device, aiming at the multi-window mode of the android compatible environment, only the keyboard mapping auxiliary prompt information of the android application is displayed after the android application currently operated by a user is determined.
Drawings
Fig. 1 is a flowchart of a method for displaying an auxiliary prompt for mapping an android application keyboard on a desktop system according to the present invention.
Detailed Description
The invention is described in detail below by way of example with reference to the accompanying drawings.
The invention provides a method for displaying an auxiliary prompt for mapping an android application keyboard on a desktop system, which has the core idea that: when a user configures and starts a keyboard mapping function in an android compatible environment, converting key values in a configuration file into corresponding auxiliary prompt information and displaying the auxiliary prompt information in an android application display interface appointed by key positions in the configuration file, wherein the auxiliary prompt information comprises two forms: the auxiliary prompt information is displayed in a colored text mode, and the auxiliary prompt information needs to be displayed so that a user can obviously recognize the auxiliary prompt information and the normal visual experience effect of the user is not affected.
The invention provides a method for displaying an auxiliary prompt for mapping an android application keyboard on a desktop system, which comprises the following steps of:
step 1, when a user opens an android application in an android compatible environment, if keyboard mapping configuration information of the android application exists in a keyboard mapping configuration file and is configured to open keyboard mapping, saving a current display interface of the android application and then executing step 2; otherwise, after the user configures and starts the keyboard mapping and saves the keyboard mapping configuration file, saving the current display interface of the android application and then executing the step 2.
The keyboard mapping configuration file comprises an auxiliary prompt display mode, a key position and a key value. The auxiliary prompt display mode comprises the following two modes: firstly, the auxiliary prompt text corresponding to the key values of the keys is displayed in a frame with background color, the frame can be in a rounded rectangle, rectangle or other areas, and in addition, the auxiliary prompt text can be displayed in the frame in the middle; and secondly, the auxiliary prompt text is displayed in a text mode, namely, only the auxiliary prompt text is displayed in a mode which can be recognized by a user.
Step 2, acquiring a window handle, a window size and a window left vertex position of a current display window of the android application; the method comprises the steps of establishing a transparent window with the property of being always displayed on the top as an auxiliary prompt window in a desktop system, setting the size and the position of the auxiliary prompt window by adopting the window size and the window left vertex position of a current display window of an android application, and setting a parent window of the auxiliary prompt window by adopting a window handle of the current display window of the android application.
In order to further improve the user experience, the auxiliary prompt window can be set as a frameless window.
And 3, setting the whole area of the auxiliary prompt window into a mouse and keyboard event penetrating mode, namely responding to the mouse and keyboard events in the auxiliary prompt window by the parent window android application current display window of the auxiliary prompt window, so that all mouse and keyboard events aiming at the auxiliary prompt window in the desktop system can be sent into an android compatible environment.
Specifically, the method comprises the following steps:
acquiring a display server of the desktop system, if the display server is a display server supporting an X11 protocol, informing a window manager to traverse all windows currently displayed in the desktop system to determine auxiliary prompt windows therein, and setting a display area corresponding to the auxiliary prompt windows as a mouse penetration area by adopting XShapeCombineRectangle;
if the display server is a display server supporting a Wayland protocol, acquiring global mouse and keyboard events through a DBus interface, if the mouse and keyboard events contain event types and position information, judging whether the mouse and keyboard events are events occurring in an auxiliary prompt window area or not through the position information and android application window size and window left vertex position of a current display window, if not, not responding to the events, and if so, modifying a Wayland window manager to send the mouse and keyboard events in the auxiliary prompt window area to a parent window of the auxiliary prompt window;
if the display server is a Windows system display server, the window manager is informed to traverse all Windows currently displayed in the desktop system to determine auxiliary prompt Windows therein, and the SetWindowLong is called to set the attributes of the auxiliary prompt Windows to WS _ EX _ TRANSPARENT and WS _ EX _ LAYERED.
Step 4, converting key values in the android application keyboard mapping configuration file into corresponding auxiliary prompt texts, and acquiring text display ranges of the auxiliary prompt texts; and acquiring an auxiliary prompt display mode and key positions in the keyboard mapping configuration file, executing step 5 if the auxiliary prompt display mode is displayed in a frame mode, and executing step 6 if the auxiliary prompt display mode is displayed in a text mode.
Step 5, according to the text display range and the key positions of the auxiliary prompt text obtained in the step 4, determining a frame body to be drawn in the auxiliary prompt window as a target frame body, wherein the left vertex position or the center position of the target frame body is the key position, the size of the target frame body is a set value added on the basis of the text display range, the filling color of the target frame body is a set value with transparency, and the auxiliary prompt text is drawn in the target frame body; step 7 is performed.
Step 6, determining a corresponding target area in the current display interface of the android application according to the text display range and the key positions of the auxiliary prompt text obtained in the step 4, acquiring RGB values of all pixel points in the target area, calculating the mean value of all the RGB values, and taking a color inversion value for the mean value; determining a frame body which is fully transparent in the background drawn in the auxiliary prompt window as a target frame body, wherein the left vertex position or the center position of the target frame body is a key position, the size of the target frame body is a text display range, an auxiliary prompt text is drawn in the target frame body, and the color of the auxiliary prompt text is set as a counter color value; step 7 is performed.
And 7, drawing the target frame determined in the step in the auxiliary prompt window to finish the display of the keyboard mapping auxiliary prompt information.
Further, when the android application in the desktop system is in a multi-window display mode, in order to meet the operation requirement that only one window can be used for keyboard mapping under multiple windows and solve the problem of window overlapping possibly caused by the fact that the auxiliary prompt window is always set to the top, the auxiliary prompt display in the multi-window display mode provided by the invention has the following processing mode: the android application display unit monitors a focus event of a current display window of the android application, if the current display window loses focus, the auxiliary prompt window is hidden, and the auxiliary prompt window is displayed when the current display window acquires the focus again.
Further, when the auxiliary prompt information is displayed in the android application interface, since the android application interface corresponding to the auxiliary prompt information display position contains characters, which may cause that a user cannot clearly see key contents and bring difficulty to user operations, in order to achieve a better display effect, the present invention provides a manner of dynamically adjusting the auxiliary prompt information display position based on the position information of the text displayed in the android application display interface, and specifically, the following steps are executed after the step 5 or the step 6:
and 6.1, acquiring position information of texts in all text boxes in the current display window of the android application in the android compatible environment, storing the position information as a display interface text position, and sending the display interface text position to an android application display unit.
6.2, the android application display unit acquires the target frame determined in the step 5 or the step 6, if the left vertex position of the target frame is the same as the display interface text position or the difference value is smaller than the threshold, a new region which meets the requirements of the same area as the target frame and is different from the display interface text position or the difference value is not smaller than the threshold is calculated by taking the target frame as the center, if the new region exists, the step 7 is executed by taking the new region as a new target frame, and if the new region does not exist, the step 6.4 is executed; and if the left vertex position of the target frame body is different from the text position of the display interface or the difference value is not smaller than the threshold value, executing the step 6.3.
Step 6.3, identifying whether a text is contained in a region, having the same position as the target frame, in the current display interface of the android application, if so, taking the text region, having the same position as the target frame, as a display interface text region, making the value of the test frequency N zero, and executing step 6.4; and if the text is not contained, reserving the target frame body as a new target frame body and then executing the step 7.
Step 6.4, if N is not greater than the threshold, moving the target frame body on a circle which takes the center point of the frame body as the center and takes the set pixel as the radius, acquiring the new position of the target frame body according to the set step in the moving process, taking the frame body at the new position as the target frame body, adding 1 to N, adding the set pixel with the set value, and executing step 6.3; if N is greater than the threshold, the display interface text region having the smallest text area is selected as the target box, and then step 7 is executed.
Example (b):
in this embodiment, the method for displaying the keyboard mapping auxiliary prompt of the android application on the desktop system, provided by the present invention, is used for displaying the keyboard mapping auxiliary prompt information of the android application in the android compatible environment of the Linux system, and specifically includes the following steps:
s1, when the user opens the android application on the Linux system, the xDreidUI judges whether the user is configured for the android application and starts the keyboard mapping function according to the keyboard mapping configuration file, and the method comprises the following steps:
if the user does not configure and start the keyboard mapping function, waiting for the user to start the keyboard mapping configuration until the user enters the configuration interface and completes the configuration, storing the screenshot of the current display interface of the android application as Image, and executing S2; if the user has configured and turned on the keyboard mapping function, step S2 is executed.
The keyboard mapping configuration file comprises an auxiliary prompt display mode, a key position pos and a key value keyvalue.
S2, obtaining the window handle, the window size and the left vertex position of the android application by the xDroidUI.
S3, the xdroid ui creates a Qt window QWidget as an auxiliary prompt window, and adds flag information to the auxiliary prompt window, for example, a setObjectName () function is called to set the objectName attribute of the QWidget window to xdroid keynapingTip, and the auxiliary prompt window is an xdroid keynapingTip window; setting the auxiliary prompting window to be always top-mounted for display by adopting a Qt interface, and setting the auxiliary prompting window to be a frameless window with transparent background color; the size of the auxiliary prompt window is set to the window size obtained in S2, the position of the auxiliary prompt window is set to the left vertex position in S2, and the parent window of the auxiliary prompt window is set to the window handle in S2.
S4, aiming at a display server adopted by the Linux system, processing mouse and keyboard events of an xdroid KeymappangTip window:
if the display server of the Linux system supports the X11 protocol, traversing all current windows through a window manager to find a window with an xdrid KeymappingTip mark, and setting an interface area where the window with the xdrid KeymappingTip mark is located as a mouse penetration area by adopting an XShapeCombineRectangle function;
if the display server of the Linux system supports the Wayland protocol, acquiring global mouse and keyboard events through a DBus interface, wherein the mouse and keyboard events comprise event types and position information, judging whether the mouse and keyboard events are located in an android keyboarding tip window area through the position information, the window size and the left vertex position acquired in S2, if the mouse and keyboard events are not located in the android keyboarding tip window area, not processing the mouse and keyboard events, and if the mouse and keyboard events are located in the android keyboarding tip window area, modifying a Wayland window manager to send all the mouse and keyboard events in the android keyboarding tip window area to a parent window of the android keyboarding tip window.
Therefore, all mouse and keyboard events belonging to the android KeymappingTip window are received by the parent window of the android KeymappingTip window, the parent window corresponds to the current display interface of the android application, and the mouse and keyboard events of the android KeymappingTip window are sent to the android compatible environment through the xDreidUI.
S5, the xDreidUI reads the key position pos and the key value keyvalue in the keyboard mapping configuration file, and the key position pos and the key value keyvalue are stored in a variable map; where the keyvalue value is converted to text value text, for example, keyA is converted to A, and if the keyvalue is multiple values such as keySHIFT and keyCtrl, it is converted to SHIFT + Ctrl.
S6, obtaining the display range of text value in variable map, for example, QFNtMetrics is adopted, wherein bounding guide can obtain the display range corresponding to the text, namely rectangular range rect; acquiring an auxiliary prompt display mode in the keyboard mapping configuration file, and executing S7 if the auxiliary prompt display mode is a frame mode display; if the auxiliary indication display mode is a text mode, S8 is executed.
S7, drawing a rectangular background by adopting draw RoundedRect in pos and rect, android KeymapppingTip windows, wherein the size of the background area is 5px increased for the length and width of the rect area, the color is rgba (100,100,100,0.8), white text is drawn in the background area in the middle, and then the text is centered on the rectangular area draw Rect with a transparent background, and S9 is executed.
S8, determining a rectangular region by pos and rect, acquiring RGB values of all points corresponding to the rectangular region by adopting OpenCV in an Image layer saved in S1, calculating a mean value of the RGB values, then taking a counter color value for the mean value, drawing a rectangular region drawRect with a completely transparent text in the middle of the text according to the pos, rect and text values, setting the color value of the text in the drawRect as the counter color value, and executing S9.
S9, if all the values in the map are processed, executing S10, otherwise executing S6.
S10, acquiring all DecorViews in a current display window of the android application in the android compatible environment, acquiring position information of texts in the TextView and the sub-View, sending the position information to the xDroidUI through RPC, and storing the position information into android TextRects by the xDroidUI.
S11, obtaining an Image layer, and identifying whether all drawRect areas in S8 contain texts or not through OpenCV, wherein the method specifically comprises the following steps:
s11.1, comparing the area ranges of the drawRect and the android TextRects, if the drawRect is not in the range of the android TextRects, considering that no text possibly exists in the drawRect, and executing S11.2; if the drawRect is in the range of the android TextRects, calculating a rectangular area which takes the center of the drawRect as the center point, has the length and the width of BN pixel points and the area the same as that of the drawRect and is not in the range of the android TextRects, modifying the drawRect into an obtained new rectangular area if the rectangular area exists, and executing S11.3 if the rectangular area does not exist. In this way the frequency of OpenCV recognition can be significantly reduced.
S11.2, identifying whether the drawRect area contains a text by OpenCV, if so, recording the text area as textRect, and executing S11.3; if not, save drawRect as destRect, go to S11.4.
S11.3, if i is not larger than n, moving i pixel points to the periphery by taking the drawRect area as the center, wherein the value range of i is [1, n ], n can be dynamically configured, increasing m pixels each time i is moved, forming a new newDrawRect area each time i is moved, modifying the drawRect value to newDrawRect, and executing S11.2; if i is larger than n, saving the drawRect corresponding to the textRect with the minimum text area as destRect.
S11.4, if all the drawRect in the drawRect set are processed, obtaining a destRect set and executing S12, otherwise executing S11.1.
S12, drawing the destRect region set into an xdroid KeymappanpingTip window.
Aiming at a multi-window mode of an android compatible environment, the invention provides the following processing modes:
monitoring a window focus event of an android application in a starting state in a desktop system, hiding an android KeymapppingTip window when the window loses focus, and displaying the android KeymapppingTip window when the window acquires focus. Therefore, the operation requirement that only one window can be used for keyboard mapping in a multi-window mode can be met, and the problem that due to the fact that the window is set, when other windows are superposed on the window of the android application, the window of the android application still sets the top can be solved.
In summary, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A method for displaying auxiliary prompt for mapping an android application keyboard on a desktop system is characterized by comprising the following steps:
step 1, when a user opens an android application in an android compatible environment and starts a keyboard mapping, saving a current display interface of the android application; acquiring a window handle, a window size and a window left vertex position of a current display window of the android application; establishing a window with top display at the beginning and end and transparent background in a desktop system as an auxiliary prompt window, wherein the size and the position of the auxiliary prompt window are the same as those of the current display window, setting a parent window of the auxiliary prompt window by adopting the window handle, and setting the auxiliary prompt window into a mouse and keyboard event penetration mode;
step 2, converting key values of keys mapped by the keyboard into corresponding auxiliary prompt texts, and acquiring text display ranges of the auxiliary prompt texts; if the auxiliary prompt display mode is a frame mode, executing the step 3, and if the auxiliary prompt display mode is a text mode, executing the step 4;
step 3, constructing a target frame body according to the text display range and the key positions mapped by the keyboard, wherein the left vertex position or the center position of the target frame body is the key position, the size of the target frame body is a set value increased on the basis of the text display range, the filling color of the target frame body is a set value with transparency, and an auxiliary prompt text is drawn in the target frame body; executing the step 5;
step 4, determining a corresponding target area in the current display interface of the android application according to the text display range and the key positions mapped by the keyboard, calculating the mean value of RGB values of all pixel points in the target area, and taking a reflection color value for the mean value; constructing a target frame body with a completely transparent background, wherein the left vertex position or the center position of the target frame body is a key position, the size of the target frame body is a text display range, an auxiliary prompt text is drawn in the target frame body, and the color of the auxiliary prompt text is set as a counter color value; executing the step 5;
and 5, drawing the target frame in the auxiliary prompt window to finish the display of the keyboard mapping auxiliary prompt information.
2. The display method according to claim 1, wherein in the step 1, when a user opens an android application in an android compatible environment and starts a keyboard map, if keyboard map configuration information of the android application exists in a keyboard map configuration file and the keyboard map is configured to be opened, a current display interface of the android application is saved; and otherwise, storing the current display interface of the android application after the user configures and stores the keyboard mapping configuration file.
3. The display method as claimed in claim 1, wherein the auxiliary prompt window is a frameless window.
4. The display method according to claim 1, wherein after the target frame is drawn in the auxiliary prompt window in step 5, if an android application in the desktop system is a multi-window display, a focus event of a current display window is monitored, and if the current display window loses focus, the auxiliary prompt window is hidden.
5. The display method according to claim 1, wherein after drawing the auxiliary prompt text in the target frame in step 3, the following steps are performed:
step 5.1, acquiring the positions of texts in all text boxes in a current display window in an android compatible environment as display interface text positions, and sending the display interface text positions to an android application display unit;
step 5.2, if the left vertex position of the target frame body is the same as the display interface text position or the difference value is smaller than the threshold value, calculating a new region which is the same as the target frame body in area and different from the display interface text position or the difference value is not smaller than the threshold value by the android application display unit by taking the target frame body as the center, if the new region exists, taking the new region as a new target frame body and executing the step 5, and if the new region does not exist, executing the step 5.4; if the left vertex position of the target frame body is different from the text position of the display interface or the difference value is not smaller than the threshold value, executing the step 5.3;
step 5.3, identifying whether a text is contained in the region with the same position as the target frame body in the current display window, if so, taking the text region with the same position as the target frame body as a display interface text region, making the value of the test frequency N zero, and executing step 5.4; if the text is not contained, the target frame body is reserved as a new target frame body, and then the step 5 is executed;
step 5.4, if N is not greater than the threshold, moving the target frame body on a circle which takes the center point of the frame body as the center and takes the set pixel as the radius, acquiring the new position of the target frame body according to the set step in the moving process, taking the frame body at the new position as the target frame body, adding 1 to N, adding the set pixel with the set value, and executing step 5.3; and if N is larger than the threshold value, selecting the display interface text area with the minimum text area as the target frame body and then executing the step 5.
6. The display method according to claim 1, wherein in step 4, after the color of the auxiliary prompt text is set to a color reversal value, the following steps are performed:
6.1, acquiring the positions of texts in all text boxes in a current display window in an android compatible environment as display interface text positions, and sending the display interface text positions to an android application display unit;
6.2, if the left vertex position of the target frame body is the same as the text position of the display interface or the difference value is smaller than the threshold value, calculating a new region which has the same area as the target frame body and is different from the text position of the display interface or the difference value is not smaller than the threshold value by the android application display unit by taking the target frame body as the center, if the new region exists, taking the new region as a new target frame body and executing the step 5, and if the new region does not exist, executing the step 6.4; if the left vertex position of the target frame body is different from the text position of the display interface or the difference value is not smaller than the threshold value, executing the step 6.3;
step 6.3, identifying whether a text is contained in the region with the same position as the target frame body in the current display window, if so, taking the text region with the same position as the target frame body as a display interface text region, making the value of the test frequency N zero, and executing step 6.4; if the text is not contained, the target frame body is reserved as a new target frame body, and then the step 5 is executed;
step 6.4, if N is not greater than the threshold, moving the target frame body on a circle which takes the center point of the frame body as the center and takes the set pixel as the radius, acquiring the new position of the target frame body according to the set step in the moving process, taking the frame body at the new position as the target frame body, adding 1 to N, adding the set pixel with the set value, and executing step 6.3; and if N is larger than the threshold value, selecting the display interface text area with the minimum text area as the target frame body and then executing the step 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210992711.6A CN115079887B (en) | 2022-08-18 | 2022-08-18 | Display method of auxiliary prompt for mapping android application keyboard on desktop system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210992711.6A CN115079887B (en) | 2022-08-18 | 2022-08-18 | Display method of auxiliary prompt for mapping android application keyboard on desktop system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115079887A true CN115079887A (en) | 2022-09-20 |
| CN115079887B CN115079887B (en) | 2022-10-25 |
Family
ID=83244386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210992711.6A Active CN115079887B (en) | 2022-08-18 | 2022-08-18 | Display method of auxiliary prompt for mapping android application keyboard on desktop system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115079887B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115495192A (en) * | 2022-11-17 | 2022-12-20 | 北京麟卓信息科技有限公司 | Cross-operation-environment low-delay input method based on interrupt penetration |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020163542A1 (en) * | 1999-06-11 | 2002-11-07 | Mihai Costea | System, method, and computer-readable medium for displaying keyboard cues in a window |
| CN104820551A (en) * | 2015-04-24 | 2015-08-05 | 青岛海信移动通信技术股份有限公司 | Method and device for touch operation in terminal |
| CN112882625A (en) * | 2021-02-10 | 2021-06-01 | 南京苏宁软件技术有限公司 | Element pickup method, element pickup device, computer equipment and storage medium |
| CN113220135A (en) * | 2021-04-27 | 2021-08-06 | 北京麟卓信息科技有限公司 | Touch event-based adaptive keyboard mapping method and device |
| CN113220136A (en) * | 2021-04-27 | 2021-08-06 | 北京麟卓信息科技有限公司 | Method and device for automatically switching keyboard mapping schemes based on window layout perception |
| CN113238663A (en) * | 2021-04-27 | 2021-08-10 | 北京麟卓信息科技有限公司 | Adaptive keyboard mapping method and device based on view matching |
-
2022
- 2022-08-18 CN CN202210992711.6A patent/CN115079887B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020163542A1 (en) * | 1999-06-11 | 2002-11-07 | Mihai Costea | System, method, and computer-readable medium for displaying keyboard cues in a window |
| CN104820551A (en) * | 2015-04-24 | 2015-08-05 | 青岛海信移动通信技术股份有限公司 | Method and device for touch operation in terminal |
| CN112882625A (en) * | 2021-02-10 | 2021-06-01 | 南京苏宁软件技术有限公司 | Element pickup method, element pickup device, computer equipment and storage medium |
| CN113220135A (en) * | 2021-04-27 | 2021-08-06 | 北京麟卓信息科技有限公司 | Touch event-based adaptive keyboard mapping method and device |
| CN113220136A (en) * | 2021-04-27 | 2021-08-06 | 北京麟卓信息科技有限公司 | Method and device for automatically switching keyboard mapping schemes based on window layout perception |
| CN113238663A (en) * | 2021-04-27 | 2021-08-10 | 北京麟卓信息科技有限公司 | Adaptive keyboard mapping method and device based on view matching |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115495192A (en) * | 2022-11-17 | 2022-12-20 | 北京麟卓信息科技有限公司 | Cross-operation-environment low-delay input method based on interrupt penetration |
| CN115495192B (en) * | 2022-11-17 | 2023-01-13 | 北京麟卓信息科技有限公司 | Cross-operation-environment low-delay input method based on interrupt penetration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115079887B (en) | 2022-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9870136B2 (en) | Controlling visualization of data by a dashboard widget | |
| KR101278771B1 (en) | Interface and system for manipulating thumbnails of live windows in a window manager | |
| JP3486459B2 (en) | Electronic information equipment and control method thereof | |
| US9639238B2 (en) | Modification of a characteristic of a user interface object | |
| US9401099B2 (en) | Dedicated on-screen closed caption display | |
| US20060248471A1 (en) | System and method for providing a window management mode | |
| US20120110483A1 (en) | Multi-desktop management | |
| US20090019385A1 (en) | Management of Icons in a Display Interface | |
| US20020015064A1 (en) | Gesture-based user interface to multi-level and multi-modal sets of bit-maps | |
| KR20110084194A (en) | Child window surfacing and management | |
| KR20120054750A (en) | Method and apparatus for selective display | |
| US20130093782A1 (en) | Color Selection and Chart Styles | |
| US11734805B2 (en) | Utilizing context-aware sensors and multi-dimensional gesture inputs to efficiently generate enhanced digital images | |
| US20170168653A1 (en) | Context-driven, proactive adaptation of user interfaces with rules | |
| US20100077304A1 (en) | Virtual Magnification with Interactive Panning | |
| WO2014013949A1 (en) | Character string selection device, character string selection method, control program, and recording medium | |
| CN115079887B (en) | Display method of auxiliary prompt for mapping android application keyboard on desktop system | |
| US11922543B2 (en) | Device and method for coloring sketch image with color pointer | |
| US20190107925A1 (en) | Optimizing a display of a user device | |
| US20220319388A1 (en) | Display control method and electronic device | |
| WO2010119975A1 (en) | Relationship information display device, relationship information display method, and relationship information display program | |
| US10241663B2 (en) | Information processing apparatus capable of being operated by multi-touch | |
| US9190018B2 (en) | Vector-based customizable pointing indicia | |
| KR101628972B1 (en) | Method and apparatus for controlling graphic user interface based on illumination | |
| CN108932054B (en) | Display device, display method, and non-transitory recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |