WO2013134975A1

WO2013134975A1 - Method of Scene Recognition Based Virtual Touch Event

Info

Publication number: WO2013134975A1
Application number: PCT/CN2012/073434
Authority: WO
Inventors: Kuizhi MEI; Guohui Li; Nanning ZHENG; Sheng FENG; Wenlong CHOU; Lei Zhang
Original assignee: Xi'an Jiaotong University
Priority date: 2012-03-13
Filing date: 2012-03-31
Publication date: 2013-09-19
Also published as: CN102662559B; CN102662559A

Abstract

In general, this disclosure describes a technique to simulate the keystrokes on some handset device like remote controller, keyboard, to the touch-based event like slide, tab on the current scene in order to manage the applications and operating systems which are originally designed around the touch interface, with the help of scene recognition algorithm. To realize this simulation, the information for touch-based events and characteristic measurement should be acquired from the scenes of application which have been collected previously. Then, if a scene of the application is recognized with the characteristic information, the layout of the touch-based events in the scene is sure to be known, and consequently the operation of handset device can be mapped to the touch-based events. For instance, if a scene with four buttons is identified, the 'up' keystrokes can be translated to the event of touching the point located above the current scene.

Description

Specification

Method of Scene Recognition Based Virtual Touch Event Background

Sorts of intelligent operating systems, such as Android®, IOS®, Bada® and Meego®, are widely used in portable devices like cell phones, and are popular around the consumers as a result of rich applications and powerful functions. Therefore, many electronics and information technology companies spend lots of efforts in this field and make a great deal of profit. Currently, these systems have been introduced to the aspect of household appliances, like digital television. For examples, Samsung® and Sony® are active in research in this kind of television and have exhibited their own Android® smart televisions on the CES 2012.

The operating systems used in portable devices and their corresponding applications, however, are designed around the interface of touch in order to provide convenient operations for users, which is not feasible for television. It is unpractical to produce a touch screen which expends as large as TV due to the great expense and lots of energy costing. Some producers attach a wired or wireless mouse to television for purpose of touch-like operations may leave users with an uncomfortable manner in operating television, especially when users sit on sofa or lie in bed. Considering this, television producers begin to renew the means in interaction and put forward multiple intelligent methods to interact with these popular systems, such as gesture remote, voice recognition, multi-screen interaction, etc, to strive for a market share. The exploration of new interaction has been tried since Google TV is showed out and it is capable to use the wireless full-keyboard remote controller to surf the internet or listen to music on TV. This method, however, is nearly the same to use the PC at home, and it is more difficult to the user unfamiliar to computer, which leaves a poor user experience. In another example, LG's Magic Motion Remote Control, a stick-like controller, is used to mimic the operation of mouse with the help of sensors and RF infrared. Although the remote controller is useful in control digital television, it consumes huge amount of energy and costs a lot both in software and hardware.

With respect to the cost and feasibility, this disclosure puts forward a technique to simulate the keystrokes on some handset device to the operations like slide and tab on the current scene in order to manage the applications as if they were operated by some touch-based devices which are originally supported by these operating system, like touch screen, mouse device, based on the scene recognition algorithm.

Summary

An application in the operating system is always composed of limited pictures with regular frame. The picture which is capable to respond to input events, such as touch event, slide event and so on, can be called as the scene. Considering that the layout style of these scenes in one application is simple so as to be friendly for users, the image is capable to be distinguished from others. In another hand, as the result of limited amount, the scenes of an application can be acquired previously in order to analyze for feature information like pixel, shape and gradient, and for the native-supported touch-based events, like touch, slide, multi-touch. Once the current scene being recognized, the touch-based events of the scene are clear and therefore can be mapped to the various keystrokes of handset devices.

The operating system which is invented originally for cell phones is controlled around the touch interface and thus functions well. The digital television with this kind of system, however, is stuck in a deprivation of touch screen and therefore is not capable to be operated by the traditional infrared remote controller. Through the method mentioned in this disclosure, the remote controller, keyboard or some nonnative-supported means will be simulated to the touch-based events like touch, mouse click, slide, pinch-to-zoom and so on, depend on recognition of scenes and acquirement of its native supported events. In one example, the navigation keys of 'up' in remote controller or keyboard can be translated to touch or click on the position above the current scene in order to let car accelerate in an car contest application, or to force the game character to jump the obstructed Or the 'up' key may be interpreted to touch or click the button of current scene above the previous button. In another example, the navigation keys of 'right' in remote controller or keyboard can be translated to touch or click on current scene from a point to the right as effect of the slide towards right in purpose of switching to another window in a web application or scrolling to the next page in the e-book software. In another example, the navigation keys of 'down' in remote controller or keyboard can be simulated as the pinch-to-zoom touch gesture on current scene in order to adjust the image or use the online map application.

As the layout of the buttons and the virtual touch event in the scene is clear once the scene is recognized, the scene recognition plays the essential role in using remote controller, keyboard or some means to control digital television equipped with an OS with touch interface. The recognition algorithm requires the acquirement of the scenes previously in order to extract the characteristic information to distinguish the scene from the others. The data collected may be a set of characteristic points with unique color information in certain location, or a set of shape and gradient information, in suppose of identifying the scene in the application. When a scene is displayed on the screen, the sets of characteristic information will try to match the current scene and results declare that the scene belongs to the certain one in the application.

Brief description

The accompanying drawings, which are incorporated in and form a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the invention. In the drawings:

Figure 1 displays a normal scene of an application for recognition. Figure 2 displays another scene which possesses scatter touch-based events on buttons.

Figure 3 displays another scene where no apparent button exists while clicking on certain place may produce touch-based events.

Figure 4 is an example of scene which shows responding on slide beside tab or click

Figure 5 is another example of scene where the touch event of multiple fingers leads to the adjustment of the current scene.

Figure 6 illustrates how the scenes of application are preprocessed into two kinds of points set.

Figure 7 illustrates how the input of remote controller is simulated to touch-based event through the recognition of current scene.

Figure 8 illustrates how the keystrokes simulate to the touch-based output event with the help of preprocessed information, based on the scene recognition algorithm.

Figure 9 illustrates how the simulation manager sends the application identity to assist the recognition of scene and the simulation of keystrokes.

Figure 10 illustrates the process of the translation of events from keystrokes to touch-based events when operating the application.

Figure 11 illustrates the process of scene recognition algorithm.

Detailed description

Some of the operating systems are designed to be controlled around the touch user interface, since it is convenient for users to operate the system and applications in a cell phone or a pad. These programs rely on the touch-based event as their input source and therefore could only be operated by the means of touch or click operations for interaction.

Applications in the operating systems are always composed of several pictures with regular frame, and the picture which provides ways for users to interact with can be called as the scene. In order to control the scene, users may touch or click on buttons or certain place, slide in one direction, or pitch-to-zoom on the scene.

Figure 1 is an example of the scene with two icons and five buttons. When touching the button A 103, it will change the current scene or enter to a new scene. There are also other scenes in the application without apparent buttons and Figure 3 is an instance. It is an application of book reader which possesses no buttons at all on the scene, but it will scroll to the last page when touching the left side of scene shown in dotted circle 301, turn to the next page when touching the right side one 302 or evoke the quick settings when touching the bottom range 303.

Figure 4 is an example of scene which shows special responding on slide beside tab or click. It permits users to switch to the previous picture if sliding to the left 405 or the latter ones when sliding to the right. Figure 5 is another example of scene in the picture application or map application. When user touches the screen outward with two fingers simultaneously (503 and 504), the picture will be enlarged while touching the screen inward leads the picture to shrink.

Through the simulation, the user will map the keystrokes on the handset device, like infrared remote controller or keyboard, to the touch-based events on the screen so as to operate the application. The keystrokes used may be but not limited to the navigation keys, such as 'up', 'down', 'left', 'right', 'ok' and 'cancel'.

The map choice from the keystrokes on the handset device to the touch-based events depends on the layout of the touch-based events on the scene. If there is the slide operation that functions on scene, then the navigation keystrokes will map to the slide events with the corresponding direction. That is to say, the 'right' key will always map to the event of slide to right. Or the certain keystrokes could be translated to the events like touch, click, pinch-to-zoom on the certain location in the scene. For example, the keystrokes may correspond to each button separately, if the button locations are scattered. Figure 2 is a scene of game application wherein the touch of the upper right button 203 to jump, bottom right button 202 to go forward and bottom left button 201 to go backward. Considering the layout of these touch events, the keystrokes of 'up', 'right', 'left' would be mapped to the event of tapping on these buttons respectively.

In another circumstance, the layout of button is arranged orderly and the set of navigation keys can be simulated to the mouse event in order to control the scene. Since the buttons are located in a regular way, it is easy to locate each button through the mouse cursor based on the layout of buttons. If the button 'left' is pressed, the cursor will move to the left button compared to the previous one. When the targeted button is reached, the 'ok' keystrokes will be mapped to the touch or click event on the location of button. For example, in figure 1, press 'up' keystrokes can direct the mouse cursor from button C 104 to button A 103 and 'Right' directs to button D 105, etc. When the mouse cursor points to the button E 106, the press of 'Ok' key will be translated into the event of touch or click on the button wherein the mouse cursor stays and it is supposed to change the current scene or enter into a new one.

In another example, the scene in figure 4 makes use of the slide operations in order to operate the application. In detail, the slide from a point to its left 405 lead to the browse of previous pictures 402 and the slide from a point to its right lead to the observation of the latter pictures. Then, the keystrokes of 'left' and 'right' could be used to map to the corresponding slide events.

In another example, the scene in figure 5 may use the pinch-to-zoom events in order to adjust the picture. When the user presses the keystrokes of 'up', it may simulate an event of the touch on the screen outward with two fingers simultaneously (503 and 504), which leads the picture to be enlarged, while the keystrokes of 'down' could be mapped to the touching on screen inward, resulting to the picture to shrink.

In order to distinguish the scene from the others in an application, some characteristic signs should be collected from the scene in order for identification. Take figure 1 for example, there is a title 'ABE Games' 101 with the certain pixel information above the picture which is unique and can be distinguished from other scenes. In another aspect, the shape of buttons of curtain location is similar as the circle which can be identified through the image shape recognition algorithm, and therefore the scene can be picked out. Some other information such as the layout of scene, the pixel of the icons and buttons and gradient information could be selected as the characteristic information to identify the scenes.

Once the scenes have been recognized, the location of touch-based events on the scenes could be got. This is because that the amount of scenes of an application is limited and could be collected previously. After the scenes of application have been collected, the original supported input events, such as touch, click, slide and pinch-to-zoom, will be analyzed and collected as various kinds of data sets in order to be mapped with the keystrokes. Thus, if a scene has been recognized as one of the collected scenes, the keystrokes on the handset device can be simulated to native supported touch-based events on the screen.

In one example, as the layout of the buttons in a scene is clear, the navigation keys , which concludes 'up', 'down', 'left', 'right' and 'ok' can be used to map the touch event on each button. If the button locations are scattered, these keystrokes will correspond to the touch event on each button separately. Figure 2 is a scene of game application which users touch the upper right button 203 to jump, bottom right button 202 to go forward and bottom left button 201 to go backward. Once the scene is recognized, the keystrokes of 'up', 'right', 'left' will be mapped to the event of touching on these buttons respectively.

When the application is running, one of its scenes is displayed on the screen and the image data of the scene is stored in the memory occupied for framebuffer. Framebuffer is a device which drives the video output from the memory buffer containing image data to the display.

In order to operate the applications which are original designed around the touch interface with the handset device, the scene of applications should be collected previously 401. The scene will be collected from the framebuffer in turn when the applications are running. Then, two kinds of information should be acquired from the scenes collected:

the information set for scene recognition 402. The set is selected from the points which may be located at a range of area which is special on some feature such as color, shape, gradient or so and consequently distinguish the scene from others, and

the information set for simulation events 403. The set is used to simulate the input event from the keystrokes on the handset device to the corresponding touch or click operations based on the layout of the buttons of the scene.

These two kinds of preprocessing information is stored into information base respectively (404 and 405), which contributes to the recognition of current scene and translation of keystrokes to native-supported events.

Once the scene is acquired from the framebuffer 803, a scene recognition algorithm 804 is used to pick out the right scene identity with the help of pre-obtain information base of scene 805. After that, if the user presses a key on remote controller or keyboard, and the event is captured by keystrokes simulation engine 910. Based on the recognized scene and information base of the layout of the touch-based events 806, the engine then interpret the keystrokes, like 'ok', 'cancel', 'up' or so to the certain native-supported events like touch or click on buttons, a slide in one direction or pinch-to-zoom in some position. For example, if a key of up is pressed and informed with an input event 801, and then the keystrokes analysis 802 will produce an event of the touch or click on the certain place above the scene. In another instance, users may strike a key of 'left' and analysis engine 802 may interpret it as an event of slide towards left, which is likely to browse another image in a picture viewer application. In order to operate multiple applications which can be controlled by remote controller in the operating system, a simulation manager 908 is introduced in order to inform the keystrokes translation engine 910 with the application identity 909. Based on the identity, the keystrokes translation engine 910 will select the corresponding recognition characteristic information set from information base of scene 905 to identify the scenes of current applications and map the navigation keystrokes 901 to the various native-supported touch events 907, according to the information base 906 ,which describes the touch-screen event could be mapped. The simulator manager may be but not limited to an application, a system tool or the system component.

Considering that the layout of the buttons in a scene is different from the others and the location of buttons can be acquired previously, a scene recognition algorithm is requested to distinguish the scene and therefore is helpful to locate the touch-based events. The algorithm varies on different features on the scene like shape feature, gradient feature, pixel feather and so on. Although we will take the pixel feature as an instance, it is not limited to this feature in the disclosure. Initially, take the image information of each scene from framebuffer and collect the pixel information different from other scenes in the same location (Namely, characteristic points set). Every characteristic points set belongs to a scene and is signed as set S, (l^i^N, N stands for the amount of scenes) and is comprised of characteristic point Ay (l^j^ M, M stands for the amount of characteristic points in a scene) , where an element Ay possesses attributes R_xy and R_rgb to represent its information of location and pixel. The color attributes R_rgb use the tricolor: red, green and blue to match whether the color is equal or not in the certain location, and it is easy to get from the framebuffer which stores the image scene in the memory in the form of R, G, B. As the point on a scene constitutes a characteristic points set, the whole scene of the applications forms a superset S and S, e s. Then, the algorithm can be described as below:

a) When the application is running, acquire the image data of the current scene from framebuffer and sign it as P. Then variable i is valued as 1, turn to b);

b) Based on the attributes of R_xy of elements in characteristic set S,, read out every element A's actual color value (signed as B_rgb) in the position R_xy from P. If it exits one element whose color of R_rgb is not equal to B_rgb, then turn to c), otherwise turn to d);

c) Evaluate i to i+1 and compare the value of i and N. If i is not bigger than N, then try to match the set of S,, the next set, and jump to b). Otherwise, state the failure in recognition of the current scene and leave the system.

d) Declare the success in identifying the current scene.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

Claimed

1. Scene recognition based virtual touch event method, comprising:

the respective information bases of touch-based events and characteristic measurement collected from the scenes of applications which have been acquired previously. a scene recognition algorithm to recognize the current scene of the application with the help of information base of characteristic measurement. the map of the keystrokes of the handset devices to the touch-based events, based on the recognized scene and the information base of touch-based events.

The method of claiml, wherein the scene stands for one of the view of the application which provides means for users to interact with.

The method of claiml, wherein the acquirement of the scenes in applications comes from the data which is located in memory and controlled by framebuffer.

The method of claim 1, wherein the characteristic measurement of the scene is the points set with the information of several points that can identify the scene. The information of points set can be the pixel, shape of buttons, gradient, etc.

5. The method of claim 1, wherein the scene recognition algorithm involves:

every group of characteristic points set belongs to a scene and is signed as set S, (l^i^ N, N stands for the amount of scenes). The set is comprised of characteristic point Ay (l^j^ M, M stands for the amount of characteristic points in a scene) , which possesses attributes R_xy and R_rgb to represent its information of location and color. The color attributes R_rgb use the tricolor: red, green and blue to match whether the color is equal or not, and it is easy to get from the framebuffer which stores the image scene in the memory in the form of R, G, B. As the points on a scene constitute a characteristic points set, the whole scene of the applications forms a superset S and S, e s. Then, the algorithm can be described as below:

a) When the application is running, acquire the image data of the current scene from framebuffer and signs it as P. Then variable i is valued as 1, turn to b);

b) Based on the attributes of R_xy of characteristic set S,, read out every element A's actual color value (signed as B_rgb) in the position R_xy from P. If it exists one element whose color of R_rgb is not equal to B_rg , then turn to c), or turn to d);

d) Declare the success in identifying the current scene.

6. The method of claim 1, wherein handset device may be the portable device like infrared remote controller, joy stick, wireless keyboard and so on.

7. The method of claim 1, wherein touch-based event means the event is produced by touching the screen/pad with the effect of the tab, slide and so on.

8. The method of claim 1, wherein the scene will be recognized with the data image of current scene from framebuffer only after users press the keystrokes of device.

9. The method of claim 1, wherein the map of the input keystrokes depend on the analysis of the information base of characteristic measurement to the scene recognized and the layout of touch-based events.

10. The method of claim 1, wherein the input keystrokes may be but not limited to navigation keys, namely the 'up' key, 'down' key, 'left' key, 'right' key, 'ok' key and 'cancel' key.

11. The method of claim 10,

wherein the layout of the touch-based events on the scene may be scattered and therefore the navigation keystrokes of the input device can be mapped directly to touch or click the certain location of the button, and wherein the layout of the touch-based events on the scene may be arranged orderly and the navigation keystrokes of the input device acts as the movement of mouse cursor to another button from the previous one, according to the meaning of the keystrokes, and it will not touch or click the button only when press the 'ok' keystroke, and wherein the navigation keys may be mapped to the operation of the slide to certain directions, or the touch on screen inward or outward with multiple fingers simultaneously.

12. Keystrokes translation engine requires knowing the running application's identity in order to manage the multiple applications with the handset device. The identity is informed by the simulation manager which may be but not limited to an application, a system tool or the system component.