1 Title [0001] System and method of implementing a software game controller on a smart phone or tablet. Technical Field [0002] This invention relates to a game controller. In particular, this invention relates to a software system which implements utilizing a smart phone or tablet as a game controller. Background Art [0003] Smart phones and tablets are extremely popular nowadays in many countries. There are hundreds of thousands of games are available in App stores over the two main platforms: Android and iOS. People can download and play a game on the smart phone or tablet, with using the touch screen or gravity sensors to control the game which is displayed on the same screen. Some physical game controllers had been available to separate the screen and the control, which normally uses Bluetooth to connect with the device. In this way, it can greatly reduce the interference between the game display and the control, although the small screen cannot be changed. [0004] Along with the increasing popularity of digital contents, now Smart TVs, TV boxes, TV dongles, Mini Computers and so on are entering people's life, which are not only used to watch a TV program or a movie, but also used to play games. Particularly, the Android devices are becoming more and more popular because of its open-source feature and the great number of App resources over the platform, including games. Now Google is moving its Android mobile resources onto a TV screen. It is foreseen that, in the near future, people would be able to play all sorts of games on a TV screen with a Smart TV or a top-set box, and most of these devices would be Android based. A game controller is an essential part to play the game no matter it is a physical one or software one.
2 Summary of Invention [0005] This invention implements a software game controller on a smart phone or tablet to control a game console device which is based on Linux-kernel such as Android TV sets and Android TV boxes. [0006] Briefly summarizing the invention, it includes a system and a method. The system is in client-server architecture where a client application is installed on a smart phone or tablet and a server application is installed on the game console device; it simulates the way in which a physical controller works, that the client mobile application captures events from the GUI (Graphical User Interface) and convert them into the events which a physical controller uses; and then send them to the controlled device; the server dispatches the received events to the local operating system; before starting capturing the GUI events, the target device must be found first and a connection must be established as well. The method is about how to utilize the smart phone or tablet's movements to control a game. [0007] By this invention, the user can simply use his or her smart phone or tablet to play a game over the screen, without having to buy a physical controller. In addition, the invention enables users to control a game by swinging the smart phone or tablet rather than pressing buttons, which can bring a different experience that an ordinary game controller is unable to achieve. [0008] The technologies and methods involved in this invention are not limited to the devices or systems mentioned in this document; for example, the event conversion method can be used for any systems. Drawings [0009] Figure 1 shows the general system devices. [0010] Figure 2 shows the general work flow involving six processes. [0011] Figure 3 shows the key components of the system.
3 [0012] Figure 4 shows a game controller GUI which simulates the traditional game one. [0013] Figure 5 shows the gravity GUI which uses the gravity sensor events to control directions. [0014] Figure 6 shows the effect on the gravity GUI when the gravity sensor event is converted to the "Down" event of the direction key "KEYLEFT". [0015] Figure 7 shows the effect on the gravity GUI when the gravity sensor event is converted to the "Down" event of the direction key "KEYRIGHT". [0016] Figure 8 shows the effect on the gravity GUI when the gravity sensor event is converted to the "Down" event of the direction key "KEYUP". [0017] Figure 9 shows the effect on the gravity GUI when the gravity sensor event is converted to the "Down" event of the direction key "KEYDOWN". [0018] Figure 10 shows the packet format for sending the converted events from the client to the server. Invention Embodiments [0019] The invention includes a system in client-server architecture and a method of converting gravity sensor events to the key events. For the system, the client is a mobile application which can be installed on a smart phone or tablet, regardless of the platform; the server is a service application which is installed on the game console device; and the communication involves existing technologies such as TCP, UDP and Bluetooth. Figure 1 shows the involved devices in the system. For the conversion method, it is about how to utilize the smart phone or tablet's gravity sensor or similar sensors to control a game. [0020] The system comprises six main processes, as shown in Figure 2, including "Find 4 Device", "Establish Connection", "Capture Events", "Convert to Key Events", "Send to the Device" and "Dispatch Key Events". The key components involved in the six processes are shown in Figure 3, including GUI, Device Finder, Event Handlers, Key Event Sender at the client side; and Broadcast Responder, Key Event Receiver and Key Event Dispatcher at the server side. [0021] The first process is to find the controlled device. This can be implemented by UDP broadcast in the local network. To find a device, the client's Device Finder sends a UDP package containing a pre-defined flag such as "100" in one byte, with a destination IP address "255.255.255.255" and a pre-defined port number like "1001"; after that, the Finder will keep listening to a pre-defined port like "1002" of the UDP socket for a response; the device's Broadcast Responder always listens to the pre-defined port ("1001") of a UDP socket and when it receives a broadcast with the "100", it sends a response with the device name and ID to the sender by the pre-defined port ("1002") and the sender's IP address; the client will keep a target list containing the found devices and the corresponding information like name, ID and IP address. This is a universal way to find a device which is supported by nearly all the current operating systems. By Bluetooth, different operating systems may provide different APIs, and it can just utilize these APIs to find a device; for example, for an Android mobile device, "BluetoothAdapter" class is used to find a Bluetooth device and at the server side, it just needs to set the device discoverable. [0022] The second process is to establish a connection. In a universal way, it can be a TCP connection. Open a TCP socket connection by the target device's IP address and a pre defined port number. When the connection is opened successfully, a verification code such as "101" in one byte can be sent to the device to avoid unauthorised connections. If the device's Key Event Receiver receives the verification code, it will keep the connection and start to listen to the input stream waiting for key events; otherwise just drop the connection. At the client side, the Key Event Sender will keep the output stream preparing to be called to send key events. By Bluetooth, it is the same with finding a device that, using the provided AlPs to establish a connection; for example, for an Android mobile device, the "BluetoothSocket" can be used to establish the connection and communicate with the "BluetoothServerSocket" at the server side.
5 [0023] The third process is to capture events from GUI. As shown in Figure 4, the GUI is a simulated physical game controller, containing the common game control keys. To capture the events of these keys are simply registering an "OnTouchListener" on the view and capture the "Down" and "Up" events in a CallBack function or method "onTouch". Figure 5 shows a special layout of the keys excluding the "Start" and "Select" keys. This GUI is for gravity control. In this case, the four direction keys are put in the centre in a small size. They are not used as touch buttons but used for displaying the effects of the gravity control. For example, when the gravity events are calculated to a "Left Key Down Motion", the "Left Key" (20 in Figure 6) will be highlighted meaning the direction control key "Left" is pressed. The gravity events can be captured by a "SensorEventListener" and the CallBack function or method "onSensorChanged". Different operating systems may have different names but they do the same things. [0024] After capturing the touch or gravity events, the next process is to convert them to the game control key events which can be recognized by the operation system on the game console device. Every operation system may have its own definitions on the key values, which are mapped to the kernel level key values, for example, in Android, the "Direction Up" key is recognized as "KeyEvent.KEYCODEDPADUP" and the corresponding integer value is 19 while the Linux kennel defines it as "KEYUP" which has a value of 103. In this system, the original key values in kennel level are used, which are defined as the following. 1. KEYUP: 103 (11 in Figure 4) 2. KEYDOWN: 108 (10 in Figure 4) 3. KEYLEFT: 105 (9 in Figure 4) 4. KEYRIGHT: 106 (12 in Figure 4) 5. BTN_X: 307 (15 in Figure 4) 6. BTN_Y: 308 (16 in Figure 4) 7. BTN_A: 304 (18 in Figure 4) 8. BTN_B: 305 (17 in Figure 4) 9. BTNTL: 310 (7 in Figure 4) 10. BTNTR: 311 (8 in Figure 4) 6 11. BTN_START: 315 (13 in Figure 4) 12. BTNSELECT: 314 (14 in Figure 4) There are two events on each of the keys above: Up and Down with the integer codes 0 and 1 respectively. [0025] To convert gravity events to the key events, it is to calculate the offset of the sensor movement from the balance point. Every captured "SensorEvent" has three float values which represent the movement in X, Y and Z dimensions respectively. For this application, only the X and Y dimensions are required. Y value is used to judge KEYLEFT and KEYRIGHT events; and X value is used to judge KEYUP and KEYDOWN events. Figure 6 to Figure 9 show the movements which trigger the direction key events. The conversion involves two values x and y, two flags last_x, last-y, and one gap. x and y are the values in X axis and Y axis in the coordinate when capturing each SensorEvent; last_x and last y are two flags recording the values of previous x and y; and the gap is used to keep a movement space where no key events are triggered. Take the KEYLEFT and KEYRIGHT events as an example, the simplest algorithm is that, If y<0 then If last_y>0 then send KEYRIGHT Up event and KEYLEFT Down event Else if y>0 then If lasty<0 then send KEYLEFT Up event and KEYRIGHT Down event With the algorithm above, it is always switching between KEYLEFT and KEYRIGHT keys because 0 is the balance point. To keep a movement range where no key events are triggered, the gap value can be used as the following: If y<(0-gap) then If last_y> (0-gap) then send KEYLEFT Down event Else if y>(0-gap) & y<gap then If last_y< (0-gap) then send KEYLEFT Up event If last_y>gap then send KEYRIGH Up event Else if y>gap then If last_y<gap then send KEYRIGHT Down event The gap's value decides the range of swinging the device to control "left" and "right". This conversion method can be applied in any mobile devices which have the gravity sensor or 7 similar sensors. For an advanced controller, the Z value can be also used to add one more dimension control. [0026] After a key event has been converted, the next process is to send it to the server. Four bytes are sufficient to represent all the key events. As shown in Figure 10, the first byte is used to store the event type; the second byte and the third byte together is used to store the key value; the last byte is used to store the status: 0 means Down, 1 means Up. Every converted event is sent to the server and the server will check the event type and dispatch the events accordingly. The event type may be used to distinguish a connection verification message or messages with specific purposes from the real events. [0027] The last process is to dispatch the events. There are two ways to dispatch the events: by the operating system's APIs and by "uinput" in Linux kernel level. The operating system like Android has its own mechanism to dispatch all the events and the corresponding APIs can be used to achieve the job. For example, in Android, "Instrumentation" class has a method "sendKeySync(KeyEvent event)" which can be used to dispatch any key events to the currently focused view; another way is to use "IWindowManager" class to "injectKeyEvent" but it is an unpublished API. The second method to dispatch the events is to use "uinput" in Linux kernel level by JNI (Java Native Interface). First of all, create an input device by C language: "open("/dev/uinput", ORDWR)" ; then directly write the event in a system struct "uinput event", for example, "write(device, event, sizeof(event)" where the "device is the opened device reference, and the "event" is the system struct "uinput event" which contains the key value and the status. The "uinput" method can be applied on any Linux kernel based operating systems.