US20160085419A1 - Simultaneous input across multiple applications - Google Patents

Simultaneous input across multiple applications Download PDF

Info

Publication number
US20160085419A1
US20160085419A1 US14/857,391 US201514857391A US2016085419A1 US 20160085419 A1 US20160085419 A1 US 20160085419A1 US 201514857391 A US201514857391 A US 201514857391A US 2016085419 A1 US2016085419 A1 US 2016085419A1
Authority
US
United States
Prior art keywords
user input
application
computing device
input
display device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/857,391
Inventor
Curt Carpenter
Debora Everett
Doug Kramer
Derek E. Sunday
Chris Whytock
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Priority to US14/857,391 priority Critical patent/US20160085419A1/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CARPENTER, CURT, EVERETT, DEBORA, KRAMER, DOUG, SUNDAY, DEREK E., WHYTOCK, CHRIS
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Publication of US20160085419A1 publication Critical patent/US20160085419A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842Selection of displayed objects or displayed text elements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03545Pens or stylus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0412Digitisers structurally integrated in a display
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482Interaction with lists of selectable items, e.g. menus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/451Execution arrangements for user interfaces
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04104Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger

Definitions

  • Interaction with computing devices has changed significantly since the inception of the graphical user interface.
  • Users may control a variety of functions and interact with numerous types of applications through a graphical user interface using one or more associated hardware input devices.
  • computing devices are often used for gaming and allow multiple users to play the same game at the same time.
  • Various input devices include joysticks, touch-sensitive screens, keyboards, mice and the like. Users may open several different applications at the same time and multi-task by switching between applications and viewing multiple documents simultaneously. Input events are determined and rendered based on a currently focused application of the user interface and the type of user input. Thus, in one situation, the pressing of keys on a keyboard may correspond to entering text when a word processing application is active (i.e., has input focus).
  • Embodiments include methods, systems, and computer program products for processing multiple inputs received across simultaneously displayed applications.
  • a computing device displays a first application on a first portion of a display device, while simultaneously displaying a second application on a second, different, portion of the display device.
  • the computing device receives a first user input in connection with the first portion of the display device, and also receives a second user input in connection with the second portion of the display device.
  • the first user input and the second user input are received at substantially the same time.
  • the computing device routes the first user input to the first application based on the first user input having been received in connection with first portion of the display device, and routes the second user input to the second application based on the second user input having been received in connection with the second portion of the display device.
  • the second user input is routed to the second application substantially at the same time as routing the first user input to the first application
  • a computing device displays a first application in a first portion of the display device. While displaying the first application in the first portion of the display device, the computing device simultaneously displays a second application in a second and different portion of the display device. The computing device simultaneously receives, at one or more user input devices, both (i) a first user input that is detected by the one or more user input devices to be directed at the first portion of the display device, and (ii) a second user input that is detected by the one or more user input devices to be directed at the second portion of the display device.
  • the computing device simultaneously routes both (i) the first user input to the first application based at least on the first user input being detected to be directed at the first portion of the display device, and (ii) the second user input to the second application based at least on the second user input being detected to be directed at the second portion of the display device.
  • FIG. 1 illustrates a schematic diagram of a general-purpose digital computing environment that can be used to implement various aspects of the present disclosure.
  • FIG. 2 is a diagram of interaction by multiple users with user interfaces of multiple applications using a single input device according to one or more aspects described herein.
  • FIG. 3 is a diagram of user interaction with one or more applications using multiple input devices according to one or more aspects described herein.
  • FIG. 4 is a block diagram of a multiple input aware application system according to one or more aspects described herein.
  • FIG. 5 is a flowchart illustrating a method for handling multiple simultaneous user inputs for a plurality of different applications according to one or more aspects described herein.
  • FIG. 6 is a flowchart illustrating a method for processing input from multiple input hardware systems according to one or more aspects described herein.
  • FIG. 7 is a flowchart illustrating a method for routing input signals and events according to one or more aspects described herein.
  • FIG. 1 illustrates a schematic diagram of a general-purpose digital computing environment.
  • a computer 100 includes a processing unit 110 , a system memory 120 , and a system bus 130 that couples various system components including the system memory 120 to the processing unit 110 .
  • the system bus 130 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
  • the system memory 120 may include read only memory (ROM) 140 and random access memory (RAM) 150 .
  • ROM read only memory
  • RAM random access memory
  • a basic input/output system 160 which contains the basic routines that help to transfer information between elements within the computer 100 , is stored in the ROM 140 .
  • the computer 100 also may include a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190 , and an optical disk drive 191 for reading from or writing to a removable optical disk 199 , such as a CD ROM or other optical media.
  • the hard disk drive 170 , magnetic disk drive 180 , and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 192 , a magnetic disk drive interface 193 , and an optical disk drive interface 194 , respectively.
  • These drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for the personal computer 100 . It will be appreciated by those skilled in the art that other types of computer-readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment.
  • RAMs random access memories
  • ROMs read only memories
  • a number of program modules can be stored on the hard disk drive 170 , magnetic disk 190 , optical disk 199 , ROM 140 , or RAM 150 , including an operating system 195 , one or more application programs 196 , other program modules 197 , and program data 198 .
  • a user can enter commands and information into the computer 100 through input devices, such as a keyboard 101 and pointing device 102 (such as a mouse).
  • Other input devices may include a microphone, joystick, game pad, satellite dish, scanner, or the like.
  • serial port interface 106 that is coupled to the system bus 130 , but they also may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB), and the like. Further still, these devices may be coupled directly to the system bus 130 via an appropriate interface (not shown).
  • a monitor 107 or other type of display device also may be connected to the system bus 130 via an interface, such as a video adapter 108 .
  • personal computers typically include other peripheral output devices (not shown), such as speakers and printers.
  • a stylus digitizer 165 and accompanying stylus 166 are provided in order to digitally capture freehand input.
  • the digitizer 165 may be directly coupled to the processing unit 110 , or it may be coupled to the processing unit 110 in any suitable manner, such as via a parallel port or another interface and the system bus 130 as is known in the art.
  • the digitizer 165 is shown apart from the monitor 107 in FIG. 1 , the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 107 . Further still, the digitizer 165 may be integrated in the monitor 107 , or it may exist as a separate device overlaying or otherwise appended to the monitor 107 .
  • the computer 100 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 109 .
  • the remote computer 109 can be a server, a router, a network PC, a peer device or other common network node, and it typically includes many or all of the elements described above relative to the computer 100 , although for simplicity, only a memory storage device 111 has been illustrated in FIG. 1 .
  • the logical connections depicted in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113 .
  • LAN local area network
  • WAN wide area network
  • the computer 100 When used in a LAN networking environment, the computer 100 is connected to the local area network 112 through a network interface or adapter 114 . When used in a WAN networking environment, the computer 100 typically includes a modem 115 or other means for establishing a communications link over the wide area network 113 , such as the Internet.
  • the modem 115 which may be internal or external to the computer 100 , may be connected to the system bus 130 via the serial port interface 106 .
  • program modules depicted relative to the personal computer 100 may be stored in the remote memory storage device.
  • network connections shown are examples, and other techniques for establishing a communications link between computers can be used.
  • the existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP, UDP, and the like is presumed, and the computer 100 can be operated in a user-server configuration to permit a user to retrieve web pages from a web-based server.
  • Any of various conventional web browsers can be used to display and manipulate data on web pages.
  • FIG. 1 environment shows one example environment, it will be understood that other computing environments also may be used.
  • an environment may be used having fewer than all of the various aspects shown in FIG. 1 and described above, and these aspects may appear in various combinations and subcombinations that will be apparent to one of ordinary skill. Additional elements, devices or subsystems also may be included in or coupled to the computer 100 .
  • FIG. 2 illustrates a touch-sensitive input system for receiving multiple simultaneous user inputs intended for different applications.
  • the system includes a multiple input hardware system such as touch-sensitive screen 200 (e.g., combination of screen 107 and digitizer 165 ) displaying a user interface 201 and applications 205 and 206 .
  • User interface 201 further includes a plurality of graphical interface elements including interfaces associated with application A 205 and application B 206 .
  • a user must first choose an application with which to interact before entering input.
  • user interface 201 and/or the underlying operating system or application routes all subsequently received user input to the selected application.
  • multiple users may simultaneously interact with multiple applications without having to initially choose an application with which to interact. In other words, users of the same device may interact with different applications independently of and simultaneously with one another. This allows multiple users to work with multiple different applications using a single shared computing device.
  • user 210 may play an electronic memory card game at the same time user 211 is engaging in a game of minesweeper. Specifically, user 210 contacts one area of display 200 corresponding to the memory game application simultaneously with user 211 touching a different location of display 200 associated with the minesweeper game. The user inputs from both user 210 and user 211 are delivered simultaneously to each application. Significantly, neither user 210 nor user 211 needs to change input focus from the memory game application to the minesweeper application or vice versa in order for input from each user to register with their respective application. Since some computing devices process input and data sequentially, true simultaneity might not be achieved.
  • simultaneous is used herein to refer to a difference in time that is virtually imperceptible to one or more users of a computing device.
  • a first input and a second input delivered to an application or rendered on an application user interface simultaneously may be delivered sequentially in such a way that the user perceives the delivery or rendering as simultaneous.
  • multiple input capable hardware such as touch-sensitive input display 200 rather than traditional single input hardware devices like current joysticks and mice.
  • Multiple input hardware refers to devices which are capable of detecting and transmitting simultaneous user inputs at the same time.
  • a touch sensitive display device may detect simultaneous inputs by one or more users by capturing an image of user inputs (i.e., capture the image of light reflecting off of multiple touch contacts on the surface) and parsing the image to determine input information.
  • Input devices transmit detected input signals to a destination device or application in periodic transmission intervals or frames. With traditional single input hardware devices, multiple simultaneous inputs are sent to an application in sequential transmission frames.
  • the correspondence between a user input and a particular application may be determined based on a user input location and an application location and size.
  • User interface 201 or an underlying operating system may determine whether touch input from user 210 is directed to application 205 or application 206 based on the input location and the locations and sizes of applications 205 and 206 .
  • a table containing application locations and sizes may be stored.
  • a correspondence may be established between the input and application 205 using the application location table.
  • the touch input may be routed or sent to application 206 based on the aforementioned determination techniques.
  • input routing may be based on other heuristics including the type of input. For example, one application may receive all inputs that are recognized as finger input and another application may receive the input if the input is determined to originate from a stylus. Inputs may also be delivered to all applications in cases where either the input applies to all applications or where it is not possible to determine which application is interested in the input. In one example, a special gesture may correspond to a “hide all applications” command. In another example, a user may roll dice on an input device. Since the user may not be able to entirely control where the dice lands, the dice input may be transmitted to all applications.
  • FIG. 3 illustrates various aspects of a system 300 for processing multiple inputs across multiple applications.
  • System 300 includes a computing device (e.g., computer 100 ) having a touch-sensitive display screen 303 (e.g., combination of screen 107 and digitizer 165 ) oriented such that touch-sensitive display 303 may be used as the surface of a table or workspace. Users may be seated around the display screen 303 of computing device and simultaneously interact with applications 315 , 316 , 317 and 318 by contacting corresponding locations or portions of touch-sensitive display screen 303 .
  • a computing device e.g., computer 100
  • touch-sensitive display screen 303 e.g., combination of screen 107 and digitizer 165
  • Users may be seated around the display screen 303 of computing device and simultaneously interact with applications 315 , 316 , 317 and 318 by contacting corresponding locations or portions of touch-sensitive display screen 303 .
  • a user may play a memory card game 318 by touching a first portion of screen 303 while one or more users are drafting a document on word processing application 317 , playing a second memory card game 316 and/or drawing a picture in paint application 315 by contacting a second, third and/or fourth portion of screen 303 .
  • the user inputs may be differentiated and routed to the appropriate application based on the location of user input and the locations and sizes of the relevant applications. Input routing may be performed via a variety of other mechanisms as well.
  • two applications may be linked such that input between the two or more applications is synchronized.
  • user input to application 318 may be synchronized and registered with application 316 and vice versa.
  • Synchronization may include determining a time of receipt for each user input and sequencing the order in accordance with the determined receipt times.
  • two users playing a competitive memory card game may simultaneously interact with two separate application instances, such as applications 316 and 318 , of the memory game application.
  • the game may display the same field of cards for both application instances 316 and 318 and each application instance 316 and 318 may register input from both players.
  • simultaneous card selections from the players may be registered simultaneously on both applications 316 and 318 .
  • the synchronization of the user inputs ensures that a single card selection is not registered with both players. For example, once a pair of cards has been eliminated by a first player, the eliminated pair of cards is removed from both application instances 316 and 318 so that a second user cannot select that pair of cards. Additionally, input corresponding to flipping a card on application 315 may be simultaneously registered on application 318 (i.e., the input event associated with the card flipping is processed by application 318 as well).
  • the synchronization of application instances 316 and 318 may be coordinated by a third application (not shown). Each application may then communicate their inputs (i.e., selected cards) back to the third application.
  • a coordinating third application provides for a situation where two or more players attempted to claim the same matched cards at the same time. In such a situation, the third application may determine the winner of the matched cards based on which player obtained the next match.
  • each user of system 300 may simultaneously play a scrabble board game generated by interface 301 .
  • Each user may be assigned a block holder application that displays each user's scrabble blocks.
  • Another application may be generated for displaying the scrabble board on which the blocks are to be placed.
  • Users may place blocks on the scrabble board by initially touching a location on display 303 corresponding to the location of a particular scrabble block in the block holder application and subsequently touching a location of display 303 corresponding to a desired placement location in the scrabble board application. Simultaneously, other users may be rearranging their blocks by selecting a block and subsequently choosing a desired placement location in their block holder.
  • the multiple inputs from the users are differentiated and routed based on the users' input location.
  • Other techniques may also be used to route the user's input including routing input based on a unique characteristic of the user's input. For example, a user may use his or her finger to provide touch input. Accordingly, the user's input may be routed based on an identification of the user's fingerprint.
  • System 300 may further include input devices such as devices 310 , 311 , 312 and 313 .
  • input devices 310 , 311 , 312 and 313 may include a variety of device types such as joysticks, mice, keyboards and the like.
  • Input devices 310 , 311 , 312 and 313 may further be of the same type or of different types depending on the applications with which the devices are associated. For example, in an electronic video game environment, each application interface 315 , 316 , 317 and 318 may be similar and require the users to use the same type of device, e.g., a joystick. However, if multiple users are working with different applications, the device types may differ.
  • These input devices may act as supplemental input sources to touch-sensitive display 303 or, alternatively, function as the primary input device.
  • Each of input devices 310 , 311 , 312 and 313 may further be associated with one of applications 315 , 316 , 317 or 318 .
  • the associations may be chosen based on a user selection, input device type or a host of other factors.
  • Indicators 325 , 326 , 327 and 328 are generated and displayed to allow multiple users to recognize and identify which applications correspond to which input devices. For example, indicators 325 , 326 , 327 and 328 may display a number that corresponds to a port on the computing device.
  • the input devices, themselves, may also include an indicator such as an indicator light that identifies the device-application association.
  • the input device-application association may be modified in a variety of ways including changing the settings in a control panel (not shown) and/or altering settings on the input device hardware.
  • FIG. 4 is a block diagram illustrating a system for processing input from single input 405 and multiple input 406 hardware sources across a series of applications 401 , 402 and 403 .
  • Single input devices differ from multiple input devices in that single input devices transmit simultaneous input sequentially while multiple input devices may transmit simultaneous input simultaneously in the same transmission frame.
  • input from single input devices are processed by an operating system (OS) event thread 415 , 416 or 417 depending on which application currently has focus.
  • OS Event processing threads 415 , 416 and 417 are limited to sequentially processing input events for a single application at a time. Thus, OS Event processing threads 415 , 416 and 417 are incapable of processing input corresponding to users contacting multiple locations of a touch-sensitive display at the same time.
  • OS operating system
  • Applications A 401 , B 402 and C 403 each correspond to an OS Event processing thread 415 , 416 and 417 and an application user interface (UI) 410 , 411 and 412 , respectively.
  • applications B 402 and C 403 further include secondary event processing threads 418 and 419 , respectively, in order to bypass the limits of OS Event processing threads 415 , 416 and 417 and in order to process simultaneous input received from multiple input hardware system 406 .
  • Secondary event processing threads 418 and 419 unlike OS Event threads 415 , 416 and 417 , are not focus oriented and may thus process multiple simultaneous input received from a multiple input device such as system 406 .
  • secondary event processing threads 418 and 419 operate in parallel to OS Event processing threads 416 and 417 .
  • Each input signal detected in the multiple input hardware system 406 is initially evaluated to determine a corresponding or receiving application.
  • Application correspondence of each input signal may be determined based on a location of input and/or any one of a plurality of other factors.
  • a first user input at a first location of a touch-sensitive display may be associated with and routed to a different application than a second user input received at a second location of the same touch-sensitive display.
  • Input location and application location and size may be matched in order to determine appropriate correspondencies.
  • Each user input may then be processed by either secondary event thread 418 or 419 depending on the determined application correspondency of each input.
  • the multiple input hardware system 406 may include several types of input devices and each type of input may further be associated with a different application. Accordingly, the input signals may be routed based on, at least in part, the type of input. Accordingly, each secondary processing thread 418 and 419 may receive a plurality of input signals detected by the input hardware system 406 and perform parsing functions to determine which signals are relevant to their underlying application. Event routing may further be performed by a third party router component that is implemented between multiple input hardware system 406 and applications 402 and 403 . Additionally, each secondary processing thread 418 and 419 may further be responsible for determining input events corresponding to the received input signals associated with their application, i.e., B 402 and C 403 . Other processing steps may be implemented separately or in combination to properly render the input signals and/or changes on application UIs 411 and 412 . In one or more configurations, multiple input hardware system 406 might not provide input or event routing.
  • a single input hardware device 405 may be used in combination with the multiple input hardware system 406 and multiple input aware applications 411 and 412 .
  • an event router 420 may be added between the hardware 405 and multiple input aware applications 411 and 412 .
  • the event router 420 receives input events, generated by the OS, from single input hardware device 405 and determines an application to which the input event should be sent. The determination may be based on a subscription design pattern whereby multiple input applications 411 and 412 submit requests for input from the event router 420 .
  • the subscription requests may identify a particular area of the display screen that corresponds to the requesting application, a corresponding input type or the like.
  • the event router 420 in response to the input requests, sends the appropriate input events to the requesting application.
  • a user may play a video game on a computing device using a joystick input device while another user performs word processing tasks using a keyboard input device simultaneously on the same computing device.
  • secondary event processing threads 418 and 419 upon determining an input event from one or more received input signals, may send the input event to Application OS Event processing threads 416 and 417 , respectively, for synchronization.
  • the input events determined by the secondary threads 418 and 419 may be synchronized to one or more events currently in OS Event threads 416 and 417 , respectively. Synchronization ensures that events are properly rendered on an application user interface by determining a sequence of events based on a receiving time of the user input.
  • a computing device may be connected to both a multiple input hardware system such as system 406 as well as a single input device such as hardware 405 . Events detected and determined from both single input hardware 405 and multiple input system 406 may then be synchronized by, e.g., ordering the input events according to received time and ensuring appropriate rendering on a user interface of application 411 .
  • FIG. 5 is a flowchart illustrating a method of handling multiple simultaneous inputs directed at multiple different applications.
  • a first user input and a second user input are received or detected simultaneously.
  • the inputs may correspond to users touching different locations of a touch-sensitive display or, in another example, the inputs may be associated with detected contacts with different sections of a touch input device, such as a touchpad or touch screen.
  • input locations are determined for each of the first and second inputs in steps 510 and 515 .
  • the input location may correspond to the location of user contact, i.e., input location.
  • the input location information is then processed in step 520 to determine one or more applications that correspond to the first and second user inputs.
  • the determination may involve a further determination of locations and sizes for each of the applications displayed on the user interface.
  • An application may then be identified for each user input based on matching the input location with the location and size of the application.
  • the user inputs may correspond to different applications or the same applications, again, based on the locations of user input and of the applications. Once applications corresponding to each user input have been identified, each user input is sent to the identified corresponding application in step 525 .
  • FIG. 6 is a flowchart showing a method for processing input from a multiple input system across at least two applications.
  • a multiple input hardware system or device is detected by an application.
  • a hardware connection may be detected by an underlying operating system and broadcast to applications. Any one of numerous detection methods known in the art may be used or implemented in conjunction with this step.
  • a determination is made to evaluate whether the detected input system is associated with the application.
  • the input system may be associated with an application by, e.g., altering control settings and associations in the operating system. If the input system is not associated with the application, the application does not process input signals from the detected system.
  • a secondary processing thread corresponding to the input system is created in step 610 .
  • the secondary processing thread is also created to run in parallel with an event processing thread of the operating system.
  • the secondary processing thread may be created when the application is initialized.
  • step 620 the secondary thread processes the user input to determine one or more input events.
  • the secondary thread identifies an action corresponding to the user input that is to be performed on the application user interface.
  • input signals from a touch sensitive display screen may reflect a change in position of a user's finger or stylus.
  • the change in a user's finger or stylus position may be determined by the secondary processing thread upon receiving the corresponding input signals.
  • Input signals may further include multiple events such as selection and movement events corresponding to a select-and-drag interaction by the user with an element of the user interface.
  • step 625 another determination is made to decide whether the determined input event or events should be synchronized with one or more events of the operating system event processing thread.
  • various user interface components such as user interface controls might be designed for access through OS Event processing threads.
  • input from a multiple input device directed toward such interface components may be synchronized with input events in the OS Event processing thread to properly register the input.
  • the application receiving the multiple input events may be responsible for determining whether the events need to be synchronized.
  • the input events determined by the secondary processing thread are to be synchronized with the events of the OS Event processing thread, the input events of the secondary thread are sent to the OS Event thread in step 630 .
  • the OS thread may then synchronize, e.g., arrange the events in order of time received, and perform further processing before sending the events to the application UI in step 635 .
  • the events are sent to the application user interface, as in step 635 , without initially passing through the OS Event thread.
  • FIG. 7 is a flowchart illustrating a method of routing input signals and/or input events.
  • the user input is evaluated to determine various characteristics of the input. These characteristics may include a type of input, duration of input, input location, identification of the input source and the like.
  • applications active on the user interface and/or subscribing to input from the input source are identified. For example, in one or more configurations, the user interface or underlying operating system may maintain a database of applications that subscribe to particular input devices or systems.
  • characteristics of the identified applications are determined. Application characteristics may include application type (e.g., game, tool, word processing, etc.), application location and application control and input type.
  • an electronic video game for example, may be identified as an application that subscribes to input from a particular joystick connected to the computing device.
  • the received input is sent to the one or more applications in step 720 . If, however, no applications subscribe or request the input, another determination may be made, in step 725 , to evaluate whether the determined input characteristics matches any of the characteristics of the identified applications.
  • a user input location may be determined to match a location of a user interface corresponding to a particular application.
  • the user input may be sent automatically to the particular application, in step 730 , even if the application did not request or subscribe to the user input.
  • the application is free to use or reject the received user input based on a separate determination.
  • input may be sent to all applications regardless of whether matches are determined in steps 720 or 725 . Referring to the previous dice rolling example, dice input may be sent to all applications regardless of subscriptions or matches.
  • a single user input device may also be used to interact with a single or multiple application system.
  • the single user input device may comprise a touch-sensitive screen upon which one or more applications are displayed. By determining a location of touch, users may interact with multiple applications simultaneously by correlating the location of touch with an application or a component thereof.

Abstract

Processing multiple inputs received across simultaneously displayed applications. In an embodiment, a computing device displays a first application on a first portion of a display device, and simultaneously displays a second application on a second, different, portion of the display device. The computing device receives a first user input in connection with the first portion of the display device, and, at substantially the same time, also receives a second user input in connection with the second portion of the display device. The computing device routes the first user input to the first application based on the first user input having been received in connection with first portion of the display device, and, at substantially the same time, also routes the second user input to the second application based on the second user input having been received in connection with the second portion of the display device.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • This application is a continuation of U.S. patent application Ser. No. 13/692,110, filed on Dec. 3, 2012, entitled “SIMULTANEOUS INPUT ACROSS MULTIPLE APPLICATIONS,” which issued as U.S. Pat. No. ______ on ______, and which is a continuation of U.S. patent application Ser. No. 12/616,667, filed on Nov. 11, 2009, entitled “SIMULTANEOUS INPUT ACROSS MULTIPLE APPLICATIONS,” which issued as U.S. Pat. No. 8,347,215 on Jan. 1, 2013, and which is a continuation of U.S. patent application Ser. No. 11/384,510, filed Mar. 21, 2006, entitled “SIMULTANEOUS INPUT ACROSS MULTIPLE APPLICATIONS,” which issued as U.S. Pat. No. 7,620,901 on Nov. 17, 2009, the entirety of each of which are incorporated herein by reference.
  • BACKGROUND Background and Relevant Art
  • Interaction with computing devices has changed significantly since the inception of the graphical user interface. Users may control a variety of functions and interact with numerous types of applications through a graphical user interface using one or more associated hardware input devices. For example, computing devices are often used for gaming and allow multiple users to play the same game at the same time. Various input devices include joysticks, touch-sensitive screens, keyboards, mice and the like. Users may open several different applications at the same time and multi-task by switching between applications and viewing multiple documents simultaneously. Input events are determined and rendered based on a currently focused application of the user interface and the type of user input. Thus, in one situation, the pressing of keys on a keyboard may correspond to entering text when a word processing application is active (i.e., has input focus).
  • However, current computing systems require a user to alter the focus of the user interface in order to enter input into a different application. Without changing the focus, user input is directed to and registered on the same application. Multiple users wishing to interact with different applications would need to take turns and switch between the two applications for their input to register correctly. Users wanting to use two applications simultaneously are unable to do so due to this limitation. As such, users wishing to share a computing device must typically agree on a single application with which to interact. Accordingly, the efficiency of using a single device is tremendously decreased.
  • BRIEF SUMMARY
  • This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
  • Embodiments include methods, systems, and computer program products for processing multiple inputs received across simultaneously displayed applications. In an embodiment, a computing device displays a first application on a first portion of a display device, while simultaneously displaying a second application on a second, different, portion of the display device. The computing device receives a first user input in connection with the first portion of the display device, and also receives a second user input in connection with the second portion of the display device. The first user input and the second user input are received at substantially the same time. The computing device routes the first user input to the first application based on the first user input having been received in connection with first portion of the display device, and routes the second user input to the second application based on the second user input having been received in connection with the second portion of the display device. The second user input is routed to the second application substantially at the same time as routing the first user input to the first application
  • In another embodiment, a computing device displays a first application in a first portion of the display device. While displaying the first application in the first portion of the display device, the computing device simultaneously displays a second application in a second and different portion of the display device. The computing device simultaneously receives, at one or more user input devices, both (i) a first user input that is detected by the one or more user input devices to be directed at the first portion of the display device, and (ii) a second user input that is detected by the one or more user input devices to be directed at the second portion of the display device. The computing device simultaneously routes both (i) the first user input to the first application based at least on the first user input being detected to be directed at the first portion of the display device, and (ii) the second user input to the second application based at least on the second user input being detected to be directed at the second portion of the display device.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present disclosure is illustrated by way of example and not by limitation in the accompanying figures in which like reference numerals indicate similar elements and in which:
  • FIG. 1 illustrates a schematic diagram of a general-purpose digital computing environment that can be used to implement various aspects of the present disclosure.
  • FIG. 2 is a diagram of interaction by multiple users with user interfaces of multiple applications using a single input device according to one or more aspects described herein.
  • FIG. 3 is a diagram of user interaction with one or more applications using multiple input devices according to one or more aspects described herein.
  • FIG. 4 is a block diagram of a multiple input aware application system according to one or more aspects described herein.
  • FIG. 5 is a flowchart illustrating a method for handling multiple simultaneous user inputs for a plurality of different applications according to one or more aspects described herein.
  • FIG. 6 is a flowchart illustrating a method for processing input from multiple input hardware systems according to one or more aspects described herein.
  • FIG. 7 is a flowchart illustrating a method for routing input signals and events according to one or more aspects described herein.
  • DETAILED DESCRIPTION
  • In the following description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration various embodiments. It is to be understood that other embodiments may be utilized and structural and functional modifications may be made without departing from the scope of the present disclosure.
  • FIG. 1 illustrates a schematic diagram of a general-purpose digital computing environment. In FIG. 1, a computer 100 includes a processing unit 110, a system memory 120, and a system bus 130 that couples various system components including the system memory 120 to the processing unit 110. The system bus 130 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory 120 may include read only memory (ROM) 140 and random access memory (RAM) 150.
  • A basic input/output system 160 (BIOS), which contains the basic routines that help to transfer information between elements within the computer 100, is stored in the ROM 140. The computer 100 also may include a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a removable optical disk 199, such as a CD ROM or other optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194, respectively. These drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for the personal computer 100. It will be appreciated by those skilled in the art that other types of computer-readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment.
  • A number of program modules can be stored on the hard disk drive 170, magnetic disk 190, optical disk 199, ROM 140, or RAM 150, including an operating system 195, one or more application programs 196, other program modules 197, and program data 198. A user can enter commands and information into the computer 100 through input devices, such as a keyboard 101 and pointing device 102 (such as a mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices often are connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus 130, but they also may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB), and the like. Further still, these devices may be coupled directly to the system bus 130 via an appropriate interface (not shown).
  • A monitor 107 or other type of display device also may be connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor 107, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In some example environments, a stylus digitizer 165 and accompanying stylus 166 are provided in order to digitally capture freehand input. Although a connection between the digitizer 165 and the serial port interface 106 is shown in FIG. 1, in practice, the digitizer 165 may be directly coupled to the processing unit 110, or it may be coupled to the processing unit 110 in any suitable manner, such as via a parallel port or another interface and the system bus 130 as is known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 107 in FIG. 1, the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 107. Further still, the digitizer 165 may be integrated in the monitor 107, or it may exist as a separate device overlaying or otherwise appended to the monitor 107.
  • The computer 100 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 109. The remote computer 109 can be a server, a router, a network PC, a peer device or other common network node, and it typically includes many or all of the elements described above relative to the computer 100, although for simplicity, only a memory storage device 111 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet, using both wired and wireless connections.
  • When used in a LAN networking environment, the computer 100 is connected to the local area network 112 through a network interface or adapter 114. When used in a WAN networking environment, the computer 100 typically includes a modem 115 or other means for establishing a communications link over the wide area network 113, such as the Internet. The modem 115, which may be internal or external to the computer 100, may be connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted relative to the personal computer 100, or portions thereof, may be stored in the remote memory storage device.
  • It will be appreciated that the network connections shown are examples, and other techniques for establishing a communications link between computers can be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP, UDP, and the like is presumed, and the computer 100 can be operated in a user-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages.
  • Although the FIG. 1 environment shows one example environment, it will be understood that other computing environments also may be used. For example, an environment may be used having fewer than all of the various aspects shown in FIG. 1 and described above, and these aspects may appear in various combinations and subcombinations that will be apparent to one of ordinary skill. Additional elements, devices or subsystems also may be included in or coupled to the computer 100.
  • FIG. 2 illustrates a touch-sensitive input system for receiving multiple simultaneous user inputs intended for different applications. The system includes a multiple input hardware system such as touch-sensitive screen 200 (e.g., combination of screen 107 and digitizer 165) displaying a user interface 201 and applications 205 and 206. User interface 201 further includes a plurality of graphical interface elements including interfaces associated with application A 205 and application B 206. Traditionally, a user must first choose an application with which to interact before entering input. Upon receiving the selection, user interface 201 and/or the underlying operating system or application routes all subsequently received user input to the selected application. Using one or more aspects described herein, however, multiple users may simultaneously interact with multiple applications without having to initially choose an application with which to interact. In other words, users of the same device may interact with different applications independently of and simultaneously with one another. This allows multiple users to work with multiple different applications using a single shared computing device.
  • In one example, user 210 may play an electronic memory card game at the same time user 211 is engaging in a game of minesweeper. Specifically, user 210 contacts one area of display 200 corresponding to the memory game application simultaneously with user 211 touching a different location of display 200 associated with the minesweeper game. The user inputs from both user 210 and user 211 are delivered simultaneously to each application. Significantly, neither user 210 nor user 211 needs to change input focus from the memory game application to the minesweeper application or vice versa in order for input from each user to register with their respective application. Since some computing devices process input and data sequentially, true simultaneity might not be achieved. As such, the term simultaneous is used herein to refer to a difference in time that is virtually imperceptible to one or more users of a computing device. Thus, a first input and a second input delivered to an application or rendered on an application user interface simultaneously may be delivered sequentially in such a way that the user perceives the delivery or rendering as simultaneous.
  • As previously mentioned, the system of FIG. 2 uses a multiple input capable hardware system such as touch-sensitive input display 200 rather than traditional single input hardware devices like current joysticks and mice. Multiple input hardware, as used herein, refers to devices which are capable of detecting and transmitting simultaneous user inputs at the same time. For example, a touch sensitive display device may detect simultaneous inputs by one or more users by capturing an image of user inputs (i.e., capture the image of light reflecting off of multiple touch contacts on the surface) and parsing the image to determine input information. Input devices transmit detected input signals to a destination device or application in periodic transmission intervals or frames. With traditional single input hardware devices, multiple simultaneous inputs are sent to an application in sequential transmission frames. Thus, input resulting from two users simultaneously pressing two buttons of a single input keyboard is detected and transmitted to an associated computing device in two sequential transmission slots. This results in a perceptible lag between the processing of the first user input and the processing of the second user input on user interface 201. Using a multiple input hardware system, however, two user inputs received simultaneously are transmitted in a single transmission frame to the computing device. As a result, simultaneous input received through the multiple input hardware can be rendered simultaneously through user interface 201 instead of in a perceptibly sequential order. For example, using a multiple input hardware device such as touch sensitive display 200, user 210 may enter numbers into a calculator application displayed on interface 201 at the same time a user 211 is navigating through web pages on a second application. The input corresponding to the entry of numbers in the calculator may be sent in the same transmission frame as the browsing input from user 211.
  • The correspondence between a user input and a particular application may be determined based on a user input location and an application location and size. User interface 201 or an underlying operating system may determine whether touch input from user 210 is directed to application 205 or application 206 based on the input location and the locations and sizes of applications 205 and 206. In one or more configurations, a table containing application locations and sizes may be stored. In response to detecting user input at location 220, a correspondence may be established between the input and application 205 using the application location table. Similarly, if a user touches location 221 on interface 201, the touch input may be routed or sent to application 206 based on the aforementioned determination techniques. Alternatively or additionally, input routing may be based on other heuristics including the type of input. For example, one application may receive all inputs that are recognized as finger input and another application may receive the input if the input is determined to originate from a stylus. Inputs may also be delivered to all applications in cases where either the input applies to all applications or where it is not possible to determine which application is interested in the input. In one example, a special gesture may correspond to a “hide all applications” command. In another example, a user may roll dice on an input device. Since the user may not be able to entirely control where the dice lands, the dice input may be transmitted to all applications.
  • FIG. 3 illustrates various aspects of a system 300 for processing multiple inputs across multiple applications. System 300 includes a computing device (e.g., computer 100) having a touch-sensitive display screen 303 (e.g., combination of screen 107 and digitizer 165) oriented such that touch-sensitive display 303 may be used as the surface of a table or workspace. Users may be seated around the display screen 303 of computing device and simultaneously interact with applications 315, 316, 317 and 318 by contacting corresponding locations or portions of touch-sensitive display screen 303. For example, a user may play a memory card game 318 by touching a first portion of screen 303 while one or more users are drafting a document on word processing application 317, playing a second memory card game 316 and/or drawing a picture in paint application 315 by contacting a second, third and/or fourth portion of screen 303. The user inputs may be differentiated and routed to the appropriate application based on the location of user input and the locations and sizes of the relevant applications. Input routing may be performed via a variety of other mechanisms as well.
  • In one or more configurations, two applications may be linked such that input between the two or more applications is synchronized. In other words, user input to application 318 may be synchronized and registered with application 316 and vice versa. Synchronization may include determining a time of receipt for each user input and sequencing the order in accordance with the determined receipt times. In one example, two users playing a competitive memory card game may simultaneously interact with two separate application instances, such as applications 316 and 318, of the memory game application. The game may display the same field of cards for both application instances 316 and 318 and each application instance 316 and 318 may register input from both players. Thus, simultaneous card selections from the players may be registered simultaneously on both applications 316 and 318. The synchronization of the user inputs ensures that a single card selection is not registered with both players. For example, once a pair of cards has been eliminated by a first player, the eliminated pair of cards is removed from both application instances 316 and 318 so that a second user cannot select that pair of cards. Additionally, input corresponding to flipping a card on application 315 may be simultaneously registered on application 318 (i.e., the input event associated with the card flipping is processed by application 318 as well).
  • Alternatively or additionally, the synchronization of application instances 316 and 318 may be coordinated by a third application (not shown). Each application may then communicate their inputs (i.e., selected cards) back to the third application. In one example, the use of a coordinating third application provides for a situation where two or more players attempted to claim the same matched cards at the same time. In such a situation, the third application may determine the winner of the matched cards based on which player obtained the next match.
  • In yet another example, four users of system 300 may simultaneously play a scrabble board game generated by interface 301. Each user may be assigned a block holder application that displays each user's scrabble blocks. Another application may be generated for displaying the scrabble board on which the blocks are to be placed. Users may place blocks on the scrabble board by initially touching a location on display 303 corresponding to the location of a particular scrabble block in the block holder application and subsequently touching a location of display 303 corresponding to a desired placement location in the scrabble board application. Simultaneously, other users may be rearranging their blocks by selecting a block and subsequently choosing a desired placement location in their block holder. Again, users are not required to wait for other users to finish entering input before entering their own input. The multiple inputs from the users are differentiated and routed based on the users' input location. Other techniques may also be used to route the user's input including routing input based on a unique characteristic of the user's input. For example, a user may use his or her finger to provide touch input. Accordingly, the user's input may be routed based on an identification of the user's fingerprint.
  • System 300 may further include input devices such as devices 310, 311, 312 and 313. In addition to touch-sensitive displays, input devices 310, 311, 312 and 313 may include a variety of device types such as joysticks, mice, keyboards and the like. Input devices 310, 311, 312 and 313 may further be of the same type or of different types depending on the applications with which the devices are associated. For example, in an electronic video game environment, each application interface 315, 316, 317 and 318 may be similar and require the users to use the same type of device, e.g., a joystick. However, if multiple users are working with different applications, the device types may differ. These input devices may act as supplemental input sources to touch-sensitive display 303 or, alternatively, function as the primary input device.
  • Each of input devices 310, 311, 312 and 313 may further be associated with one of applications 315, 316, 317 or 318. The associations may be chosen based on a user selection, input device type or a host of other factors. Indicators 325, 326, 327 and 328 are generated and displayed to allow multiple users to recognize and identify which applications correspond to which input devices. For example, indicators 325, 326, 327 and 328 may display a number that corresponds to a port on the computing device. The input devices, themselves, may also include an indicator such as an indicator light that identifies the device-application association. The input device-application association may be modified in a variety of ways including changing the settings in a control panel (not shown) and/or altering settings on the input device hardware.
  • FIG. 4 is a block diagram illustrating a system for processing input from single input 405 and multiple input 406 hardware sources across a series of applications 401, 402 and 403. Single input devices differ from multiple input devices in that single input devices transmit simultaneous input sequentially while multiple input devices may transmit simultaneous input simultaneously in the same transmission frame. In addition, input from single input devices are processed by an operating system (OS) event thread 415, 416 or 417 depending on which application currently has focus. OS Event processing threads 415, 416 and 417 are limited to sequentially processing input events for a single application at a time. Thus, OS Event processing threads 415, 416 and 417 are incapable of processing input corresponding to users contacting multiple locations of a touch-sensitive display at the same time. Applications A 401, B 402 and C 403 each correspond to an OS Event processing thread 415, 416 and 417 and an application user interface (UI) 410, 411 and 412, respectively. In this example, applications B 402 and C 403 further include secondary event processing threads 418 and 419, respectively, in order to bypass the limits of OS Event processing threads 415, 416 and 417 and in order to process simultaneous input received from multiple input hardware system 406. Secondary event processing threads 418 and 419, unlike OS Event threads 415, 416 and 417, are not focus oriented and may thus process multiple simultaneous input received from a multiple input device such as system 406.
  • In applications B 402 and C 403, secondary event processing threads 418 and 419 operate in parallel to OS Event processing threads 416 and 417. Each input signal detected in the multiple input hardware system 406 is initially evaluated to determine a corresponding or receiving application. Application correspondence of each input signal may be determined based on a location of input and/or any one of a plurality of other factors. Thus, a first user input at a first location of a touch-sensitive display may be associated with and routed to a different application than a second user input received at a second location of the same touch-sensitive display. Input location and application location and size may be matched in order to determine appropriate correspondencies. Each user input may then be processed by either secondary event thread 418 or 419 depending on the determined application correspondency of each input.
  • In one or more instances, the multiple input hardware system 406 may include several types of input devices and each type of input may further be associated with a different application. Accordingly, the input signals may be routed based on, at least in part, the type of input. Accordingly, each secondary processing thread 418 and 419 may receive a plurality of input signals detected by the input hardware system 406 and perform parsing functions to determine which signals are relevant to their underlying application. Event routing may further be performed by a third party router component that is implemented between multiple input hardware system 406 and applications 402 and 403. Additionally, each secondary processing thread 418 and 419 may further be responsible for determining input events corresponding to the received input signals associated with their application, i.e., B 402 and C 403. Other processing steps may be implemented separately or in combination to properly render the input signals and/or changes on application UIs 411 and 412. In one or more configurations, multiple input hardware system 406 might not provide input or event routing.
  • Alternatively or additionally, a single input hardware device 405 may be used in combination with the multiple input hardware system 406 and multiple input aware applications 411 and 412. To facilitate such an implementation, an event router 420 may be added between the hardware 405 and multiple input aware applications 411 and 412. The event router 420 receives input events, generated by the OS, from single input hardware device 405 and determines an application to which the input event should be sent. The determination may be based on a subscription design pattern whereby multiple input applications 411 and 412 submit requests for input from the event router 420. The subscription requests may identify a particular area of the display screen that corresponds to the requesting application, a corresponding input type or the like. The event router 420, in response to the input requests, sends the appropriate input events to the requesting application. Thus, in one example, a user may play a video game on a computing device using a joystick input device while another user performs word processing tasks using a keyboard input device simultaneously on the same computing device.
  • In one or more configurations, secondary event processing threads 418 and 419, upon determining an input event from one or more received input signals, may send the input event to Application OS Event processing threads 416 and 417, respectively, for synchronization. In particular, the input events determined by the secondary threads 418 and 419 may be synchronized to one or more events currently in OS Event threads 416 and 417, respectively. Synchronization ensures that events are properly rendered on an application user interface by determining a sequence of events based on a receiving time of the user input. For example, a computing device may be connected to both a multiple input hardware system such as system 406 as well as a single input device such as hardware 405. Events detected and determined from both single input hardware 405 and multiple input system 406 may then be synchronized by, e.g., ordering the input events according to received time and ensuring appropriate rendering on a user interface of application 411.
  • FIG. 5 is a flowchart illustrating a method of handling multiple simultaneous inputs directed at multiple different applications. In step 500 and 505, a first user input and a second user input are received or detected simultaneously. The inputs may correspond to users touching different locations of a touch-sensitive display or, in another example, the inputs may be associated with detected contacts with different sections of a touch input device, such as a touchpad or touch screen. Upon receiving the simultaneous first and second inputs, input locations are determined for each of the first and second inputs in steps 510 and 515. In the example of the touch-sensitive display, the input location may correspond to the location of user contact, i.e., input location. The input location information is then processed in step 520 to determine one or more applications that correspond to the first and second user inputs. The determination may involve a further determination of locations and sizes for each of the applications displayed on the user interface. An application may then be identified for each user input based on matching the input location with the location and size of the application. The user inputs may correspond to different applications or the same applications, again, based on the locations of user input and of the applications. Once applications corresponding to each user input have been identified, each user input is sent to the identified corresponding application in step 525.
  • FIG. 6 is a flowchart showing a method for processing input from a multiple input system across at least two applications. In step 600, a multiple input hardware system or device is detected by an application. In one or more aspects, for example, a hardware connection may be detected by an underlying operating system and broadcast to applications. Any one of numerous detection methods known in the art may be used or implemented in conjunction with this step. In step 605, a determination is made to evaluate whether the detected input system is associated with the application. The input system may be associated with an application by, e.g., altering control settings and associations in the operating system. If the input system is not associated with the application, the application does not process input signals from the detected system. However, if the input system is associated with the application, then a secondary processing thread corresponding to the input system is created in step 610. The secondary processing thread is also created to run in parallel with an event processing thread of the operating system. Alternatively, the secondary processing thread may be created when the application is initialized.
  • Following the creation of the secondary processing thread, user input signals are received from the multiple input hardware system at the secondary processing thread in step 615. In step 620, the secondary thread processes the user input to determine one or more input events. In other words, the secondary thread identifies an action corresponding to the user input that is to be performed on the application user interface. For example, input signals from a touch sensitive display screen may reflect a change in position of a user's finger or stylus. As such, the change in a user's finger or stylus position may be determined by the secondary processing thread upon receiving the corresponding input signals. Input signals may further include multiple events such as selection and movement events corresponding to a select-and-drag interaction by the user with an element of the user interface. In step 625, another determination is made to decide whether the determined input event or events should be synchronized with one or more events of the operating system event processing thread. In one or more configurations, various user interface components such as user interface controls might be designed for access through OS Event processing threads. As such, input from a multiple input device directed toward such interface components may be synchronized with input events in the OS Event processing thread to properly register the input. In some instances, the application receiving the multiple input events may be responsible for determining whether the events need to be synchronized.
  • If the input events determined by the secondary processing thread are to be synchronized with the events of the OS Event processing thread, the input events of the secondary thread are sent to the OS Event thread in step 630. The OS thread may then synchronize, e.g., arrange the events in order of time received, and perform further processing before sending the events to the application UI in step 635. Alternatively, if the input events of the secondary thread do not need to be synchronized, the events are sent to the application user interface, as in step 635, without initially passing through the OS Event thread. One of skill in the art will appreciate that there are myriad of methods for synchronizing input events and any of those methods may be implemented with the systems and methods described herein.
  • FIG. 7 is a flowchart illustrating a method of routing input signals and/or input events. Upon receiving user input in step 700, the user input is evaluated to determine various characteristics of the input. These characteristics may include a type of input, duration of input, input location, identification of the input source and the like. In step 705, applications active on the user interface and/or subscribing to input from the input source are identified. For example, in one or more configurations, the user interface or underlying operating system may maintain a database of applications that subscribe to particular input devices or systems. In step 710, characteristics of the identified applications are determined. Application characteristics may include application type (e.g., game, tool, word processing, etc.), application location and application control and input type. Once the relevant applications have been identified and any characteristics have been determined from the applications and input, a determination is then made in step 715 to identify any applications that subscribe to or are requesting the received input. Thus, an electronic video game, for example, may be identified as an application that subscribes to input from a particular joystick connected to the computing device. In response to determining that one or more applications subscribe to or requested the input, the received input is sent to the one or more applications in step 720. If, however, no applications subscribe or request the input, another determination may be made, in step 725, to evaluate whether the determined input characteristics matches any of the characteristics of the identified applications. In one example, a user input location may be determined to match a location of a user interface corresponding to a particular application. In such an instance, the user input may be sent automatically to the particular application, in step 730, even if the application did not request or subscribe to the user input. Alternatively or additionally, the application is free to use or reject the received user input based on a separate determination. Further, input may be sent to all applications regardless of whether matches are determined in steps 720 or 725. Referring to the previous dice rolling example, dice input may be sent to all applications regardless of subscriptions or matches.
  • The methods and systems described herein have been discussed, in large part, with respect to a multiple application system. However, the methods and systems may further be implemented and applied to single application systems have multiple components. Individual secondary threads may be created for each of the multiple components to handle user input from a multiple input hardware system. Additionally, a single user input device may also be used to interact with a single or multiple application system. In one or more configurations, the single user input device may comprise a touch-sensitive screen upon which one or more applications are displayed. By determining a location of touch, users may interact with multiple applications simultaneously by correlating the location of touch with an application or a component thereof.
  • Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (20)

What is claimed is:
1. A computing device for processing multiple inputs received across simultaneously displayed applications, the computing device comprising:
one or more processors;
a display device; and
a storage medium having stored thereon executable instructions that are executable by the one or more processors to configure the computing device to perform at least the following:
display a first application on a first portion of the display device, and simultaneously display a second application on a second, different, portion of the display device;
receive a first user input in connection with the first portion of the display device;
receive a second user input in connection with the second portion of the display device, the first user input and the second user input being received at substantially the same time;
route the first user input to the first application based on the first user input having been received in connection with first portion of the display device; and
route the second user input to the second application based on the second user input having been received in connection with the second portion of the display device, the second user input being routed to the second application substantially at the same time as routing the first user input to the first application.
2. The computing device of claim 1, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route the first user input to a first secondary input event processing thread that is running in parallel to a first main operating system event processing thread of the first application.
3. The computing device of claim 2, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route the second user input to a second secondary input event processing thread that is running in parallel to a second main operating system event processing thread of the second application.
4. The computing device of claim 1, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
determine a type of the first user input; and
determine a type of the second user input.
5. The computing device of claim 4, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
when determining the type of the first user input, determine that the first user input is from a first type of object; and
when determining the type of the second user input, determine that the second user input is from a second type of object that is different than the first type of object.
6. The computing device of claim 5, wherein the first type of object comprises a finger, and the second type of object comprises a stylus.
7. The computing device of claim 5, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route only user input from the first type of object to the first application; and
route only user input from the second type of object to the second application.
8. The computing device of claim 1, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
synchronize the first and second user interface components, including sending the first user input to the second application associated with the second user interface component.
9. The computing device of claim 1, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
transmit the first user input and the second touch input in a single transmission frame.
10. The computing device of claim 1, wherein the display device includes a digitizer, and wherein one or more of the first user input and the second user input comprises a touch user input.
11. The computing device of claim 1, wherein the display device includes a digitizer, and wherein one or more of the first user input and the second user input comprises a stylus user input.
12. A computing device for processing multiple inputs received across simultaneously displayed applications, the computing device comprising:
one or more processors;
a display device;
one or more user input devices that are associated with the display device; and
a storage medium having stored thereon executable instructions that are executable by the one or more processors to configure the computing device to perform at least the following:
display a first application in a first portion of the display device;
while displaying the first application in the first portion of the display device, simultaneously display a second application in a second and different portion of the display device;
simultaneously receive, at the one or more user input devices, both (i) a first user input that is detected by the one or more user input devices to be directed at the first portion of the display device, and (ii) a second user input that is detected by the one or more user input devices to be directed at the second portion of the display device;
simultaneously route both (i) the first user input to the first application based at least on the first user input being detected to be directed at the first portion of the display device, and (ii) the second user input to the second application based at least on the second user input being detected to be directed at the second portion of the display device.
13. The computing device of claim 12, the one or more user input devices comprising one or more digitizers that detect touch-based user input.
14. The computing device of claim 12, the one or more user input devices comprising one or more digitizers that detect stylus-based user input.
15. The computing device of claim 12, the one or more user input devices comprising one or more digitizers that detect both touch-based user input and stylus-based user input, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
distinguish between touch-based user input and stylus-based user input at the one or more user input devices.
16. The computing device of claim 15, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route only touch-based user input to the first application; and
route only stylus-based user input to the second application.
17. The computing device of claim 15, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route only touch-based user input to the first application; and
route stylus-based user input to one or both of the first application and the second application.
18. The computing device of claim 15, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route only stylus-based user input to the first application; and
route touch-based user input to one or both of the first application and the second application.
19. The computing device of claim 12, the executable instructions including instructions that are executable by the one or more processors to configure the computing device to:
route the first user input to a first secondary input event processing thread that is running in parallel to a first main operating system event processing thread of the first application; and
route the second user input to a second secondary input event processing thread that is running in parallel to a second main operating system event processing thread of the second application.
20. A method, implemented at a computing device that includes one or more processors and a display device, the method for processing multiple inputs received across simultaneously displayed applications, the method comprising:
displaying a first application on a first portion of the display device, while simultaneously displaying a second application on a second, different, portion of the display device;
receiving a first user input in connection with the first portion of the display device;
receiving a second user input in connection with the second portion of the display device, the first user input and the second user input being received at substantially the same time;
routing the first user input to the first application based on the first user input having been received in connection with first portion of the display device; and
routing the second user input to the second application based on the second user input having been received in connection with the second portion of the display device, the second user input being routed to the second application substantially at the same time as routing the first user input to the first application.
US14/857,391 2006-03-21 2015-09-17 Simultaneous input across multiple applications Abandoned US20160085419A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/857,391 US20160085419A1 (en) 2006-03-21 2015-09-17 Simultaneous input across multiple applications

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US11/384,510 US7620901B2 (en) 2006-03-21 2006-03-21 Simultaneous input across multiple applications
US12/616,667 US8347215B2 (en) 2006-03-21 2009-11-11 Simultaneous input across multiple applications
US13/692,110 US9164659B2 (en) 2006-03-21 2012-12-03 Simultaneous input across multiple applications
US14/857,391 US20160085419A1 (en) 2006-03-21 2015-09-17 Simultaneous input across multiple applications

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US13/692,110 Continuation US9164659B2 (en) 2006-03-21 2012-12-03 Simultaneous input across multiple applications

Publications (1)

Publication Number Publication Date
US20160085419A1 true US20160085419A1 (en) 2016-03-24

Family

ID=38535076

Family Applications (4)

Application Number Title Priority Date Filing Date
US11/384,510 Expired - Fee Related US7620901B2 (en) 2006-03-21 2006-03-21 Simultaneous input across multiple applications
US12/616,667 Active 2026-12-30 US8347215B2 (en) 2006-03-21 2009-11-11 Simultaneous input across multiple applications
US13/692,110 Expired - Fee Related US9164659B2 (en) 2006-03-21 2012-12-03 Simultaneous input across multiple applications
US14/857,391 Abandoned US20160085419A1 (en) 2006-03-21 2015-09-17 Simultaneous input across multiple applications

Family Applications Before (3)

Application Number Title Priority Date Filing Date
US11/384,510 Expired - Fee Related US7620901B2 (en) 2006-03-21 2006-03-21 Simultaneous input across multiple applications
US12/616,667 Active 2026-12-30 US8347215B2 (en) 2006-03-21 2009-11-11 Simultaneous input across multiple applications
US13/692,110 Expired - Fee Related US9164659B2 (en) 2006-03-21 2012-12-03 Simultaneous input across multiple applications

Country Status (1)

Country Link
US (4) US7620901B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10620819B2 (en) 2016-12-20 2020-04-14 Samsung Electronics Co., Ltd. Display apparatus and controlling method thereof

Families Citing this family (110)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7620901B2 (en) 2006-03-21 2009-11-17 Microsoft Corporation Simultaneous input across multiple applications
US8997015B2 (en) * 2006-09-28 2015-03-31 Kyocera Corporation Portable terminal and control method therefor
US9575655B2 (en) * 2006-12-29 2017-02-21 Nokia Technologies Oy Transparent layer application
US20080168478A1 (en) 2007-01-07 2008-07-10 Andrew Platzer Application Programming Interfaces for Scrolling
US20080168402A1 (en) 2007-01-07 2008-07-10 Christopher Blumenberg Application Programming Interfaces for Gesture Operations
US10338694B2 (en) * 2007-01-22 2019-07-02 Sony Corporation Multiple focus control
US8436815B2 (en) * 2007-05-25 2013-05-07 Microsoft Corporation Selective enabling of multi-input controls
US9953392B2 (en) 2007-09-19 2018-04-24 T1V, Inc. Multimedia system and associated methods
US8583491B2 (en) * 2007-09-19 2013-11-12 T1visions, Inc. Multimedia display, multimedia system including the display and associated methods
US9965067B2 (en) 2007-09-19 2018-05-08 T1V, Inc. Multimedia, multiuser system and associated methods
US20100179864A1 (en) * 2007-09-19 2010-07-15 Feldman Michael R Multimedia, multiuser system and associated methods
US8600816B2 (en) * 2007-09-19 2013-12-03 T1visions, Inc. Multimedia, multiuser system and associated methods
US8217854B2 (en) * 2007-10-01 2012-07-10 International Business Machines Corporation Method and system for managing a multi-focus remote control session
US20090106667A1 (en) * 2007-10-19 2009-04-23 International Business Machines Corporation Dividing a surface of a surface-based computing device into private, user-specific areas
US8645827B2 (en) 2008-03-04 2014-02-04 Apple Inc. Touch event model
US20090225040A1 (en) * 2008-03-04 2009-09-10 Microsoft Corporation Central resource for variable orientation user interface
US8717305B2 (en) * 2008-03-04 2014-05-06 Apple Inc. Touch event model for web pages
US20090273569A1 (en) * 2008-05-01 2009-11-05 Microsoft Corporation Multiple touch input simulation using single input peripherals
US9268483B2 (en) * 2008-05-16 2016-02-23 Microsoft Technology Licensing, Llc Multi-touch input platform
KR101666831B1 (en) 2008-11-26 2016-10-17 캘거리 싸이언티픽 인코포레이티드 Method and system for providing remote access to a state of an application program
JP5361355B2 (en) * 2008-12-08 2013-12-04 キヤノン株式会社 Information processing apparatus and control method thereof, and printing apparatus and control method thereof
US8407606B1 (en) * 2009-01-02 2013-03-26 Perceptive Pixel Inc. Allocating control among inputs concurrently engaging an object displayed on a multi-touch device
SG2014007801A (en) * 2009-02-03 2014-04-28 Calgary Scient Inc Method and system for enabling interaction with a plurality of applications using a single user interface
US10055105B2 (en) 2009-02-03 2018-08-21 Calgary Scientific Inc. Method and system for enabling interaction with a plurality of applications using a single user interface
US8432366B2 (en) * 2009-03-03 2013-04-30 Microsoft Corporation Touch discrimination
US8285499B2 (en) 2009-03-16 2012-10-09 Apple Inc. Event recognition
US9684521B2 (en) 2010-01-26 2017-06-20 Apple Inc. Systems having discrete and continuous gesture recognizers
US8566045B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US20110161824A1 (en) * 2009-12-10 2011-06-30 France Telecom Process and system for interaction with an application that is shared among multiple users
EP2333651B1 (en) * 2009-12-11 2016-07-20 Dassault Systèmes Method and system for duplicating an object using a touch-sensitive display
US20110241981A1 (en) * 2010-03-30 2011-10-06 Lena Sojian Input Routing for Simultaneous USB Connections of Similar Device Types
US9740364B2 (en) * 2010-05-03 2017-08-22 Microsoft Technology Licensing, Llc Computer with graphical user interface for interaction
US10216408B2 (en) 2010-06-14 2019-02-26 Apple Inc. Devices and methods for identifying user interface objects based on view hierarchy
GB2485223A (en) * 2010-11-05 2012-05-09 Promethean Ltd Forwarding input data associated with location-aware events to an application for processing
US9741084B2 (en) 2011-01-04 2017-08-22 Calgary Scientific Inc. Method and system for providing remote access to data for display on a mobile device
US8589950B2 (en) * 2011-01-05 2013-11-19 Blackberry Limited Processing user input events in a web browser
US20120185936A1 (en) * 2011-01-19 2012-07-19 General Electric Company Systems and Methods for Detecting Fraud Associated with Systems Application Processing
US8839390B2 (en) * 2011-03-08 2014-09-16 Microsoft Corporation Grouping personal accounts to tailor a web service
CA2734860A1 (en) 2011-03-21 2012-09-21 Calgary Scientific Inc. Method and system for providing a state model of an application program
KR101774316B1 (en) * 2011-04-18 2017-09-04 엘지전자 주식회사 Image display device and method of managing conents using the same
JP5857443B2 (en) * 2011-05-12 2016-02-10 ソニー株式会社 Content sharing system
JP2012238219A (en) * 2011-05-12 2012-12-06 Sony Corp Information processing apparatus, server device, information processing method, computer program and content sharing system
JP5880916B2 (en) 2011-06-03 2016-03-09 ソニー株式会社 Information processing apparatus, information processing method, and program
US10748233B2 (en) * 2011-06-14 2020-08-18 Intel Corporation Full screen processing in multi-application environments
US10976981B2 (en) * 2011-07-15 2021-04-13 Vmware, Inc. Remote desktop exporting
SG2014011506A (en) 2011-08-15 2014-05-29 Calgary Scient Inc Non-invasive remote access to an application program
EP2745467A4 (en) 2011-08-15 2015-10-14 Calgary Scient Inc Method for flow control and for reliable communication in a collaborative environment
US9596320B2 (en) 2011-09-30 2017-03-14 Calgary Scientific Inc. Uncoupled application extensions including interactive digital surface layer for collaborative remote application sharing and annotating
US9142182B2 (en) * 2011-10-07 2015-09-22 Lg Electronics Inc. Device and control method thereof
KR101888457B1 (en) * 2011-11-16 2018-08-16 삼성전자주식회사 Apparatus having a touch screen processing plurality of apllications and method for controlling thereof
JP6172537B2 (en) 2011-11-23 2017-08-02 カルガリー サイエンティフィック インコーポレイテッド Method and system for federated remote application sharing and conferencing
US9817442B2 (en) * 2012-02-28 2017-11-14 Razer (Asia-Pacific) Pte. Ltd. Systems and methods for presenting visual interface content
CA2865707A1 (en) 2012-03-02 2013-09-06 Calgary Scientific Inc. Remote control of an application using dynamic-linked library (dll) injection
US9002929B2 (en) * 2012-03-02 2015-04-07 Fisher Controls International Llc Methods and apparatus to reduce memory requirements for process control system software applications
US9729673B2 (en) 2012-06-21 2017-08-08 Calgary Scientific Inc. Method and system for providing synchronized views of multiple applications for display on a remote computing device
JP5929572B2 (en) * 2012-07-09 2016-06-08 コニカミノルタ株式会社 Operation display device and program
US9507513B2 (en) 2012-08-17 2016-11-29 Google Inc. Displaced double tap gesture
US9582133B2 (en) 2012-11-09 2017-02-28 Sap Se File position shortcut and window arrangement
KR20140066378A (en) * 2012-11-23 2014-06-02 삼성전자주식회사 Display apparatus and method of controlling the same
US9535503B2 (en) * 2013-01-29 2017-01-03 Blackberry Limited Methods and devices for simultaneous multi-touch input
KR102102157B1 (en) 2013-03-29 2020-04-21 삼성전자주식회사 Display apparatus for executing plurality of applications and method for controlling thereof
US9733716B2 (en) 2013-06-09 2017-08-15 Apple Inc. Proxy gesture recognizer
USD747344S1 (en) 2013-08-02 2016-01-12 Apple Inc. Display screen with graphical user interface
JP6117053B2 (en) * 2013-08-23 2017-04-19 シャープ株式会社 Display control apparatus, display control method, and program
USD754749S1 (en) * 2013-08-29 2016-04-26 Samsung Electronics Co., Ltd. Display screen or portion thereof with icon
US9740361B2 (en) 2013-10-14 2017-08-22 Microsoft Technology Licensing, Llc Group experience user interface
US9596319B2 (en) 2013-11-13 2017-03-14 T1V, Inc. Simultaneous input system for web browsers and other applications
CN105765923B (en) 2013-11-29 2019-11-12 卡尔加里科技股份有限公司 The method of the connection of client to non-managed service is provided in client-server remote access system
US20150193096A1 (en) * 2014-01-07 2015-07-09 Samsung Electronics Co., Ltd. Electronic device and method for operating the electronic device
EP2913749A1 (en) * 2014-02-26 2015-09-02 Advanced Digital Broadcast S.A. Method and system for focus management in a software application
USD759724S1 (en) * 2014-06-03 2016-06-21 Samsung Electronics Co., Ltd. Display screen or portion thereof with graphical user interface
JP2016035706A (en) * 2014-08-04 2016-03-17 パナソニックIpマネジメント株式会社 Display device, display control method and display control program
US10146409B2 (en) * 2014-08-29 2018-12-04 Microsoft Technology Licensing, Llc Computerized dynamic splitting of interaction across multiple content
KR20160053462A (en) * 2014-11-04 2016-05-13 삼성전자주식회사 Terminal apparatus and method for controlling thereof
KR102183212B1 (en) 2014-11-18 2020-11-25 삼성전자주식회사 Method for controlling display and an electronic device thereof
WO2016120730A1 (en) 2015-01-30 2016-08-04 Calgary Scientific Inc. Highly scalable, fault tolerant remote access architecture and method of connecting thereto
US10015264B2 (en) 2015-01-30 2018-07-03 Calgary Scientific Inc. Generalized proxy architecture to provide remote access to an application framework
USD775185S1 (en) * 2015-03-06 2016-12-27 Apple Inc. Display screen or portion thereof with graphical user interface
US9690400B2 (en) 2015-04-21 2017-06-27 Dell Products L.P. Information handling system interactive totems
US9804733B2 (en) * 2015-04-21 2017-10-31 Dell Products L.P. Dynamic cursor focus in a multi-display information handling system environment
US11106314B2 (en) 2015-04-21 2021-08-31 Dell Products L.P. Continuous calibration of an information handling system projected user interface
US10139854B2 (en) 2015-04-21 2018-11-27 Dell Products L.P. Dynamic display resolution management for an immersed information handling system environment
US9921644B2 (en) 2015-04-21 2018-03-20 Dell Products L.P. Information handling system non-linear user interface
US9983717B2 (en) 2015-04-21 2018-05-29 Dell Products L.P. Disambiguation of false touch inputs at an information handling system projected user interface
US11243640B2 (en) 2015-04-21 2022-02-08 Dell Products L.P. Information handling system modular capacitive mat with extension coupling devices
KR102404356B1 (en) * 2015-09-11 2022-06-02 엘지전자 주식회사 Digital device and method of processing data the same
EP3329351B1 (en) * 2016-01-13 2022-08-24 Hewlett-Packard Development Company, L.P. Simultaneously executing multiple pen inputs
USD790575S1 (en) 2016-06-12 2017-06-27 Apple Inc. Display screen or portion thereof with graphical user interface
US11567785B2 (en) 2016-10-31 2023-01-31 Microsoft Technology Licensing, Llc Integrated multitasking interface for communication sessions
US10139930B2 (en) 2016-11-09 2018-11-27 Dell Products L.P. Information handling system capacitive touch totem management
US10496216B2 (en) 2016-11-09 2019-12-03 Dell Products L.P. Information handling system capacitive touch totem with optical communication support
US10139973B2 (en) 2016-11-09 2018-11-27 Dell Products L.P. Information handling system totem tracking management
US10139951B2 (en) 2016-11-09 2018-11-27 Dell Products L.P. Information handling system variable capacitance totem input management
US10146366B2 (en) 2016-11-09 2018-12-04 Dell Products L.P. Information handling system capacitive touch totem with optical communication support
US10739993B2 (en) * 2017-01-19 2020-08-11 Microsoft Technology Licensing, Llc Simultaneous authentication system for multi-user collaboration
US11188152B2 (en) * 2017-07-05 2021-11-30 Mitsubishi Electric Corporation Operation unit control device and operation unit control method
CN109558187B (en) * 2017-09-27 2022-08-23 斑马智行网络(香港)有限公司 User interface rendering method and device
US10459528B2 (en) 2018-02-28 2019-10-29 Dell Products L.P. Information handling system enhanced gesture management, control and detection
WO2019195817A1 (en) * 2018-04-07 2019-10-10 Zte Corporation Application mobility mechanism for edge computing
USD877174S1 (en) 2018-06-03 2020-03-03 Apple Inc. Electronic device with graphical user interface
US10635199B2 (en) 2018-06-28 2020-04-28 Dell Products L.P. Information handling system dynamic friction touch device for touchscreen interactions
US10664101B2 (en) 2018-06-28 2020-05-26 Dell Products L.P. Information handling system touch device false touch detection and mitigation
US10817077B2 (en) 2018-06-28 2020-10-27 Dell Products, L.P. Information handling system touch device context aware input tracking
US10852853B2 (en) 2018-06-28 2020-12-01 Dell Products L.P. Information handling system touch device with visually interactive region
US10761618B2 (en) 2018-06-28 2020-09-01 Dell Products L.P. Information handling system touch device with automatically orienting visual display
US10795502B2 (en) 2018-06-28 2020-10-06 Dell Products L.P. Information handling system touch device with adaptive haptic response
US10996839B2 (en) * 2019-05-20 2021-05-04 Microsoft Technology Licensing, Llc Providing consistent interaction models in communication sessions
SE2030154A1 (en) * 2020-05-11 2021-11-12 Hm Group Ab Apparatus and method for supporting touch input events
US11698940B1 (en) * 2021-03-17 2023-07-11 Amazon Technologies, Inc. Caching item information for display in an interface overlay
US11714539B1 (en) 2022-10-28 2023-08-01 Honeywell International Inc. Cursor management methods and systems

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5107443A (en) * 1988-09-07 1992-04-21 Xerox Corporation Private regions within a shared workspace
US4975690A (en) * 1988-11-07 1990-12-04 Ibm Corporation Method for concurrent data entry and manipulation in multiple applications
JP2865827B2 (en) * 1990-08-13 1999-03-08 株式会社日立製作所 Data storage method in conference system
EP0622930A3 (en) * 1993-03-19 1996-06-05 At & T Global Inf Solution Application sharing for computer collaboration system.
US6241612B1 (en) * 1998-11-09 2001-06-05 Cirrus Logic, Inc. Voice communication during a multi-player game
US6951516B1 (en) * 2001-08-21 2005-10-04 Nintendo Co., Ltd. Method and apparatus for multi-user communications using discrete video game platforms
US6966837B1 (en) * 2001-05-10 2005-11-22 Best Robert M Linked portable and video game systems
US20040139156A1 (en) * 2001-12-21 2004-07-15 Matthews W. Donald Methods of providing direct technical support over networks
US7278107B2 (en) * 2002-12-10 2007-10-02 International Business Machines Corporation Method, system and program product for managing windows in a network-based collaborative meeting
US20050162402A1 (en) * 2004-01-27 2005-07-28 Watanachote Susornpol J. Methods of interacting with a computer using a finger(s) touch sensing input device with visual feedback
US20060287865A1 (en) * 2005-06-16 2006-12-21 Cross Charles W Jr Establishing a multimodal application voice
US7620901B2 (en) 2006-03-21 2009-11-17 Microsoft Corporation Simultaneous input across multiple applications

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Entertainment Computing - ICEC 2005: 4th International Conference" available at https://books.google.com/books?id=YpEECAAAQBAJ&pg=PA194&lpg=PA194&dq=2005+simultaneous+touch+input&source=bl&ots=rh1-OlUY0H&sig=Has8GUgzcxh5PsgJMIOQnrjOAow&hl=en&sa=X&ved=0ahUKEwiC0Ybcs7TWAhXLrVQKHUSlAMsQ6AEILDAB#v=onepage&q=2005%20simultaneous%20touch%20input&f=false *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10620819B2 (en) 2016-12-20 2020-04-14 Samsung Electronics Co., Ltd. Display apparatus and controlling method thereof

Also Published As

Publication number Publication date
US20100053110A1 (en) 2010-03-04
US8347215B2 (en) 2013-01-01
US7620901B2 (en) 2009-11-17
US20130097561A1 (en) 2013-04-18
US9164659B2 (en) 2015-10-20
US20070226636A1 (en) 2007-09-27

Similar Documents

Publication Publication Date Title
US9164659B2 (en) Simultaneous input across multiple applications
US9335860B2 (en) Information processing apparatus and information processing system
US8046701B2 (en) Peer to peer gesture based modular presentation system
US8762863B2 (en) Method and apparatus for gesture manipulation across multiple devices
JP4658480B2 (en) Utility object for specialized data entry
EP3491506B1 (en) Systems and methods for a touchscreen user interface for a collaborative editing tool
US20080040692A1 (en) Gesture input
US20050015731A1 (en) Handling data across different portions or regions of a desktop
US20110163986A1 (en) Mobile device and method for operating content displayed on transparent display panel
WO2017056487A1 (en) Communication system, information processing apparatus, and method for communication
US20140157128A1 (en) Systems and methods for processing simultaneously received user inputs
US20040139351A1 (en) Method and apparatus for generating secured attention sequence
JP5846270B2 (en) Image processing system and information processing apparatus
EP2965181B1 (en) Enhanced canvas environments
CN104412213A (en) Image display device and image display system
US20120242589A1 (en) Computer Interface Method
EP2746922A2 (en) Touch control method and handheld device utilizing the same
JP7435184B2 (en) Communication device, communication device control method, and program
KR20050074269A (en) Event processing for a navigation control device
CN100378659C (en) Insertion location tracking for controlling a user interface
JP7383959B2 (en) Display device, usage providing method, program, image processing system
CN116661656B (en) Picture interaction method and shooting display system
JP7388159B2 (en) Display device, display method
JP7363064B2 (en) Image processing device, method, and program
US20140292671A1 (en) Data sharing system and data sharing method thereof

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:037247/0647

Effective date: 20141014

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CARPENTER, CURT;EVERETT, DEBORA;KRAMER, DOUG;AND OTHERS;REEL/FRAME:037247/0609

Effective date: 20060317

STCV Information on status: appeal procedure

Free format text: NOTICE OF APPEAL FILED

STCV Information on status: appeal procedure

Free format text: APPEAL BRIEF (OR SUPPLEMENTAL BRIEF) ENTERED AND FORWARDED TO EXAMINER

STCV Information on status: appeal procedure

Free format text: EXAMINER'S ANSWER TO APPEAL BRIEF MAILED

STCV Information on status: appeal procedure

Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS

STCV Information on status: appeal procedure

Free format text: BOARD OF APPEALS DECISION RENDERED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION