GB2485145A - Audio command routing method for voice-controlled applications in multi-display systems - Google Patents
Audio command routing method for voice-controlled applications in multi-display systems Download PDFInfo
- Publication number
- GB2485145A GB2485145A GB201018268A GB201018268A GB2485145A GB 2485145 A GB2485145 A GB 2485145A GB 201018268 A GB201018268 A GB 201018268A GB 201018268 A GB201018268 A GB 201018268A GB 2485145 A GB2485145 A GB 2485145A
- Authority
- GB
- United Kingdom
- Prior art keywords
- user
- audio input
- display device
- audio
- application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title abstract description 17
- 238000012545 processing Methods 0.000 abstract description 50
- 230000005236 sound signal Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 239000000872 buffer Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
Abstract
A data processing system comprises a processing device, a plurality of display devices connected to the processing device and an audio input device connected to the processing device. A method of operating this system comprises the steps of running a plurality of applications, displaying a window on the display devices for each application, receiving an audio input for an application, determining the specific display device towards which the audio input is directed, and routing the audio input to the application whose associated window is displayed on the determined display device. The method ensures that instructions are relayed to the application for which they are intended, and reduces the risk of erroneous operation. A video input device may also be present to determine the direction of the user, as may further audio input devices to determine direction via amplitude differentials.
Description
DESCRIPTION
AUDIO ROUTING
This invention relates to a data processing system and to a method of operating the data processing system. The system is operated to route audio received from an audio input device.
In desktop computing, it is now common to use more than one display device. Traditionally, a user would have a computer with a single display device attached, but now it is possible to have more than one display device attached to the computer, which increases the usable desktop area for the worker. For example, International Patent Application Publication WO 2007/020408 discloses a display system which comprises a plurality of display is devices, each displaying respectively an image, a data processing device connected to each display device and controlling the image displayed by each display device, and a user interface device connected to the data processing device. Connecting multiple display devices to a computer is a proven method for improving productivity.
Many applications exist that can be run by a computer that require or use an audio input. For some such applications, a microphone, acting as an audio input device, is connected to the computer and this can be used to provide the audio for the application. For example, if a user is communicating with one or more colleagues who are located remotely from the user, then the user may have an Internet-based telephone conversation that will be run by an application that is running on their computer. Other utility applications may require audio inputs to control components within the application or may use voice recognition technologies to produce a text output, for example.
Given the use of multiple display devices, described above, it is common for users who operate such desktop set-ups to have a large number of different applications running at the same time. Each application that is currently running will have a respective window, which will be displayed by a respective display device. It is quite possible that a user will have running multiple applications that each utilise an audio input. In order to use these applications effectively, the user will have to ensure that the audio input being generated via the input device is directed to the correct application, for example by use the continual changing of device settings or the continual opening and closing of applications. These existing techniques are not efficient enough in many situations.
It is therefore an object of the invention to improve upon the known art.
io According to a first aspect of the present invention, there is provided a A method of operating a data processing system, the system comprising a processing device, a plurality of display devices connected to the processing device and an audio input device connected to the processing device, the method comprising the steps of running a plurality of applications, displaying a is window for each application, receiving an audio input for an application, determining the display device towards which the audio input is directed, and routing the audio input to the application with a window displayed on the determined display device.
According to a second aspect of the present invention, there is provided a data processing system, the system comprising a processing device, a plurality of display devices and an audio input device connected to the processing device, wherein the processing device is arranged to run a plurality of applications, the display devices are arranged to display a window for each application, the audio input device is arranged to receive an audio input for an application, and the processing device is arranged to determine the display device towards which the audio input is directed and to route the audio input to the application with a window displayed on the determined display device.
According to a third aspect of the present invention, there is provided a computer program product on a computer readable medium for operating a data processing system, the system comprising a processing device, a plurality of display devices connected to the processing device and an audio input device connected to the processing device, the product comprising instructions for running a plurality of applications, displaying a window for each application, receiving an audio input for an application, determining the display device towards which the audio input is directed, and routing the audio input to the application with a window displayed on the determined display device.
Owing to these aspects of the invention, it is possible to determine towards which display device a user will be talking. For example, if the user has two different voice messaging applications open, each on a different microphone-equipped monitor, the data processing system can determine to which monitor the user is speaking and route the microphone audio to the io appropriate application. Thus, the user can switch conversations simply by talking towards a different display device. Much research exists on microphone arrays, but most of this research focuses on determining the position of a person talking, not on taking actions based on the direction the person is talking.
is Preferably, the system further comprises a video input device connected to the processing device and the step of determining the display device towards which the audio input is directed comprises determining the direction of a user from the input of the video input device. An alternative method by which the data processing system can determine the display device to which the user is directing audio input is to use image processing techniques on a video input of the user. Only a single audio input device is required, and the user will provide their audio input through this device. The user's direction is determined from face-tracking or eye-tracking and will determine the display device towards which the user is currently facing. The audio input that the user makes is routed accordingly.
Advantageously, the system comprises a plurality of audio input devices and the step of determining the display device towards which the audio input is directed comprises determining amplitude differentials between the audio inputs and selecting an associated display device accordingly. The data processing system can be configured so that multiple microphones are being used. The processor can detect the amplitude of the received audio signal at each microphone and can use this information to determine the direction of the user's audio. For example, there may be two display devices in the system, each with a respective audio input device. The audio input device with the greatest amplitude of the received audio signal will indicate that the associated display device is the one to which the user is directing their audio.
In one embodiment, the number of display devices is greater than the number of audio input devices and the step of determining the display device towards which the audio input is directed further comprises mapping the determined amplitude differentials to an arrangement of the display devices. If more than two display devices are present in the system, then it is still possible to determine the direction of the user's audio output with just two audio input devices, using a process of comparing the measured amplitudes at each audio input device to a known arrangement of the display devices.
Embodiments of the present invention will now be described, by way of is example only, with reference to the accompanying drawings, in which:-Figure 1 is a schematic diagram of a data processing system, Figure 2 is a schematic diagram of a processing device of the data processing system, Figures 3 to 5 are further schematic diagrams of embodiments of the data processing system, and Figure 6 is a flowchart of a method of operating the data processing system.
A data processing system is shown in Figure 1. The system comprises a processing device 10, display devices 12 and user interface devices 14 and.
The user interface devices are a keyboard 14a and a mouse 14b. The system shown in Figure 1 is a standard desktop computer, with an additional display device 12b, which is composed of discrete components that are locally located but could equally be a device such as a laptop computer or suitably enabled handheld device such as a mobile phone or pda (personal digital assistant) all using an additional display 12b. Similarly, the system may comprise part of a networked or mainframe computing system, in which case the processing device 10 may be located remotely from the user input devices 14, the display devices 12 and the audio output devices 15, or indeed may have its function distributed amongst separate devices.
The display devices 12 show respective images 16, and the display of the images 16 is controlled by the processing device 10. One or more applications are running on the processing device 10 and these are represented to the user by corresponding windows 18, with which the user can interact in a conventional manner. A cursor 20 is shown, and the user can control the movement of the cursor 20 about the images 16 shown on the display device 12 using the computer mouse 14b, again in a totally conventional manner. The user can perform actions with respect to the running application via the user interface device 14 and these actions result in corresponding changes in the images 16, displayed by the display device 12.
The operating system run by the processing device 10 uses virtual is desktops to manage the multiple display devices 12. Each physical display device 12 is represented by a frame buffer that contains everything currently shown on that display device 12. The operating system arranges these frame buffers into a single virtual desktop. When these frame buffers are arranged in the virtual desktop 22 in the same relative positions in which the physical display devices 12 are relatively placed, then the operating system can draw objects on all the display devices 12 in a natural way. The virtual desktop is a combination of the respective images I 6a and I 6b being shown by the display devices 12. If the user moves the mouse 14a such that the cursor 20 moves right off the edge of one display device 1 2a, then the cursor 20 appears on the left of the display device 12b to the right. Similarly a window 18 spread across several display devices 12 appears properly lined up between the display devices 12.
Figure 2 illustrates the virtual desktop 22 within the processing device 10. The images 16a and 16b being shown by the respective display devices 12 make up the virtual desktop 22. Within the processing device 10 is a processing area 24, which is running the applications 26. A window 18 is displayed for each application 26 that is running. These windows 18 form part of the virtual desktop 22, with a suitable background making up the remainder.
The virtual desktop 22 may be an addressable location within a central processing unit, or may be part of dedicated graphics component which is managing the overall image output. The processing area 24 will form part of a central processing unit that is controlling the overall operation of the processing device 10.
An operating system within the processing device 10 manages the interaction between the user inputs from the user interface devices 14 and the applications 26, and adjusts the virtual desktop 22 accordingly. For example, the user may drag windows 18 around the virtual desktop 22. The images 16 are updated to reflect the change in location of the windows 18. The user can drag a window 18 such that it will move from one display device 12 to another display device 12. For example, the window 18a may be moved by the user from the primary display device 12a to the secondary (additional) display is device 12b. The application 26 represented by the window 18a may be an email client and the user wishes to keep the application open and running while they start a different task. The virtual desktop 22 will be updated to reflect the movement of the window I 2a, as will the images 16, all in real-time.
The system is configured to determine which display device 12 a user 30 is currently operating, as shown in Figure 3. In this Figure, a user 30 has two different voice messaging applications open, represented by the windows 18, each on a different display device 12. A microphone (an audio input device) 32 is provided for the user 30 to speak into, or one of the display devices 12 could be microphone-equipped. Software in the processing device 10 can determine towards which display device 12 the user 30 is speaking and route the microphone audio to the appropriate application 36. Thus, the user can switch conversations simply by talking towards a different display device 12.
The system is arranged to run a plurality of applications 26 on the processing device 10, to display a window 18 on a display device 12 for each application 26, to receive an audio input for an application 26, to determine the display device 12 towards which the audio input is directed, and to route the audio input to the application 26 with a window 18 displayed on the determined display device 12. In this way, the system routes the audio input from the user to the correct application 26. The user 30 does not need to make any changes to the configuration of the running applications, the audio input software will detect the display device 12 towards which the user is directing the audio, and route that audio accordingly.
If there is a single audio input device 32, as in Figure 3, then the routing can be made based upon detection of a physiological parameter of the user 30, such as face or eye direction. In this case, a camera 34 (a video input device), which is connected to the processing device 10, can determine the direction 36 in which the user 30 is pointing, through face tracking. This allows a determination of the display device 12 towards which the audio input from the user 30 is directed. If the user 30 changes their orientation, then this is detected by the camera 34 and the routing of the audio received from the is microphone 32 is changed by the processor 10.
There are different methods for determining the directional element of the audio. If there are multiple audio input devices 32, for example with each display device 12 having its own microphone 32, then the direction in which the user is speaking can be determined using amplitude differentials of the inputs to the different microphones 32. The user 30 can be assumed to be closest to the microphone 32 for the desired display device 12, and therefore this microphone 32 will have a louder input than other microphones 32. This is illustrated in Figure 4. Each display device 12 in the system is provided with a respective audio input device 32.
The processor 10 is connected to both of the microphones 32a and 32b and can measure the amplitude of the audio signal received by each individual microphone 32. This amplitude can be calculated on a ten second running average, for example. From the different measured amplitudes, the processor can detect which of the microphones 32a and 32b is receiving a louder input from the user 30. From this information, it can be inferred that the user 30 is directing their audio at the display device 12 that is associated with the audio input device 32 with the greater measured amplitude. The audio from the microphone 32 is routed accordingly.
In Figure 4, the user 30 can be seen to be closer to the right-hand display device 12b, having shifted their position from half-way in-between the two display devices 12 (in Figure 3). The user can be assumed to be directing their audio output to the application represented by the window 18b. The two microphones 32 will both pick up the user's audio output, but the amplitude of the received signal will be different. The amplitude of the signal received by the microphone 32b will be greater than the amplitude of the signal received by the microphone 32a and the audio from the user 30 will be routed accordingly to the correct application.
If more than two display devices 12 are present then each can have their own dedicated audio input device 32 and the comparison described above of the amplitudes of the received audio signals can be extended to is determine to which of three display devices, for example, the user is directing their audio. However, even with more than two display devices 12, a pair of audio input devices 32 can be used to determine which display device 12 is the target of the audio. This is illustrated in Figure 5, where the user 30 has connected their laptop computer to the data processing system, and is using the display 12c of the laptop computer to further extend the usable display area.
The three display devices 12 are arranged in a horizontal line, and the two display devices 12a and 12b are provided with respective audio input devices 32a and 32b. The position of the user 30 changes the amplitude of the audio signals received by the audio input devices 32, both in relative terms and in absolute terms. When audio input device 32a has the greater received amplitude, the leftmost display device 12a is the target of the user's audio.
When the audio input device 32b has the greater received amplitude, then the absolute levels of received audio need to be considered, in order to determine whether the user is targeting display device 12b or 12c. Whichever display device 12 is the target of the user's audio, the routing is arranged accordingly.
Effectively, when the number of display devices 12 is greater than the number of audio input devices 32 and the determining of the display device 12 towards which the user's audio input is directed involves mapping the determined amplitude differentials to a stored arrangement of the display devices 12 maintained by the processor 12. In this way the audio is routed to the correct application. An arrangement such as that shown in Figure 5 may require the user 30 to first perform an initialisation procedure, whereby test audio is provided by the user 30 and the processor configures the display device routing by then requesting the user to make a selection between offered display device 12.
The method of operating the data processing system is summarised in Figure 6. The method comprises, firstly step SI of running a plurality of applications, secondly step S2 of displaying a window for each application, thirdly step S3 of receiving an audio input for an application, fourthly step S4 of is determining the display device towards which the audio input is directed, and finally step S5 of routing the audio input to the application with a window displayed on the determined display device. In this way a user who has several applications running at the same time, which can use an audio input, will have their audio routed to the correct application.
The principal step in the method is step S4 where the determination is made of the display device towards which the user is directing their audio. The user does not identify the desired display device directly, but the target display device is inferred, either from the user's position or from calculations made from the user's audio input, or a combination of these two factors. As discussed above, the user's eyes and/or face can be tracked with a camera, to determine the display device towards which the user is directing the audio.
Alternatively, or additionally, different amplitudes calculated at different microphones can be used to determine the direction of the user's audio input.
In step S5, the audio input from the user is then routed to an application that has a window on the display device that was determined in step S4. If there is more than one such application, then a priority decision has to be taken as to which application should receive the inputted audio. This could be based on a priority table, for example, or may be based upon the detection of a specific prior action. In the latter case, for example a video conferencing application may have made an output to a user, and the user's audio input is routed to that specific application, as a response.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB201018268A GB2485145A (en) | 2010-10-29 | 2010-10-29 | Audio command routing method for voice-controlled applications in multi-display systems |
| PCT/GB2011/001533 WO2012056198A1 (en) | 2010-10-29 | 2011-10-26 | Audio routing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB201018268A GB2485145A (en) | 2010-10-29 | 2010-10-29 | Audio command routing method for voice-controlled applications in multi-display systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| GB201018268D0 GB201018268D0 (en) | 2010-12-15 |
| GB2485145A true GB2485145A (en) | 2012-05-09 |
Family
ID=43401488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| GB201018268A Withdrawn GB2485145A (en) | 2010-10-29 | 2010-10-29 | Audio command routing method for voice-controlled applications in multi-display systems |
Country Status (2)
| Country | Link |
|---|---|
| GB (1) | GB2485145A (en) |
| WO (1) | WO2012056198A1 (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10657674B2 (en) | 2016-06-17 | 2020-05-19 | Immersive Robotics Pty Ltd. | Image compression method and apparatus |
| CN106201419A (en) * | 2016-06-28 | 2016-12-07 | 乐视控股（北京）有限公司 | Audio frequency input control method, device and terminal |
| CN110494193A (en) | 2017-02-08 | 2019-11-22 | 因默希弗机器人私人有限公司 | User into multiplayer place shows content |
| US10467088B2 (en) | 2017-02-08 | 2019-11-05 | Microsoft Technology Licensing, Llc | Audio system maintenance using system call monitoring |
| US10310800B2 (en) | 2017-02-08 | 2019-06-04 | Microsoft Technology Licensing, Llc | Selective routing of audio between applications |
| CN111837384A (en) | 2017-11-21 | 2020-10-27 | 因默希弗机器人私人有限公司 | Frequency component selection for image compression |
| EP3714602A4 (en) | 2017-11-21 | 2021-07-28 | Immersive Robotics Pty Ltd | Image compression for digital reality |
| KR102527082B1 (en) | 2018-01-04 | 2023-04-28 | 삼성전자주식회사 | Display apparatus and the control method thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000278626A (en) * | 1999-03-29 | 2000-10-06 | Sanyo Electric Co Ltd | Multiple screens sound output controller |
| JP2002091491A (en) * | 2000-09-20 | 2002-03-27 | Sanyo Electric Co Ltd | Voice control system for plural pieces of equipment |
| US20080120141A1 (en) * | 2006-11-22 | 2008-05-22 | General Electric Company | Methods and systems for creation of hanging protocols using eye tracking and voice command and control |
| JP2009182769A (en) * | 2008-01-31 | 2009-08-13 | Kenwood Corp | On-vehicle device, program and display control method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3530591B2 (en) * | 1994-09-14 | 2004-05-24 | キヤノン株式会社 | Speech recognition apparatus, information processing apparatus using the same, and methods thereof |
| US6219645B1 (en) * | 1999-12-02 | 2001-04-17 | Lucent Technologies, Inc. | Enhanced automatic speech recognition using multiple directional microphones |
| GB0516713D0 (en) | 2005-08-13 | 2005-09-21 | Newnham Res Ltd | A display system and method of operating a display system |
-
2010
- 2010-10-29 GB GB201018268A patent/GB2485145A/en not_active Withdrawn
-
2011
- 2011-10-26 WO PCT/GB2011/001533 patent/WO2012056198A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000278626A (en) * | 1999-03-29 | 2000-10-06 | Sanyo Electric Co Ltd | Multiple screens sound output controller |
| JP2002091491A (en) * | 2000-09-20 | 2002-03-27 | Sanyo Electric Co Ltd | Voice control system for plural pieces of equipment |
| US20080120141A1 (en) * | 2006-11-22 | 2008-05-22 | General Electric Company | Methods and systems for creation of hanging protocols using eye tracking and voice command and control |
| JP2009182769A (en) * | 2008-01-31 | 2009-08-13 | Kenwood Corp | On-vehicle device, program and display control method |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2012056198A1 (en) | 2012-05-03 |
| GB201018268D0 (en) | 2010-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2012056198A1 (en) | Audio routing | |
| US20230052490A1 (en) | Remote user interface | |
| US11212449B1 (en) | User interfaces for media capture and management | |
| KR102604685B1 (en) | Device control using gaze information | |
| US10152967B2 (en) | Determination of an operational directive based at least in part on a spatial audio property | |
| US20240036703A1 (en) | Electronic message user interface | |
| US11100909B2 (en) | Devices, methods, and graphical user interfaces for adaptively providing audio outputs | |
| US10649622B2 (en) | Electronic message user interface | |
| US10386995B2 (en) | User interface for combinable virtual desktops | |
| US9632618B2 (en) | Expanding touch zones of graphical user interface widgets displayed on a screen of a device without programming changes | |
| US9753579B2 (en) | Predictive input system for touch and touchless displays | |
| CN108885485A (en) | Digital assistants experience based on Detection of Existence | |
| US10999088B2 (en) | Proximity and context-based telepresence in collaborative environments | |
| EP3108422A1 (en) | Message user interfaces for capture and transmittal of media and location content | |
| JP6718460B2 (en) | Method, system, and computer program for device interaction via heads-up display | |
| WO2015183756A1 (en) | Message user interfaces for capture and transmittal of media and location content | |
| JP7440625B2 (en) | Methods and computer programs for controlling the display of content | |
| US10848895B2 (en) | Contextual center-of-gravity for audio output in collaborative environments | |
| WO2010144096A1 (en) | Data processing system and method | |
| KR20160011421A (en) | The Apparatus and Method for Display Device | |
| KR102446243B1 (en) | Methods and user interfaces for sharing audio | |
| KR20190070730A (en) | Apparatus, method and computer program for processing multi input |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 732E | Amendments to the register in respect of changes of name or changes affecting rights (sect. 32/1977) |
Free format text: REGISTERED BETWEEN 20121004 AND 20121010 |
|
| WAP | Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1) |