AU778428B2 - Local sound and graphic feedback - Google Patents

Description

V

S&F Ref: 536267

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan Timothy Merrick Long, Mark Anthony Greene Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 Local Sound and Graphic Feedback Invention Title: ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PQ5853 [32] Application Date 25 Feb 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815c -1- LOCAL SOUND AND GRAPHIC FEEDBACK Technical Field of the Invention The present invention relates generally to computer networks, and in particular, to remote terminals in such networks.

Background Art General-purpose computers typically have significant application processing capability. Such computers also typically incorporate non-volatile, mass storage elements. In contrast, 'thin client' devices, such as JavaStation NC (Network Computer) from Sun Microsystems® Inc., typically provide significant application processing capability, but have little, if any, non-volatile mass storage. Conventional remote terminals, on the other hand, typically have no application processing capability, and furthermore, no non-volatile mass storage resources. Since these remote terminals provide no application processing power, such processing power is generally provided by an external host processor (for example a network server or general-purpose computer) S 15 connected thereto. A remote multimedia terminal is a remote terminal having significant multimedia capabilities, including complex screen graphics, animation resources and audio capability, while nonetheless suffering the same constraints as the aforementioned ooooo conventional remote terminal.

As performance requirements increase, and the distances between remote 20 terminals and remote multimedia terminals, and their associated servers and/or external processors, also increase, significant delays can be incurred in the course of user interaction, such as key strokes at the remote multimedia terminal. Thus, for example, when a user of a remote multimedia terminal initiates a key stroke, feedback is typically not provided until the action has been detected and processed by the external processor, and a response transferred back to the remote terminal. This can result in the user 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc pressing a key multiple times, on the incorrect perception that the remote terminal has not registered the key stroke. The aforementioned delay between user interaction and feedback can be caused by either or both communication bandwidth limitations between the server and the remote terminal, and asynchronous processing of the user event by the server.

When a user makes direct use of a general purpose computer, the computer itself often provides indirect indications that reveal to the user that the computer is busy, and is accordingly not yet ready for the next input by the user. The noise generated by a disk drive in the computer, or Light Emitting Diodes (LEDs) associated with operation of the disk drive, are typical examples of these indirect indications.

If, however, the user accesses the general purpose computer using a remote terminal, the aforementioned indirect indications are typically absent, and the terminal can consequently often sit in apparent "mute paralysis", not providing any indication to the user whether the terminal is ready for the next input from the user or not. This is S 15 particularly troublesome when a user operation produces no perceptible feedback to the S. user, who consequently is unable to detect any response to the user operation.

One example of such a situation could arise if the user, either deliberately or accidentally, depresses a terminal keyboard key by just clipping the edge of the key, in which case the user would be unsure if an effective key-press has, in fact, taken place. In go a second example, pressing "OK" on a dialogue box might result in the dialogue box being almost immediately replaced by a very similar or identical dialogue box, in which •event the user would again be unsure if a legitimate response had been provided to the initial dialogue box.

Some traditional systems provide the user of the remote terminal with some form of local feedback. One such example is a 'key-click' sound, which is produced whenever 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -3the user presses a key on the remote terminal. One disadvantage of this method is that the feedback is 'hard-wired', and. can accordingly, not be customized. Furthermore, such key-click sounds are short, unconditional and independent of the host's readiness for further input. Accordingly, such hard-wired feedback does not aid the user in deciding if he or she may continue with the next operation.

Disclosure of the Invention It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

According to a first aspect of the invention, there is provided a method of providing a stand-by signal at a terminal, dependent upon a user input for an application running on a remote host processor, said user input being made at said terminal, the terminal being connected to the remote host processor, said method comprising steps of: communicating by the terminal to the host processor an event corresponding to S"the user input; 15 storing, in the terminal, standby-signal data which is programmable by the remote host processor; providing by the terminal a stand-by signal, derived from the standby signal data, dependent upon the user input; and terminating provision of the stand-by signal upon receipt, by the terminal, of a 0% 0 S 20 response to said event by the remote host processor.

According to another aspect of the invention, there is provided a terminal adapted to provide a stand-by signal dependent upon a user input for an application running on a remote host processor, said user input being made at said terminal, the terminal being connected to the remote host processor, said terminal comprising: 180504 536267AUclean -4communicating means for communicating to the host processor an event corresponding to the user input; local storage means, adapted to store standby-signal data; programming means, responsive to communication from the host processor, for programming said standby-signal data; selection means, responsive to said user input, for deriving the standby-signal from said standby-signal data means for providing by the terminal said stand-by signal dependent upon the user input; and means for terminating provision of the stand-by signal upon receipt by the terminal of a response to said event by the remote host processor.

According to another aspect of the invention, there is provided a system for "providing a stand-by signal at a terminal, dependent upon a user input for an application running on a remote host processor, said user input being made at said terminal, the I 15 terminal being connected to the remote host processor, wherein: the host processor is adapted to run said application; .000 the remote terminal is adapted to receive said user input, and to provide a stand- 0 0 by signal dependent upon said user input; wherein standby-signal data, being programmable by the remote host processor, is stored :00 20 in the terminal, and wherein the standby-signal, provided by the terminal in response to the user input, is derived from the standby-signal data; and wherein the terminal communicates to the host processor an event corresponding to the user input, said event being received by the host processor; and wherein provision of the stand-by signal is terminated upon a response to said event by the application being received by the remote host processor.

180504 536267AU clean According to another aspect of the invention, there is provided a computer readable memory medium for storing a program for apparatus adapted to provide a standby signal at a terminal, dependent upon a user input for an application running on a remote host processor, said user input being made at said terminal, the terminal being connected to the remote host processor, said program comprising: code for a communicating step for communicating by the terminal to the host processor an event corresponding to the user input; code for a storing step for storing in the terminal standby-signal data which is programmable by the remote host processor; code for a providing step for providing by the terminal said stand-by signal, derived from the standby signal data, dependent upon the user input; and code for a terminating step for terminating provision of the stand-by signal upon receipt, by the terminal, of a response to said event by the remote host processor.

Brief Description of the Drawings 0 o 15 A number of preferred embodiments of the present invention will now be described with reference to the drawings, in which: 0000 o000 Fig. 1 shows a system block diagram according to a preferred embodiment of the invention; •.go gee• 0 Fig. 2 shows details ofa user interface in Fig. 1; and 20 Fig. 3 is a process flow diagram for the system of Fig. 1.

Detailed Description including Best Mode Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have, for the purposes of this description, the same function(s) or operation(s), unless the contrary intention appears.

180504 536267AUclean The terms "remote terminal" and "multimedia remote terminal" are used interchangeably unless noted otherwise.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

*9SS S* r r So 5 oo C.

QS S 180504 536267AU_clean -6- The present specification discloses a hardware arrangement, and a computer readable medium comprising a computer program, which can be used either alone, or in combination, for performing the operations of the methods. The computer readable medium is taken herein to include any transmission medium for communicating the computer program between a source and a designation. The transmission medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer, and particularly in the present case, with a terminal controller. The transmission medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein.

SFig. 1 shows a computer network comprising a general purpose computer 104 upon which application software 100 is supported, as depicted by a bilateral arrow 102.

The system shown in Fig. 1 can, for example, be used in a domestic environment. In this oo case, remote terminals can be distributed around a house, these being connected to the .oo..i general purpose computer 104. One such terminal can have comprehensive multimedia capabilities, and be situated in the kitchen. This terminal can be used to access culinary S. S and recipe information, where interaction between the user and the computer system is afforded by means of a touch-screen for user inputs, and a mixture of graphics and audio *for output. The computer 104 communicates with such an integrated remote multimedia terminal, depicted as dashed box 112. Communication between the computer 104 and the remote multimedia terminal 112 is depicted by a bilateral arrow 106, the arrow representing a communication channel between the computer 104 and the remote terminal 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -7- 112. The general purpose computer 104 is configured to run the application programs 100, providing a user interface via the remote multimedia terminal 112. As already noted, there is typically little computing power in the terminal 112. The remote terminal 112 includes a communication interface 108 which provides the necessary mediation between the general purpose computer 104 and a remote multimedia terminal controller 114. The terminal 112 further comprises a small amount of local temporary storage 118, an audio and graphic output interface 126, and user interface elements 122. The remote multimedia terminal 112 receives screen graphic data and audio data from the computer 104 via the communications channel 106. The screen graphic data is used to update graphics on the screen 200 (see Fig. Audio data is used to play sounds on an audio speaker 202 (see Fig. 2).

Fig. 2 shows more detail on the audio and graphic output elements 126 and the user interface elements 122. The audio and graphic output elements 126 comprise a j graphic output screen 200, and an audio output speaker 202, both connected to a 15 communication bus 124. The user interface elements 122 comprise keys, or buttons 204, and a touch screen 206, these communicating by means of a bus 120. The audio and graphic output devices 126 and the user interface elements 122 shown in Fig. 2 are illustrative only, and not restrictive. Thus, for example, additional user interface elements such as microphones and generic dials (not shown) can be incorporated.

20 Fig. 3 shows an exemplary feedback process flow which depicts operation of a preferred embodiment of the invention. A detection process 300 detects a user event (for example a keystroke), or alternatively, is enlivened by initialization of a software application at application runtime. The occurrence of a user event, for example corresponding to a key stroke, is detected by the remote multimedia terminal controller 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -8- 114 (see Fig. The detection of initialization of a software application is performed by the computer 104.

In a first arrangement, and considering user events flowing from the user (ie the terminal 112) to the host processor 104 (see Fig. event distribution software on the host processor 104 notes a sequence number for each incoming user event, before distributing the event to the application program 100, that should respond to the event.

Considering responses from the host processor 104 in this arrangement, graphics, sound and other output generated by the application program 100 for the user of the terminal 112 are annotated with the sequence number of the event that the distribution software most recently distributed to the application 100. In this manner the remote terminal 112 can establish a "pairing' between incoming user events, and the corresponding responses from the application 100. In this arrangement, an application program 100 is not required to have any logic specifically relating to process event sequence numbers, and the event distribution software on the host processor 104 is sufficient to establish of event/response 15 pairs. This arrangement is particularly advantageous in that third party application programmers do not need to include software routines for provision of feedback at the terminal. This arrangement is described as follows in relation to Fig. 3.

Each event detected in the detection process 300 is tagged with a monotonically oo increasing event sequence number, in a sub-process 326. This sequence number is used o 20 in a manner described below in relation to responses to the detected event by the host processor. Following event detection or application initialization detection in the subprocess 300, this detection is communicated in a sub-process 316. This communication comprises two alternate branches, namely a local feedback branch depicted by an arrow 322, and a remote (or host) feedback branch depicted by an arrow 324.

130201; 16:04; 16/02/01; 09:39 AM 536267AU.doc -9- Considering the local branch 322, the event communication process 316 communicates detection of the event, in this case a user event, to a sub-process 304 which determines an appropriate feedback response. Thereafter, a decision sub-process 306 determines whether appropriate feedback data is located in the local store 118 (see Fig.

If such feedback data is, in fact, already in local storage 118, then the feedback process is directed in accordance with a "YES" arrow to a feedback-enabling decision sub-process 332. If the decision sub-process 332 returns a "YES", then the feedback process is directed to a feedback response sub-process 308, whereupon the requisite feedback is provided to the user. For example, if the event was a keystroke, then the appropriate local feedback response provided by the sub-process 308 is a distinctive chime sound.

If, on the other hand, the sub-process 306 determines that appropriate feedback data is not available in the local storage 118, then the feedback process is directed in 4. accordance with a "NO" arrow to the default local feedback selection sub-process 310.

5i 15 This process 310 selects a default feedback data which is to be found in the local storage 118, and provides this data to the feedback-enabling decision sub-process 332. If the decision sub-process 332 returns a "YES", then the feedback process is directed to the feedback response sub-process 308, where upon the default feedback signal is provided to the user. Thus, for example, the local storage 118 might presently contain only a typical 00 06 ,000. 20 "alert" signal, such as a sound made by a siren. Although such a sound is not considered appropriate feedback for a keystroke, it nonetheless would be selected as a default 0000 feedback signal in the present instance. This default feedback option is used in order to provide a low-latency response feedback to the user, even in the event that appropriate, customized feedback is temporarily not locally available. In fact, the default feedback for 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc touch-screen inputs and keypad key presses provided by the feedback response subprocess 308 in the present arrangement is preferably a soft "blue-noise" sound.

In order to minimise use of default feedback, the feedback process is also directed, after the decision sub-process 306, to a feedback data transfer sub-process 312, which fetches desired feedback data from the computer 104. After this data retrieval, this branch of the feedback process is directed to a "STOP" block 314. In summary at this point, user events, i.e. interactions between the user and the user interface elements 112, elicit low-latency feedback to the user by means of feedback data which is locally stored in temporary storage 118 in the remote terminal 112 itself. This feedback data can be updated from the host computer 104, as desired.

Returning to the 'host' branch of the feedback process, i.e. as depicted by the arrow 324, after the event communications sub-process 316, the feedback process is directed to a response determination sub-process 318. This branch of the feedback @9 Se :"°-:process is responsible for downloading appropriate feedback data to the remote 15 terminal 112 upon initialization of an application, and/or updating feedback data in the S. 055 o••0€ remote terminal 112 'on the fly', if desired. The response-determination sub-process 318

S•

thus determines the appropriate feedback for the user event or application initialization as •communicated by the communication sub-process 316, and the feedback process is 59*5 thereafter directed to a decision sub-process 320. This sub-process 320 determines if S so 20 feedback data updating is required. If so, the feedback process is directed, as indicated by a 'YES' arrow, to the feedback data transfer sub-process 312, which ensures that ,i appropriate data is transferred from the computer 104 to the remote terminal 112, and stored in the temporary storage 118. Thereafter, the feedback process is directed to the 'STOP' block 314. If the feedback data updating sub-process 320 determines that data 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -11updating is not required, then the feedback process is directed straight to the 'STOP' block 314.

Audio and graphic feedback data kept in local storage 118 can be changed at any time by the application program 100. In the preferred embodiment, each multimedia application program 100 stores individual graphic and audio data in the temporary storage area 118 on initialization of the application 100.

In the present arrangement, whether the feedback response provided by the subprocess 308 is the default response (preferably the blue noise sound), or an appropriate local feedback response (eg the distinctive chime sound), the feedback provided by the sub-process 308 is continuous (ie either a continuous blue noise sound, or an ongoing sequence of chime sounds), which terminates only when a response to the touch-screen or key event previously sent to the host processor has been received. This ensures the user knows that the host is busy processing the response to the event, and (ii) the user should wait until the feedback stops before taking a next step.

15 Coordination between generated events (as detected by the sub-process 300) and responses from the host, is achieved as follows in the first arrangement. Each event S"detected by the sub-process 300 is tagged, in a sub-process 326, with a monotonically S•increasing event sequence number. Similarly, each response by the host is similarly tagged with the same monotonically increasing response sequence number in a sub- 20 process 328. The event and response sequence numbers are compared in a subsequent decision step 330, which is thereby able to determine when the most recent (ie "current") outstanding event has been responded to. If the decision sub-process 330 determines that the current event has indeed been responded to, then the feedback process is directed in accordance with a "YES" arrow to the feedback-enabling decision sub-process 332, which then returns a thereby terminating the feedback being provided by the sub- 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -12process 308. The feedback-determining sub-process 332 enters a loop under this condition, as indicated by the "NO" arrow 334. If, however, the decision sub-process 330 determines that the current event has not yet been responded to, the feedback process is directed in accordance with a "NO" arrow 336, thereby maintaining the decision subprocess 330 in a loop. While the decision sub-process 330 is in the aforementioned loop, the feedback enabling decision sub-process 332 returns a "YES", thereby maintaining the continuous feedback being provided by the sub-process 308.

In some circumstances, the aforementioned first arrangement for coordination between generated events and responses from the host is ineffective, and a second arrangement can be used. Consider the case, for example, when the application 100 maintains a continuous animation (ie an ongoing sequence of animation updates) on the remote terminal 112 while the application 100 is processing an incoming event from the terminal 112. In this case there is a continuous stream of graphic data flowing from the host 104 to the terminal 112, however each animation update, while appearing to constitute a response from the application 100, does not in fact constitute that response, but is merely a part of that response. If the first described arrangement is employed in this situation, the first outgoing graphical command (ie response) associated with the animation updates would be tagged with the event sequence number of the incoming event as soon as the incoming event is distributed to the application 100. The first arrangement previously described would then effect a pairing between the incoming event from the terminal 112 and the first outgoing graphical command, terminating the feedback once the pairing was established. This would, however, disregard the actual time the application 100 takes to actually respond to the incoming event.

Accordingly, in a second arrangement, the incoming event that is distributed to the application 100 has included therein the corresponding event sequence number. The 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc -13application 100 itself, rather than the event distribution software on the host 104, annotates output data that is in response to the particular event with the sequence number of that particular event. The application in this case knows the composition of a correct "response" which in the above animation example, constitutes the entire animation sequence plus the actual output response which is generated in response to the incoming user event. In this manner the remote terminal 112 can make a precise pairing between a generated event, and the response to that event from the remote system.

Within the application 100, this second arrangement is implemented by keeping a record of the most recently received event sequence number being processed by each "synchronous work unit". A synchronous work unit may be a single application thread, in which case only one event is processed at a time within the thread, or alternately, may be a context data structure, in which case multiple event are processed simultaneously.

All outputs associated with a synchronous work unit are annotated with the current •.recorded sequence number.

The provision of local sound and graphic feedback by the terminal 112 is preferably implemented in dedicated hardware such as one or more integrated circuits S"performing the functions or sub functions of local feedback provision. This dedicated hardware can be operative either alone, or in conjunction with associated firmware, both of which reside in the terminal controller 114 of the remote terminal 112. Alternately, the provision of local sound and graphic feedback can be implemented substantially in software, which can reside on a comiputer readable medium such as a memory device in the terminal controller 114. The computer readable medium is taken herein to include any transmission medium for communicating the computer program between a source and a designation. The software can thus, for example, be loaded into the aforementioned memory in the controller 114 from external storage devices (not shown) such as magnetic 130201; 16:04; 16/02/01; 09:39 AM 536267AU.doc -14or optical disks, memory chips, or other storage devices suitable for interfacing with a computer, and particularly in the present case, with the terminal controller 114. The transmission medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM mobile telephone system. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

Industrial Applicability It is apparent from the above that the embodiment of the invention is applicable to the computer, and data processing industries.

The foregoing describes only one embodiment of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

o• S.i ooo oooo 130201; 16:04; 16/02/01;09:39 AM 536267AU.doc

Claims (10)

1. 3. A method according to claim 2; wherein said continuous tone is a blue noise tone.
2. 4. A method according to claim 1; wherein said terminating step comprises further steps of: 180504 536267AU clean -16- tagging said event with an event sequence number; tagging said response with a response sequence number; comparing the event sequence number to the response sequence number; and terminating provision of the standby-signal if the event sequence number matches the response sequence number. A method according to claim 4; wherein: the step of tagging the event comprises tagging, by the terminal, of the event with an event sequence number; and wherein: the step of tagging the response comprises tagging, by the remote host processor, the response with the same event sequence number.
3. 6. A method according to claim 4, wherein: the step of tagging the event comprises tagging, by the terminal, the event with S 15 an event sequence number, said event sequence number being communicated to the *i application; and wherein: the step of tagging the response comprises tagging, by the application, the .response with the same event sequence number. 20 7. A method according to claim 1, wherein the programming of the standby-signal data is performed by an application program running on the host computer.
4. 8. A terminal adapted to provide a stand-by signal dependent upon a user input for an application running on a remote host processor, said user input being made at 180504 536267AUclean -17- said terminal, the terminal being connected to the remote host processor, said terminal comprising: communicating means for communicating to the host processor an event corresponding to the user input; local storage means, adapted to store standby-signal data; programming means, responsive to communication from the host processor, for programming said standby-signal data; selection means, responsive to said user input, for deriving the standby-signal from said standby-signal data means for providing by the terminal said stand-by signal dependent upon the user input; and means for terminating provision of the stand-by signal upon receipt by the terminal ofa response to said event by the remote host processor. e S 15 9. A terminal according to claim 8; wherein said event is tagged with an S• event sequence number, and said response is tagged with a response sequence number, said standby-signal terminating means further comprising: comparing means for comparing the event sequence number to the response eeeo sequence number; wherein provision of the standby-signal is terminated if the event 20 sequence number matches the response sequence number. A terminal according to claim 8, further comprising user input means whereby said user input is provided to the terminal, said input means comprising at least one of a microphone, touch keys, and a touch screen. 180504 536267AUclean -18-
5. 11. A terminal according to claim 8, further comprising: presentation means for presenting the standby-signal to the user, wherein the presentation means comprise at least one of: graphic output means; and audio output means.
6. 12. A system for providing a stand-by signal at a terminal, dependent upon a user input for an application running on a remote host processor, said user input being made at said terminal, the terminal being connected to the remote host processor, wherein: the host processor is adapted to run said application; the remote terminal is adapted to receive said user input, and to provide a stand- by signal dependent upon said user input; wherein standby-signal data, being programmable by the remote host processor, is stored in the terminal, and wherein the standby-signal, provided by the terminal in response to the user input, is derived from the standby-signal data; and wherein the terminal communicates to the host processor an event corresponding to the user input, said event being received by the host processor; and wherein °0•0 provision of the stand-by signal is terminated upon a response to said event by 20 the application being received by the remote host processor.
7. 13. A computer readable memory medium for storing a program for apparatus adapted to provide a stand-by signal at a terminal, dependent upon a user input for an application running on a remote host processor, said user input being made at said 180504 536267AU clean -19- terminal, the terminal being connected to the remote host processor, said program comprising: code for a communicating step for communicating by the terminal to the host processor an event corresponding to the user input; code for a storing step for storing in the terminal standby-signal data which is programmable by the remote host processor; code for a providing step for providing by the terminal said stand-by signal, derived from the standby signal data, dependent upon the user input; and code for a terminating step for terminating provision of the stand-by signal upon receipt, by the terminal, of a response to said event by the remote host processor.
8. 14. A method of providing stand-by signal at a terminal substantially as described herein with reference to the accompanying drawings.
9. 15. A terminal substantially as described herein with reference to the accompanying drawings.
10. 16. A system for providing stand-by signal at a terminal, substantially as described herein with reference to the accompanying drawings. DATED this twenty-second Day of June, 2004 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON 180504 536267AU clean