JP2004355059A - Operation display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for displaying an operation such as a mouse input or a keyboard input which cannot be comprehended only by watching a screen on the screen by using texts or animation more specifically. <P>SOLUTION: An event handler 141 for monitoring event generation from a window system to an application, an event conversion table 200 for associating the event with an user operation, and an input event display 142 for displaying the result of event conversion onto the screen by using the animation, icons, the letters, etc. are provided. Since the operation information about which mouse button is pressed etc. can be acquired from the screen, the aspect of specific operations can be comprehended only by watching the screen. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a display method for displaying the contents of an input operation on a computer in an easy-to-understand manner, and more particularly to a system for displaying the contents of an input operation during screen sharing or automatic execution in a window system.
[0002]
[Prior art]
By connecting a USB camera or the like to a personal computer, a TV phone (also called video communication, video chat, etc.) can be easily made. 2. Description of the Related Art In a video conference system in which a remote location is connected by a TV phone to hold a conference, functions such as a whiteboard and application sharing can be used in addition to video (for example, see Patent Document 1 and Non-Patent Document 1). These are functions that allow multiple computers to share screens and windows. The same window is displayed on the screens of multiple computers connected to the video conferencing system via the network, and input to the shared application (displayed Window input) can also be performed from each computer. As a similar function, there is a function called Remote Assistance, which allows a supporter to view and operate the screen of a personal computer that requires assistance from a remote personal computer in order to support the operation of the computer (see Non-Patent Document 2).
[0003]
There are two main ways to share screens and windows among multiple computers connected via a network. One is to run an application on one computer and send an event that occurs in response to a user input to the application via a communication function to the computer where the application is running. This is a method of performing processing, transmitting screen information reflecting the result to each computer via a communication function, and displaying the screen information on the screen of each computer. The other method is to share data such as graphics and text, rather than sharing screen information.The same application runs on each computer, and events that occur in response to user input to the application are monitored. This is a method of sending data to another computer via a communication function. Both use the technology of communicating events, where events are used in the event driven programming paradigm used in window systems. Event-driven programming is a programming technique in which an event occurs when a user performs an operation on an application, and code for processing the event is written. For example, when the user clicks the mouse, a hardware interrupt occurs, and the OS kernel and the device driver for the mouse process and transmit a click signal to the window system. The window system determines in which window the mouse was clicked and informs the corresponding application of the click event. When a click event occurs, a click event handler in the application responds and performs event processing according to the code written by the developer. Therefore, an event can be sent to another computer, and if the event is transmitted to the application, the application running on the other computer can be operated.
[0004]
By the way, in order to explain a specific operation of a personal computer, a method in which an explainer explicitly adds an explanation has been used for a long time. For example, a hard copy of the screen or window, with descriptions such as "Click the right mouse button" written on the screen and the screen transitions can be recorded. Please click the button.]
[0005]
[Patent Document 1]
JP-A-6-337915 [Non-Patent Document 1]
NetMeeting Home, http: // www. Microsoft. com / windows / netmeeting /
[Non-patent document 2]
Step-by-Step Guide to Remote Assistance, http: // www. Microsoft. com / windowsxp / pro / techinfo / deployment / remoteguide /
[0006]
[Problems to be solved by the invention]
With the application sharing and remote assistance functions, people who are not accustomed to operating a PC can only guess what operation was performed by a remote party at a remote location from information displayed on the screen such as mouse cursor movement. Is difficult for the user to guess, and the specific operation of the conversation partner cannot be understood. For this reason, there is also a problem that even if the user tries to do what he / she has been taught by the remote conversation again, he does not know the operation method. For example, the operation of copying a certain character string and pasting it in another place includes a method using only a mouse, a method using both a keyboard and a mouse, and a method using only a keyboard. An operation example when using both a keyboard and a mouse is described below. First, the user moves the mouse cursor over the character string to be copied, and selects the character string by pressing and moving the mouse button over the character string to be copied. Next, the character string is copied by pressing the <C> key while pressing the <Ctrl> key on the keyboard. Finally, move the mouse cursor to the place where you want to paste the character string, press the left mouse button, and press the <V> key while pressing the <Ctrl> key again on the keyboard to paste the copied character string. , The operation is completed. The key operation performed while pressing the <Ctrl> key on the keyboard is a shortcut function of an operation performed by selecting a menu of the application program. The operation frequently performed by the user is replaced with <Ctrl> + <C> or a function.・ Easy operation by assigning to keys. In the above-described operation example in which the keyboard and the mouse are used in combination, only the movement of the mouse cursor can be seen by a person in a remote place, and operation information such as pressing a mouse button or inputting from the keyboard cannot be viewed. .
[0007]
In the method in which the explainer explicitly adds an explanation, it is necessary for the explainer to add an explanation in advance, and unless the explainer adds an explanation, what operation has been performed is transmitted to the explained side. There is no problem.
[0008]
An object of the present invention is to provide a system, a method, and a program that specifically describe an operation that cannot be understood only by looking at a screen such as a mouse or keyboard input using characters or animation.
[0009]
[Means for Solving the Problems]
In order to solve the above problems, a means for monitoring the occurrence of an event from a window system to an application, a means for associating an event with a specific user operation, and an animation and an icon for a user operation corresponding to the generated event By providing means for displaying on the screen using characters and the like, it is possible to display on the screen specific operations such as mouse and keyboard input performed by the user. For the following description, these means will be referred to as basic means.
[0010]
In addition to the basic means described above, a function of recording operation information in a storage area is provided, and an operation manual can be automatically created by displaying or printing on a screen when necessary. Become.
[0011]
In addition to the basic means described above, by displaying only operation information for a specific application on the screen by selecting only an event for the specific application, and displaying information focusing on an operation for a certain application by providing a recording means Will be able to
[0012]
When an application operating on one computer is shared by a plurality of computers, the above-described basic means is provided on the computer executing the application, and the screen information indicating the user operation is displayed on another computer. By providing means for sending to the other computer, the same user operation display screen can be displayed on a plurality of computers. In addition, by providing a function for distinguishing and managing events for each user (computer) and providing means for distinguishing and displaying the user who performed the operation using colors, animations, icons, characters, etc., Information on whether such an operation has been performed can be obtained from the screen.
[0013]
Similarly, when the same application is operated on each computer and data such as graphics and texts are shared, the above-described basic means are provided on each computer, and events are separately managed for each user (computer). By providing a function to perform the operation and providing a means for displaying the user who performed the operation using colors, animations, icons, characters, and the like, information on which user performed what operation can be obtained from the screen. Become like
[0014]
Further, by providing the above-described basic means and the function of recording an event on a computer executing a shared application, it is possible to reproduce a user operation, and the recorded event is downloaded and used by another computer. You can also do things. Even in a macro function for registering an operation performed by a user and automatically executing the operation, it is possible to display user operation information at the time of automatic execution.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, a first embodiment of the present invention will be described with reference to FIG. FIG. 1 shows the configuration of a computer in which the window system is incorporated in the OS. However, since the hardware configurations such as the CPU, main storage, and secondary storage are not related to the present invention, the description is omitted. I have. Further, although the mouse 102 and the keyboard 101 are used as input devices, other input devices may be connected. A device driver 110 for processing signals from devices such as a mouse 102, a keyboard 101, and a display 100; an OS kernel 120 for performing hardware interrupts and software scheduling generated from the devices; and a window system 130 for managing windows On the basis of which the application software group 150 operates. For example, when the mouse 102 is clicked, a hardware interrupt occurs. When the OS kernel 120 and the mouse device driver 110 process the signal and transmit a click signal to the window system 130, the window system 130 sends a click signal to the corresponding application. Report the occurrence of an event. The input operation display program 140 of the present invention picks up an event notification from the window system 130 to the application software group 150, and the event handler 141 analyzes the contents of the event with reference to the event conversion table 200 and interprets the interpreted contents. An input event display 142 is displayed on the screen. Since a specific input operation is displayed on the screen, even a person who is not looking at the operation of the apparatus can know what operation has been performed.
[0016]
FIG. 2 shows a configuration example of the event conversion table 200. The basic configuration consists of an event 210 and display information corresponding thereto, for example, display information 220 in text and display information 230 in an image. Argument. Although the type of the event 210 and the attribute information associated therewith are different depending on the specifications of the window system 130, the basic mechanism for handling the event is the same. For example, Click (Obj, EventArgs) is an event that occurs when a mouse is clicked. Obj is the name or ID or pointer of an object such as a clicked button or menu, and EventArgs is an attribute 240 Since it has a value indicating whether the button was pressed, the X coordinate / Y coordinate at which the button was clicked, etc., from the Click (Obj, EventArgs), which object the mouse clicked, and the mouse button at the time of the click, Information on state and coordinates can be obtained. The event conversion table 200 is a table for converting the event 210 into a user operation, and prepares a user operation corresponding to each event. Here, display information (sentence) 220 in which the user operation is converted into a sentence and display information (image) 230 in which the user operation is converted into an image are prepared. When a click event occurs, the Button attribute of MEvenArgs is referenced from Click (Obj, MEventArgs). If the value of the Button attribute is a right button, a sentence “Click the right mouse button” and a ClickR. gif image. Although the case where the value of the Button attribute is the left button is shown in parentheses in FIG. 2, similarly to the case of the right button, the sentence “Click the left mouse button” and the ClickL. gif image.
[0017]
FIG. 3 shows a screen display example when the right button is clicked. In the example of the animation display shown in FIG. Using the gif image, an animation of a right mouse click being pressed is displayed, and a sentence "Click the right mouse button" is displayed. The animation may be created in advance using the creation tool, or an image of a mouse with no buttons pressed and an image of the right button pressed alternately displayed, etc. Animations may be created. As for the sentence to be displayed, the sentence of the display information (sentence) 220 may be displayed as it is, or a result obtained by performing a knowledge process or the like may be displayed. The character may be displayed, and the content of the operation may be described by the character as a character, or may be described by a synthesized voice. The sound corresponding to the operation may be reproduced by associating the operation with the sound. FIG. 3B shows an example in which display is performed using only characters. FIG. 3C shows an example in which a list of mouse input operations is displayed in a list, and when an operation corresponding to the contents of the list is performed, a position in the corresponding list is marked. In the list-type display, a technique for highlighting, such as putting a focus instead of a mark or performing reverse display, may be used.
[0018]
For the keyboard input event, similarly to the case of the mouse, the <Alt><Shift><Ctrl> key is pressed or an attribute value indicating whether the key was pressed immediately before, and which key is pressed. Is obtained as the argument EventEvents 250 of the event. In the case of a keyboard, similar to a mouse, a specific key operation can be displayed on the screen by displaying a text or an animation that indicates which key is being pressed. The same applies to the case where a mouse and a keyboard are used in combination and the case where another input device is used. FIG. 3D shows an example in which an operation using both a mouse and a keyboard is displayed in characters. Although the moving direction of the mouse cursor is displayed, the moving direction can be calculated from the difference between the occurrence time of successive events and the coordinates.
[0019]
As shown in FIG. 2, the event 210 is converted into information 220 and 230 for visually displaying, and the converted information is displayed as shown in FIG. Information can be displayed on the screen.
[0020]
FIG. 4 is a diagram showing a processing flow of the input operation display program 140. When the program starts (step 410), the event handler 141 is activated and waits for the occurrence of an event from the window system 130. When the event occurs, the event is received (step 420), the event is analyzed, and the operations performed simultaneously. A determination is made as to whether there is a correlation between a plurality of events such as whether or not the operation is to be displayed (step 430). Next, it is determined whether or not the event is an end instruction to the input operation display program 140 (step 460). If the event is an end instruction, the program ends (step 470). If the event is not an end instruction, the analysis result of the event is passed to the input event display 142, an operation explanation / animation is created with reference to the event conversion table 200 (step 440), and displayed on the screen (step 450). ), And returns to the process of waiting for event reception again (step 420).
[0021]
FIG. 5 shows a configuration when the present invention is applied to remote conference software. Remote conference software 510 and 520 and input operation programs 140 and 540 are operated on the host computer and the participating computer at remote locations. 510 and 520, and 140 and 540 are assigned different numbers to distinguish the operating computers, but they are the same software, which can transmit and receive audio and video in real time, and operate shared applications. Can be done. The computer on the participant side also has a television function, and can perform an input operation using a TV remote control. Therefore, input to the remote conference software 520 is performed from the keyboard 531, the mouse 532, and the TV remote controller 536. When an input event occurs in the remote conference software 520, the event handler 526 of the remote conference software 520 receives the event, and the event determined to be an event to the shared application 514 is hosted by one of the communication functions TCP / IP 537. Is transmitted to the TCP / IP 104 on the side, and is transmitted to the shared application 514 of the remote conference software 510 to be processed. The window displayed by the shared application is sent from TCP / IP 104 to 537, and is displayed on display 530 via window system 560, OS kernel 570, and device driver 580. An input event from the keyboard 531, the mouse 532, and the TV remote controller 536 is transmitted to the input operation display program 540, and an input event from the keyboard 101, the mouse 102, the keyboard 531, the mouse 532, and the TV remote controller 536 are transmitted to the input operation display program 140. An input event is sent. In order to display the operation information of the host on the display 530 of the participant, the window displaying the operation information of the input operation display program 140 displayed on the display 100 is shared, and the same method as that of the remote software is used. What is necessary is just to display on the display 530. The input operation recording / reproducing program 500 may include a step of recording the event occurrence time and the event as a set, and a step of generating the recorded event in a pseudo manner while maintaining an interval between the event occurrence times. Thus, recording / reproduction of an event becomes possible. A set of event occurrence times and events recorded by providing a web server function in the host computer is downloaded to the participating computer, and the input operation recording / playback program 550 is operated to play back the downloaded event. To display operation information.
[0022]
When recording the content of the remote conference, it is expected that video and audio will be recorded in each computer, or that the window of the shared application will be recorded together with the video and audio. Since video recording has a large capacity, it is anticipated that there will be many uses in which only a portion considered important is recorded rather than recording the entire conference. Recording the event occurrence time and the event is small in capacity, so recording the entire conference is realistic. When recording the video, the time information of the video is also recorded, and the event occurrence time information and the time information of the video are associated with each other, whereby the event can be reproduced simultaneously with the reproduction of the video. By always recording the event occurrence time and event set at the time of the meeting and making it possible to refer to it via the network, when the recorded conversation is played back, it will be called the "correction part of the previous material" It is possible to construct a system that can refer to the operation information even when the operation is referred to retroactively.
[0023]
【The invention's effect】
According to the present invention, operation information such as which mouse button is pressed can be obtained from the screen, so that the user can know a specific operation method. In addition, since the operation information can be displayed as needed by storing the operation information, there is an effect that labor and cost for preparing an operation manual can be saved.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of one embodiment of the present invention.
FIG. 2 is an example of an event conversion table of FIG. 1;
FIG. 3 is a screen display example of operation contents by an input operation display program when a user presses a right button of a mouse.
FIG. 4 is a flow of processing of an input operation display program.
FIG. 5 is a configuration diagram when using remote conference software.
[Explanation of symbols]
140: input operation display program, 141: event handler, 142: input event display, 200: event conversion table.

Claims (11)

An operation display method of a computer system for transmitting a signal from an input device to an application as an event,
Monitors event transmission to applications, interprets events, converts events, and displays event conversion tables that associate events with user operations, and displays the results of event conversion on the screen using animation, icons, characters, and characters An operation display method, characterized in that: 2. The operation display method according to claim 1, wherein a result of the event conversion is notified by voice. 2. The operation display method according to claim 1, wherein a result of the event conversion is stored in a storage area, and is displayed and printed in response to a request from a user. 2. The operation display method according to claim 1, wherein the event is stored in a storage area, the event is interpreted in response to a request from a user, converted into an event, and displayed on a screen. 2. The operation display method according to claim 1, wherein only an event for a specific application is interpreted in response to a request from a user, the event is converted, and the converted event is displayed on a screen. In an operation display method of a computer system in which a plurality of computers that transmit a signal from an input device as an event to an application are connected via a communication function, the plurality of computers transmit an event to each other and share a screen display, at least One computer monitors the event transmission to the application, interprets the event, converts the event, and converts the event and the user's operation into an event conversion table, and converts the event conversion result into animation, icons, characters, and characters. An operation display method characterized in that the operation display method is used to display on a screen. 7. The operation display method according to claim 6, wherein events are identified and managed for each computer, and when the result of the event conversion is displayed on a screen, each event is displayed using a color, animation, icon, character, voice, or sound. An operation display method characterized in that the operation display method is distinguished and displayed. 7. The operation display method according to claim 6, wherein the event or the operation information after the event conversion is recorded by another computer, and the event or the operation information after the event conversion is downloaded and used in accordance with the reproduction of the screen information. An operation display method, characterized in that: When the processing device of the computer executes a predetermined process in response to an instruction from the user,
A function of retrieving a process identifier for specifying the predetermined process from a storage device of the computer,
A function of searching the storage device for characters or / and / or images or / and animations indicating the content of an operation performed by the user on the input device of the computer in order to give an instruction from the user;
A program for causing the computer to realize the function of storing the character or / and / or the image / and / or the animation and the processing identifier in the storage device in association with each other. A function of reading the character or / and / or the image or / and the animation and the process identifier from the storage device;
The program according to claim 9, wherein the program causes the computer to realize a function of associating the character or / and / or the image or / and the animation with the processing identifier and transmitting it to another computer of another user. The character or / and / or image and / or animation transmitted to the other computer is displayed on a display screen of the other computer when the predetermined process is started on the other computer. 10. The program according to 10.

Images