JP4003651B2 - Operation writing method, program, and recording medium recording the program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an operation writing method, a program, and a recording medium on which the program is recorded. Specifically, the present invention relates to a process for creating a user's operation process as a sentence.
[0002]
[Prior art and problems to be solved by the invention]
In recent years, many studies have been made to collect and analyze operation histories and use them for information recommendation or marketing based on the obtained trends. These analysis results vary greatly depending on the nature of the operation history that is the input. Therefore, if the operation history as detailed as possible can be grasped, there is a possibility that an analysis result that matches the true needs of the user is obtained.
[0003]
In order to acquire a detailed operation history of software, it is conceivable to incorporate a function for acquiring an operation history into each software. However, with this, it is difficult to acquire an operation history of arbitrary software, and there is no versatility.
[0004]
There has been a method in which a user utters an operation content by voice and dictates the utterance content by a voice recognition technique in order to acquire an operation history for a general purpose (see Non-Patent Document 1, for example). However, a decrease in the recognition rate of speech recognition due to the speed of utterance or rounding is a problem that cannot be avoided. Another problem is that the user's load is high because the user can speak the operation content.
[0005]
[Non-Patent Document 1]
Kawabata Go, Nakamata Nobuya et al., “Voice Recognition / Synthesis”, NTT Technical Journal, 1999, Vol. 11, No. 12, pp12-40
[0006]
Therefore, the present invention has been made in view of the above problems, and is a general-purpose operation writing method and program for acquiring detailed operation history without burdening the user, and the program recorded therein. An object is to provide a recording medium.
[0007]
[Means for Solving the Problems]
The present invention provides a user interface device comprising an operation means operated by a user and a display means for displaying image information including an operation target, and the operation of a processing device for creating a user's operation process as a sentence. The present invention relates to a writing method, a program, and a recording medium on which the program is recorded.
[0008]
According to the operation writing method of the present invention,
A first step of monitoring an operation event in the operation means;
A second step of specifying the operation target and the operation type from the image information displayed on the display means when the operation event occurs;
And text indicating the operation target, a third step of creating a sentence of a combination of a text indicating the type of operation, and have a,
To identify the operation target,
Obtaining surrounding image information, which is image information around coordinates on the screen indicated by the operating means when an operation event occurs;
Comparing the acquired surrounding image information with the partial image information for each operation object stored in advance and searching for partial image information having the highest degree of coincidence or similarity;
Recognizing a character string in the searched partial image information or searching the document information related to the searched partial image information based on the document information related to the stored partial image information. To do. As a result, a detailed operation history can be acquired by a general-purpose method that does not place a burden on the user.
[0011]
Furthermore, according to another embodiment of the operation writing method of the present invention, when a hierarchical structure exists in the operation target,
In the second step, an upper operation target for the operation target is specified,
In the third step, it is also preferable that a sentence further combining sentences indicating the upper operation target is created. This makes it possible to describe an operation history that makes it easy to specify the position of the operation target.
[0012]
Further, according to another embodiment of the operation writing method of the present invention, the third step further combines a sentence indicating a relative position of the operation target partial image information with respect to the upper operation target partial image information. It is also preferable to create a sentence.
[0013]
According to the operation writing program of the present invention,
A first step of monitoring an operation event in the operation means;
A second step of specifying the operation target and the operation type from the image information displayed on the display means when the operation event occurs;
And text indicating the operation target, to execute a third step of creating a sentence of a combination of a text indicating the type of operation to the computer,
To identify the operation target,
Obtaining surrounding image information, which is image information around coordinates on the screen indicated by the operating means when an operation event occurs;
Comparing the acquired surrounding image information with the partial image information for each operation object stored in advance and searching for partial image information having the highest degree of coincidence or similarity;
Recognizing a character string in the searched partial image information, or causing the computer to execute a step of searching for document information related to the searched partial image information based on the document information related to the stored partial image information. And
[0016]
Furthermore, according to another embodiment of the operation writing program of the present invention, when the operation target has a hierarchical structure,
In the second step, an upper operation target for the operation target is specified,
In the third step, it is also preferable that the computer is executed so as to create a sentence further combining sentences indicating the upper operation target.
[0017]
Furthermore, according to another embodiment of the operation writing program of the present invention, the third step further combines a sentence indicating the relative position of the operation target partial image information with respect to the upper operation target partial image information. It is also preferable to run the computer to create a sentence.
[0018]
It may be a recording medium on which the above-described operation writing program according to the present invention is recorded.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0020]
FIG. 1 is a system configuration diagram of a first embodiment according to the present invention . Also, FIGS. 2 and 3 are examples of the display information displayed on the display.
[0021]
According to FIG. 1, a processing device connected to a mouse, a keyboard, and a display includes an operation monitoring unit 1, an operation target specifying unit 2, an operation type specifying unit 3, and an explanation sentence synthesizing unit 4. .
[0022]
The operation monitoring unit 1 monitors user operations on the operation target. Examples of the operation target include “menu”, “button”, “window”, “radio button”, and the like. User operations include mouse click, keyboard input, touch panel touch, voice input, and the like. When detecting the occurrence of such an operation event, the operation monitoring unit 1 outputs an operation target specifying request to the operation target specifying unit 2 together with the operation identification information, and sends an operation type specifying request to the operation type specifying unit 3. Output.
[0023]
When the operation target specifying unit 2 receives the notification of the operation target specifying request from the operation monitoring unit 1, the operation target specifying unit 2 specifies the operation target operated by the user based on the operation identification information. Here, the operation target refers to a part of the screen information constituting the application, on which the user's operation is directly applied.
[0024]
For example, according to FIG. 2, when the user clicks the “OK button” on the screen of the application, the “OK button” to which the user's operation directly acts becomes the operation target. For example, according to FIG. 3, when the user clicks the “edit menu” button in the application screen, the “edit menu” button to which the user's operation directly acts becomes the operation target.
[0025]
The operation target may be specified using the operation identification information acquired by the operation monitoring unit 1 and the operation target correspondence table. In addition, the operation identification information that can be acquired from the operation monitoring unit 1 may be used to automatically specify the operation target by some method.
[0026]
When receiving the notification of the operation target specification request from the operation monitoring unit 1, the operation type specifying unit 3 specifies the operation type performed by the user on the operation target based on the operation identification information. For example, for the operation object “icon”, a plurality of types such as an operation of “clicking” or an operation of “dragging” can be used depending on the operation of the user. Operation type.
[0027]
Then, identifying which of the plurality of operations is the operation performed by the user is an act of identifying the operation type. Depending on the operation target, there may be only one operation type. In such an operation target, the operation type is inevitably determined uniquely.
[0028]
The explanatory note synthesizing unit 4 combines the text indicating the operation target acquired by the operation target specifying unit 2 and the text indicating the operation type acquired by the operation type specifying unit 3 into a text of the operation history. There is a method of preparing a sentence template and embedding the acquired operation target and operation type in writing a sentence. Specific examples are shown below.
[0029]
Template = “[operation target] is [operation type]”,
If the operation target = "Edit menu" and the operation type = "Select",
Created sentence = “Select” “Edit menu”. "
[0030]
Here, an image that indicates the operation target and the operation type may be extracted and added to the sentence.
[0031]
FIG. 4 is a flowchart of the writing method based on FIG.
[0032]
(S1) The operation monitoring unit 1 is monitoring an operation event. If an operation event occurs, the process proceeds to S2. If not, the process returns to S1 itself and continues to monitor the operation event.
(S2) The operation type identification unit 2 identifies the operation type based on the operation identification information acquired from the operation monitoring unit 1, and proceeds to S3.
(S3) The operation target specifying unit 3 identifies the operation target based on the operation identification information acquired from the operation monitoring unit 1, and proceeds to S4.
(S4) The explanatory note synthesizing unit 4 combines the text indicating the operation target acquired by the operation target specifying unit 2 and the text indicating the operation type acquired by the operation type specifying unit 3 into a text of the operation history. Thereafter, the process returns to Sl again, and monitoring of the next operation event is continued.
[0033]
FIG. 5 is a system configuration diagram of the second embodiment according to the present invention .
[0034]
According to FIG. 5, the processing device includes an operation monitoring unit 1 ′, an operation target specifying unit 2 ′, an operation type specifying unit 3, an explanation sentence synthesizing unit 4, and an image recognition unit 5 ′. Note that the operation type identification unit 3 and the explanation sentence synthesis unit 4 are the same as those shown in FIG.
[0035]
When an operation event occurs, the operation monitoring unit 1 ′ acquires the coordinates on the screen of a pointer that indicates an operation such as a mouse on a personal computer, and uses the coordinates as operation identification information to specify the operation target specifying unit 2 ′ and the operation type. Notify the specific unit 3.
[0036]
The operation target specifying unit 2 ′ includes a surrounding image acquisition unit 2-1, a target type identification unit 2-2, and a title recognition unit 2-3.
[0037]
The surrounding image acquisition unit 2-1 acquires the surrounding image information around the coordinates, that is, the surrounding image information of the operation target based on the coordinates of the operation identification information notified from the operation monitoring unit 1 ′, and the target type identification unit Notify 2-2.
[0038]
The target type identification unit 2-2 notifies the shape recognition unit 5-1 of the surrounding image information notified from the surrounding image acquisition unit 2-1, and whether there is a candidate for the operation target in the surrounding image information. Ask about it. If it exists, the target is specified as the operation target. The image information in the specified operation target range is notified to the title recognition unit 2-3.
[0039]
The title recognition unit 2-3 notifies the character recognition unit 5-2 of the image information notified from the target type identification unit 2-2, and inquires whether a character string candidate exists in the image information. If it exists, the character string is specified as the title of the operation target.
[0040]
The image recognition unit 5 includes a shape recognition unit 5-1 and a character recognition unit 5-2.
[0041]
The shape recognition unit 5-1 stores partial image information for each operation target in advance. Then, the surrounding image information notified from the target type identification unit 2-2 is compared with the accumulated partial image information as candidates, and the partial image information that matches or has the highest similarity is searched. The retrieved partial image information is notified to the target type identification unit 2-2. On the other hand, if it is not similar to any of the partial image information, this is returned.
[0042]
The character recognizing unit 5-2 detects whether a character string candidate exists in the partial image information notified from the title recognizing unit 2-3. Alternatively, text information related to the partial image information may be stored in advance, and text information that matches the partial image information may be searched. The detected or searched text information is notified to the title recognition unit 2-3. On the other hand, if it does not match any text information, a message to that effect is returned.
[0043]
FIG. 6 is a flowchart of the writing method based on FIG.
[0044]
(S2-1) The operation monitoring unit 1 ′ monitors an operation event. When the operation event occurs, the operation monitoring unit 1 ′ acquires the coordinates on the screen of a pointer that indicates an operation such as a mouse on a personal computer, and the coordinates together with the operation identification information. Information is notified to the operation type identification unit 3. When the operation type specifying unit 3 determines that the operation type is a mouse click, the operation target specifying unit 2 ′ is notified of the operation target specifying request together with the mouse coordinates, and the process proceeds to S2-2. In the case of other operation types, the process returns to S2-1 and continues to monitor the operation.
(S2-2) The surrounding image acquisition unit 2-1 of the operation target specifying unit 2 ′ acquires the surrounding image information of the acquired mouse coordinates. As the surrounding image information, a circular image centered on mouse coordinates or a rectangular image having a predetermined size is assumed. Then, it progresses to S2-3.
(S2-3) The surrounding image acquisition unit 2-11 The target type identification unit 2-2 notifies the acquired image to the shape recognition unit 5-1, and there are operation target candidates in the surrounding image information. Inquire whether or not. If it is determined that there is a portion having a high similarity to the shape of the “menu” or “button” in the surrounding image information, the process proceeds to S2-4. If it is determined that there is no specific operation target in the surrounding image information, the process returns to S2-1 to continue monitoring the operation.
(S2-4) The title recognition unit 2-3 notifies the character recognition unit 5-2 of the partial image information notified from the target type identification unit 2-2, and character string candidates are included in the partial image information. Ask if there exists. If it exists, the character string is recognized as a title to be operated, and the process proceeds to S2-5.
(S2-5) A part of the screen including the specified operation target is captured, and the process proceeds to S2-6.
(S2-6) The explanatory note synthesizing unit 4 combines the operation target title acquired by the operation target specifying unit 2 ′ and the operation type acquired by the operation type specifying unit 3 to synthesize an operation description. Note that the image captured in S2-5 may be added to the text.
[0045]
FIG. 7 is a system configuration diagram of the third embodiment according to the present invention . Also, FIG. 8 shows an example of display information displayed on the display. In addition, since it is the same as 2nd Embodiment except the upper hierarchy specific | specification part 6, description is abbreviate | omitted below.
[0046]
The upper layer specifying unit 6 includes an upper layer target specifying unit 6-1 and a title recognizing unit 6-2.
[0047]
The upper layer target specifying unit 6-1 specifies an operation target of the parent hierarchy of the operation target when the operation target specified by the operation target specifying unit 2 ′ has a hierarchical structure. When the operation target is “menu”, the parent menu hierarchy is followed, and an image corresponding to the parent menu name is acquired. When the operation target is “button”, an image including the title of “window” in which “button” is drawn is specified as the operation target of the parent hierarchy. The number of hierarchies to be traced is not limited to one, and the parent hierarchy may be further recursively retraced as necessary.
[0048]
The title recognizing unit 6-2 notifies the character recognizing unit 5-2 of the image acquired by the upper layer target specifying unit 6-1 and inquires whether a character string candidate exists in the image. If it exists, the character string is recognized as an operation target title.
[0049]
The explanatory note synthesizing unit 4 combines the operation target acquired by the operation target specifying unit 2, the operation type acquired by the operation type specifying unit 3, and the parent operation target acquired by the higher hierarchy specifying unit 6, Is documented.
[0050]
When the operation target is a menu, the position where the operation target menu exists in the menu hierarchy is described. For example, the parent menu hierarchy is also written as “Select [File]-[Save]”.
[0051]
When the operation target is a button, the relative position is described based on the relative coordinates arranged in the button in the parent operation target window. For example, the relative position is written together, such as “Press the OK button at the lower right of the print window”.
[0052]
Furthermore, an image that indicates the operation target and the operation type may be extracted and added to the sentence. There is a method of preparing a sentence template and embedding the acquired operation target and operation type in writing a sentence.
[0053]
FIG. 9 is a flowchart of the operation writing method based on FIG. In addition, since it is the same as that of 2nd Embodiment except S3-1, S3-2, S2-6 ', description is abbreviate | omitted below.
[0054]
(S3-1) When the hierarchical structure exists in the operation target specified by the operation target specifying unit 2 ′, the upper layer target specifying unit 6-1 specifies the operation target of the parent hierarchy of the operation target. Further, the title recognition unit 6-2 notifies the character recognition unit 5-2 of the image acquired by the upper layer target specifying unit 6-1 and determines whether or not a character string candidate exists in the image. If there is an inquiry, the character string is recognized as the title of the operation target. Then, it progresses to S3-2.
(S3-2) The target specifying unit 6-1 in the upper hierarchy determines whether there is a higher-level parent operation target, and if it exists, the process returns to S3-1. If not, the process proceeds to S2-5.
(S2-6 ′) The description synthesizing unit 4 acquires the operation target title acquired by the operation target specifying unit 2 ′, the operation type acquired by the operation type specifying unit 3, and the parent acquired by the upper hierarchy specifying unit 6. The operation history is documented in combination with the operation target.
[0055]
【The invention's effect】
As described above, according to the operation writing method, the program, and the recording medium on which the program is recorded according to the present invention, it is general-purpose and a detailed operation history can be acquired without imposing a burden on the user. In particular, it is possible to describe an operation history that makes it easy to specify the content of an operation target by a general-purpose method that does not depend on a platform such as an application or an OS. It is also possible to describe an operation history that makes it easy to specify the position of the operation target.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of a first embodiment according to the present invention.
FIG. 2 is an example of display information displayed on a display.
FIG. 3 is an example of display information displayed on a display.
FIG. 4 is a flowchart of a writing method based on FIG. 1;
FIG. 5 is a system configuration diagram of a second embodiment according to the present invention.
FIG. 6 is a flowchart of a writing method based on FIG.
FIG. 7 is a system configuration diagram of a third embodiment according to the present invention.
FIG. 8 is an example of display information displayed on a display.
FIG. 9 is a flowchart of a writing method based on FIG.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Operation monitoring part 2 Operation target specific | specification part 2-1 Surrounding image acquisition part 2-2 Target classification identification part 2-3 Title recognition part 3 Operation classification specific part 4 Description sentence composition part 5 Image recognition part 5-1 Shape recognition part 5 -2 Character recognition unit 6 Upper layer specifying unit 6-1 Upper layer target specifying unit 6-2 Title recognition unit

Claims (7)

An operation writing method in a processing device for creating a user's operation process as a sentence using a user interface device comprising an operation means operated by a user and a display means for displaying image information including an operation target. There,
A first step of monitoring an operation event in the operation means;
A second step of specifying the operation target and the operation type from the image information displayed on the display means when the operation event occurs;
A third step of creating a sentence combining the sentence indicating the operation object and the sentence indicating the operation type ;
And have a,
To identify the operation target,
Obtaining surrounding image information, which is image information around coordinates on the screen indicated by the operating means when an operation event occurs;
Comparing the acquired surrounding image information with the partial image information for each operation object stored in advance, and searching for partial image information having the highest match or similarity;
Recognizing the character string in the searched partial image information, or searching the document information related to the searched partial image information based on the document information related to the stored partial image information;
A method for writing an operation characterized by including: When a hierarchical structure exists in the operation target,
The second step specifies an upper operation target for the operation target,
The writing method according to claim 1, wherein the third step creates a sentence further combining sentences indicating the upper operation target. The third step, according to claim 2, characterized in that to create a further combined sentence text indicating the relative position of the operation target of the partial image information for the upper operation target partial image information Writing method. An operation writing program for creating a user's operation process as a sentence using a user interface device including an operation means operated by a user and a display means for displaying image information including an operation target,
A first step of monitoring an operation event in the operation means;
A second step of specifying the operation target and the operation type from the image information displayed on the display means when the operation event occurs;
A third step of creating a sentence combining the sentence indicating the operation object and the sentence indicating the operation type ;
Cause the computer to execute,
To identify the operation target,
Obtaining surrounding image information, which is image information around coordinates on the screen indicated by the operating means when an operation event occurs;
Comparing the acquired surrounding image information with the partial image information for each operation object stored in advance and searching for partial image information having the highest degree of coincidence or similarity;
Recognizing the character string in the searched partial image information, or searching the document information related to the searched partial image information based on the document information related to the stored partial image information;
An operation writing program characterized by causing a computer to execute . When a hierarchical structure exists in the operation target,
The second step specifies an upper operation target for the operation target,
5. The operation writing program according to claim 4 , wherein the third step causes the computer to execute so as to create a sentence further combining sentences indicating the upper operation target. In the third step, the computer is executed so as to create a sentence in which a sentence indicating a relative position of the partial image information of the operation target with respect to the partial image information of the upper operation target is further combined. The operation writing program according to claim 5 . The recording medium which recorded the program of any one of Claim 4 to 6 .

Images