JP2012064051A - Help display device, help display method and help display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a help display device capable of presenting a position to be operated on an application screen corresponding to a content displayed on a help screen to a user, without preliminarily storing positional relation between operational object displayed on the help screen and operational object on the application screen in a memory unit.SOLUTION: A display part of the help display device displays the application screen and the help screen related to the application screen. A character recognition processing part performs character recognition processing for a screen image of the display part. A position search part compares an item on the help screen with the processing result of the character recognition processing and searches at which position on the application screen the item on the help screen exists. A highlighting part highlights an item on the application screen corresponding to the item on the help screen based on the search result of the position search part.

Description

Embodiments described herein relate generally to a help display device, a help display method, and a help display program.

Conventionally, a help screen for various applications describes a procedure of an operation method related to the application. In order to operate the application by referring to the help screen, it is necessary to find out where the operation target (hereinafter referred to as control) exists on the application.

On the other hand, there is a technique for performing guidance to control on an application using a database in which all guidance contents are stored in advance. There is also a technique for displaying a keyboard image on the help screen and presenting the position of the control on the help screen.

JP 2002-73232 A

Operations on the application screen corresponding to the contents displayed on the help screen without saving in advance to the storage device the position of the control displayed on the help screen on the actual application screen To provide a help display device, a help display method, and a help display program capable of presenting a target position to a user.

The help display device of the embodiment includes a display unit that displays an application screen, a help screen related to the application screen, a character recognition processing unit that performs character recognition processing on a screen image of the display unit, and items on the help screen. Based on the result of the search by the position search unit that searches the position on the application screen where the item on the help screen exists by collating with the processing result of the character recognition process A highlighting unit that highlights an item on the application screen corresponding to an item on the help screen.

The block diagram which shows an example of the whole structure of the help display system which concerns on this embodiment. The flowchart which shows an example of the process of the help display system which concerns on this embodiment. The figure which shows an example of the screen displayed on the display part of the help display system which concerns on this embodiment. The figure which shows an example of the screen displayed on the display part of the help display system which concerns on this embodiment. The figure which shows an example of the search target list storage data table of the help display system which concerns on this embodiment. The figure which shows an example of the screen image produced in the help display system which concerns on this embodiment. The figure which shows an example of the data table for character recognition process result storage of the help display system which concerns on this embodiment. The figure which shows an example of the area | region excluded from the character recognition process in the screen image of the help display system which concerns on this embodiment. The figure which shows an example of the item name, the upper left coordinate, and the lower right coordinate which the operation target position search part of the help display system which concerns on this embodiment extracts. The figure which shows an example of the screen displayed on the display part of the help display system which concerns on this embodiment. The figure which shows an example of the screen displayed on the display part of the help display system which concerns on this embodiment.

(First embodiment)
A help display system according to a first embodiment will be described with reference to the drawings. Hereinafter, “help” refers to online help, and refers to a help document in a format that can be browsed on a computer for reference information such as an operating method of the OS and software. The help screen is a screen for displaying the above-described help document.

FIG. 1 is a block diagram showing an example of the overall configuration of the help display system 100 of the present embodiment.

As shown in FIG. 1, the help display system 100 of this embodiment includes an input unit 10 and a computer 2.
0 and a display unit 30, and an input unit 10 and a display unit 30 are connected to the computer 20.

The user selects a character string on the help screen displayed on the display unit 30 using the input unit 10. The selected character string information is transmitted from the input unit 10 to the computer 20. The input unit 10 is, for example, a mouse or a keyboard. In the present embodiment, it is assumed that the input unit 10 is a mouse.

The display unit 30 is, for example, a liquid crystal display, and displays an application screen that is a kind of user interface related to an arbitrary application used by the user and a help screen for the application.

Controls are displayed on the application screen, and the user realizes a desired operation by operating these controls. That is, the control refers to an object that is displayed on a display, such as a button, an icon, a link, a tab, a check box, or a combo box, and is operated by a user to realize a desired operation using a computer.

The computer 20 includes a selection range analysis unit 21, a screen image creation unit 22, and a character recognition processing unit 2.
3, an operation target position search unit 24, a screen highlight unit 25, a search target list storage unit 26, and a character recognition process result storage unit 27.

The selection range analysis unit 21 is connected to the input unit 10 and the search target list storage unit 26. The selection range analysis unit 21 analyzes the character string input by the input unit 10 and extracts nouns. The selection range analysis unit 21 creates a search target list table 260 in the search target list storage unit 26 and stores the nouns of the analysis results in the search target list table 260.

The screen image creation unit 22 is connected to the display unit 30, the selection range analysis unit 21, and the character recognition processing unit 23. The screen image creation unit 22 creates a screen image of the screen displayed on the display unit 30 and creates the character recognition processing unit 23. Send to.

The character recognition processing unit 23 is connected to the screen image creation unit 22 and the character recognition processing result storage unit 27. The character recognition processing unit 23 performs character recognition processing on the screen image created by the screen image creation unit 22 and extracts nouns. In addition, the character recognition processing unit 23 creates a character recognition process result storage table 270 in the character recognition process result storage unit 27 and stores the extraction result in the character recognition process result storage table 270.

The operation target position search unit 24 is connected to a search target list storage unit 26 and a character recognition processing result storage unit 27. The operation target position search unit 24 is included in the character string on the help screen 32 selected by the input unit based on the data stored in the search target list storage unit 26 and the data stored in the character recognition processing result storage unit 27. The position of the control name representing the control on the application screen 31 is searched in the application screen 31.

The screen highlight unit 25 is connected to the operation target position search unit 24 and the display unit 30, and the display unit 3 is based on the position information of the control name searched by the operation target position search unit 24.
A specific area of 0 is highlighted by highlighting, for example.

Processing of the help display system 100 according to the present embodiment will be described with reference to FIGS.

FIG. 2 is a flowchart showing an example of processing of the help display system 100 of the present embodiment.

The processing of the help display system 100 of this embodiment starts from a state in which an arbitrary application screen 31 and a help screen 32 corresponding to this application are displayed on the display unit 30 in advance. An example of the display unit 30 on which an arbitrary application screen 31 and a help screen 32 corresponding to this application are displayed is shown in FIG. The application screen 31 of this embodiment is a screen that functions as a user interface for operating the window business application software, and the help screen 32 is a screen that displays a help document related to the window business application.

The user operates the mouse as the input unit 10 to select a character string on the help screen 32 displayed on the display unit 30 shown in FIG. Right-clicking the mouse with the character string selected displays a shortcut menu. By pressing “position search” in the shortcut menu by the user's operation of the input unit 10, a character string in the selected range is input (step S1). An example of the display unit 30 at this time is shown in FIG. The selected character string is a character string included in the area of the upper right diagonal line in FIG.

In order to input a character string, a search button is arranged on the help screen 32. When the search button is clicked by the input unit 10 in a state where the character string is selected, the selected character string is input. It is also possible. It is also possible to automatically input a character string on the help screen 32 when it is selected. That is, it is input when the help display device 100 of this embodiment recognizes that the character string displayed on the help screen 32 has been selected.

The selection range analysis unit 21 analyzes the character string input from the input unit 10 in step S1 and extracts nouns included in the character string (step S2). The extracted nouns on the help screen 32 are used for operation target position search processing described later.

The selection range analysis unit 21 creates a search target list table 260 in the search target list storage unit 26 and stores the extracted nouns in the search item name column 261 of the created search target list table 260. FIG. 5 shows an example of the search target list table 260 in the search target list storage unit 26.

When the character string is analyzed in step S 2 and the search item name field 261 is stored, the selection range analysis unit 21 transmits a screen image creation request to the screen image creation unit 22. The screen image creation unit 22 that has received the screen image creation request creates a screen image 300 of the screen displayed on the display unit 30 (step S3). An example of the screen image 300 created in step S3 is shown in FIG.

Subsequently, the character recognition processing unit 23 performs character recognition processing on the screen image 300 created by the screen image creation unit 22 (step S4). Specifically, the character recognition processing unit 23 performs character recognition processing on the created screen image 300 and extracts a noun from the processing result.

When the character recognition processing unit 23 extracts a noun, the character recognition processing result storage table 270 is created in the character recognition processing result storage unit 27, and character recognition processing is performed in the item name field 271 of the created character recognition processing result storage table 270. Result noun and screen image 300 of this noun
Together with the position information at. The position information in the screen image 300 of the character recognition processing result item of this embodiment is a rectangular area represented by upper left coordinates and lower right coordinates.

FIG. 7 shows an example of the character recognition processing result storage table 270. As shown in FIG. 7, the processing result storage table 270 is a column for storing item names 271 which are columns for storing nouns extracted by character recognition processing, and position information in the screen image 300 of character recognition processing result items. There are certain upper left coordinates 272 and lower right coordinates 273.

The character recognition processing unit 23 stores the extraction result extracted by the character recognition processing in the character recognition processing result storage table 270 and then transmits an operation target position search request to the operation target position search unit 24.

Note that a character string existing in an area whose background color is white in a screen image to be subjected to character recognition processing generally has a high possibility of being a text area or a content portion that can be edited by the user. Therefore, you may exclude from the object of the character recognition process performed by step S4. This makes it possible to prevent erroneous extraction of non-control parts and improve processing speed.

Here, a method for excluding an area whose background color is white when performing character recognition processing on the screen image 300 shown in FIG. 6 will be described.

When the character recognition processing unit 23 performs character recognition processing and searches for a character in the screen image 300, if the background color of the searched character is white, the character recognition processing unit 23 starts from an area adjacent to the searched character. All areas with a white background color are searched, and the searched areas are excluded from character recognition processing targets. For example, when a character with a white background color of “window” is searched from the screen image 300, an area with a white background color adjacent to “window” is searched and excluded from the target of character recognition processing.
It should be noted that whether or not exclusion can be performed and the background color to be excluded can be set in advance for the help display system 100 by the user. FIG. 8 is a diagram showing areas excluded from the character recognition process in the screen image 300 by hatching.

The operation target position search unit 24 that has received the operation target position search request receives the search target list storage unit 26.
And the character recognition processing result storage unit 27 are searched for operation target position search processing (step S5).
).

That is, the operation target position search unit 24 is created in step S2, and the search target list storage unit 2
6 and the item name 27 stored in the character recognition processing result storage unit 27.
1, and the search target list storage unit 2 determines whether there is a matching item name 271.
The search item names 261 stored in 6 are sequentially performed. As a result of the search in step 05, these search item names 26 are included in the item names 271 of the character recognition processing result storage table 270.
If there is no item name that matches 1 (No in step S6), the process ends.

When there is an item name 271 that matches the search item name 261 (step S6 is Yes),
The operation target position search unit 24 recognizes that the item name 271 is the name of the control for which the user wants to confirm the position (hereinafter referred to as the control name), and refers to the character recognition processing result storage unit 27 for the target item name 271. The upper left coordinate 272 and the lower right coordinate 273 are extracted, and the item name 271 is extracted.
And the upper left coordinates 272 and the lower right coordinates 273 are transmitted to the screen highlight unit 25 (step S7).
).
In other words, in the present embodiment, the control is recognized by using the feature “noun” of the control name on the help screen. In addition, it is also possible to extract the control name using, for example, a control name description method when the control is expressed on the help screen. In addition, as a description method of the control name, there are a method of delimiting with [] or “”, a method of describing a button after the control name such as “click XX button ”, and the like.
When there are a plurality of matching controls, the upper left coordinates 272 and the lower right coordinates 273 are acquired for all the matching controls. An example of the item name 271 extracted in the present embodiment, and the upper left coordinate 272 and the lower right coordinate 273 corresponding to the item name 271 are shown in FIG. As shown in FIG. 9, in the present embodiment, the matching item names are “savings online” and “customer management”, and the upper left coordinates and lower right coordinates corresponding to the item names are extracted correspondingly.

The screen highlight unit 25 highlights the control by highlighting the area in the display unit 30 obtained from the received upper left coordinates and lower right coordinates (step S8), and ends the process. As a highlighting method, for example, a transparent fluorescent color is displayed in an overlay on the area extracted in step S7, and a character and background are reversed and displayed.

FIG. 10 shows an example of the display unit 30 highlighted by the process of step S8. As shown in FIG. 10, in the present embodiment, the areas indicated by the upper left coordinates and the lower right coordinates obtained in step S8 are respectively displayed as hatched areas or rhombus lattice areas, and the extracted item names are surrounded by a frame. In this case, this frame is displayed as a thick frame, and the background in the thick frame is changed to white.

At this time, the item names on the help screen 32 are also highlighted in the same manner, so that the relationship between the control on the application screen 31 and the control name on the help screen 32 can be presented to the user. is there. That is, the control position corresponding to the content of the help can be presented to the user. Thereby, work efficiency can be improved.

Note that when there are a plurality of matching item names 271, which item name on the help screen 32 is changed to the application screen 31 by changing the color or the like for highlighting for each item name 271.
It is possible to easily determine which item name corresponds to.

Alternatively, as shown in FIG. 11, the control name may be displayed in a balloon around the area where the control name extracted in step 06 exists.

Note that after the highlight display is performed in step 08, the highlight display may be terminated when the user performs some input operation or when a time preset by the user has elapsed.

As described above, the help display system 100 according to the present embodiment does not save the positional relationship between the control name displayed on the help screen 32 and the control on the actual application screen 31 in the storage device in advance. It is possible to present where the control name described on the screen 32 is actually displayed on the application screen 31 used by the user, that is, the position of the control on the application screen 31.

Therefore, it can be used in various applications having a help function without building a dedicated database. For example, the help display system 100 according to the present embodiment can be applied to all applications having help information, such as a window business application such as a bank, an image and video editing application, and a medical application. Suitable for many types of applications that take time to learn.
Applications such as these have a large number of functions, so when building a help system that supports the correspondence between data such as help and screens, it takes more time to build than a simple application or web application that has only a few functions. there is a possibility. Therefore,
It is not necessary to construct a dedicated help system according to the present embodiment, and the cost for the construction can be reduced.
Although the embodiment of the present invention has been described as described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof. For example,
You may delete a some component from all the components shown by embodiment.

For example, the help screen 32 may be excluded from the target of the character recognition process when the character recognition process is performed on the image 300 created in step S3. Thereby, it is possible to improve the processing speed without extracting the item name of the help screen 32.

A method of excluding the help screen 32 from the target of character recognition processing is, for example, step S1 in FIG.
The window including the region where the mouse cursor was present when “position search” was pressed may be recognized as the help screen 32, and this window may be excluded from the target of the character recognition process. That is, the computer 20 acquires the position of the mouse cursor when “position search” is pressed in step S1. An area (window) having the same background color as the position where the acquired mouse cursor exists is searched, and the searched area is excluded from the character recognition processing target as the help screen 32.

Alternatively, the screen image creating unit 22 may temporarily create the help screen 32 when creating the screen image.

DESCRIPTION OF SYMBOLS 10 ... Input part, 20 ... Computer, 21 ... Selection range analysis part, 22 ... Screen image creation part 223
... Character recognition processing unit, 24 ... Operation target position search unit, 25 ... Operation target position search unit, 26 ... Search target list storage unit, 27 ... Character recognition process result storage unit, 30 ... Display unit, 100 ... Help display system, 300 ... Screen image

Claims (7)

A display unit for displaying an application screen and a help screen for the application screen;
A character recognition processing unit that performs character recognition processing on the screen image of the display unit;
A position search unit that compares the item on the help screen with the processing result of the character recognition process to search in which position on the application screen the item on the help screen exists,
A highlighting unit that highlights an item on the application screen corresponding to an item on the help screen based on a search result by the position search unit;
A help display device comprising: The character recognition processing unit
The help display device according to claim 1, wherein when a character is present in a predetermined background color region preset by a user in the screen image, the background color region is excluded from the target of the character recognition processing. The character recognition processing unit
The help display device according to claim 1, wherein a help screen in the screen image is excluded from the character recognition processing. The character recognition processing unit
3. The help display device according to claim 1, wherein when performing the character recognition process on the screen image, a window including an area where a mouse cursor is present is excluded from a processing target. 4. The highlight section
The help display according to any one of claims 1 to 4, wherein when there are a plurality of combinations of the item on the help screen and the corresponding item on the application screen, different highlighting is performed for each combination. apparatus. Displaying an application screen and a help screen for the application screen;
Performing character recognition processing on the screen image of the display unit;
Collating the item on the help screen with the processing result of the character recognition process, and searching for the position on the application screen where the item on the help screen exists;
Highlighting an item on the application screen corresponding to an item on the help screen based on a search result by the position search unit;
Help display method with. In the help display device,
A function for displaying an application screen and a help screen for the application screen;
A function of character recognition processing of the screen image of the display unit;
A function for comparing the item on the help screen with the processing result of the character recognition processing to search in which position on the application screen the item on the help screen exists;
A function for highlighting an item on the application screen corresponding to an item on the help screen based on a search result by the position search unit;
Help display program to execute.

Images