JP6434435B2 - Information processing apparatus, operation support method, and operation support program - Google Patents

Description

  The present invention relates to an information processing apparatus, an operation support method, and an operation support program.

  Conventionally, when operating application software, a practitioner who is not familiar with the target field of application software and the operation method of application software may not know how to operate application software. This is a factor that causes a decrease in efficiency and efficiency. In the prior art, a technique for preparing a manual or displaying information on an operation method is known in order to ensure and improve the accuracy and efficiency of the operation.

  For example, prepare a manual (hereinafter referred to as `` procedure manual '') that describes the target area of application software, how to proceed with the work, and how to operate the application software, and read, understand, and memorize it before the implementer starts the work. , Refer as necessary. Conventionally, a digital content management apparatus has created a procedure manual by inserting application software operation screen information into digital content.

  In that case, the application software operation screen capture was inserted into the digital content as an image and displayed. Also, as a method of searching for digital contents, a full-text search technique is used to search by inputting a keyword, or a search is performed using an image as a search key. For example, in the creation of content related to a method of using software having a GUI (Graphical User Interface), as a conventional content creation technique used when illustrating the method of use, a captured operation screen image is inserted into the content. There is something. In addition, the content that has been created in this way is used to refer to the part that the user needs, and as a conventional content effective utilization technique, the part that matches the keyword specified by the user There are a technique for searching for images using a full-text search technique and the like, and a technique for searching for a portion matching an image using an image search technique.

  A mechanism in which knowledge about the target field required by the application software implementer and information regarding the operation method of the application software is displayed as appropriate when the software is used is prepared as a function of the software itself during software development. As such a method, a method using online help, balloon help, or pop-up help has been adopted.

“Microsoft (registered trademark) Office”, [online], [searched on January 22, 2016], Internet <http://www.microsoft.com/en-us/office> “Create a site for free”, [online], [Search January 22, 2016], Internet <https://en.wordpress.com/> “Document / content creation tool”, [online], [searched on January 22, 2016], Internet <http://www.toshiba-toac.co.jp/service/digital-contents/tool/index_j.htm > “Balloon”, [online], [Search January 22, 2016], Internet <https://msdn.microsoft.com/en-us/library/windows/desktop/aa511451.aspx>

  However, in the conventional technology, there is a problem in that the information regarding the operation method of the application software and the actual operation screen of the application software cannot be appropriately linked. For example, when referring to a procedure manual in which application software operation screen information is inserted into digital content while viewing the actual application software operation screen, the user does not know where the information that the user wants to know is written during the operation. , It sometimes took time to find it. Also, for example, when the user wants to compare the actual operation screen of the application software with the operation screen of the application software inserted in the procedure manual, it is necessary to manually compare the actual operation screen with the procedure manual. There was a case.

  In order to solve the above-described problems and achieve the object, an information processing apparatus according to the present invention includes a captured image of an operation screen of application software and structural information regarding the structure of a GUI element displayed on the operation screen. An operation information storage unit that associates the acquisition unit that acquires operation screen information, the captured image of the operation screen of the application software acquired by the acquisition unit, and the structure information related to the structure of the GUI element displayed on the operation screen An operation screen information storage unit to be stored in, an insertion unit for inserting the operation screen information acquired by the acquisition unit into content including a content element describing an operation method of the application software, and an insertion unit inserted by the insertion unit GUI element of the operation screen information and the content related to the GUI element Characterized in that it comprises a content information storage unit for storing the content information storage unit in association with each element, the.

  The information processing apparatus of the present invention is displayed on the operation screen of the operation screen information in association with each operation screen of the application software inserted in the content including the content element describing the operation method of the application software. In addition, an operation information storage unit that stores structure information related to the structure of each GUI element, and a content element that describes an operation method of application software related to each GUI element are stored in association with each GUI element. The structure information regarding the structure of the content information storage unit and the GUI element of the currently displayed operation screen is extracted, and the similarity between the extracted structure information and the structure information stored in the operation information storage unit is Each operation screen inserted in the content is compared and stored in the operation information storage unit Among the structure information, a specifying unit that specifies an operation screen inserted into the content corresponding to the structure information having the highest similarity, structure information of each GUI element of the operation screen specified by the specifying unit, and the current The structure information of each GUI element of the displayed operation screen is compared, and each GUI element of the operation screen specified by the specifying unit and each of the currently displayed operation screens corresponding to the GUI elements respectively A GUI element storage unit that associates the GUI elements with each other and stores them in the GUI element storage unit.

  The operation support method of the present invention is an operation support method executed by the information processing apparatus, and includes a captured image of an operation screen of application software, structure information on a structure of a GUI element displayed on the operation screen, and The operation information is obtained by associating the acquisition step for acquiring the operation screen information, the capture image of the operation screen of the application software acquired by the acquisition step, and the structure information regarding the structure of the GUI element displayed on the operation screen. An operation screen information storing step for storing in the storage unit, an insertion step for inserting the operation screen information acquired in the acquisition step into content including a content element describing the operation method of the application software, and the insertion step The GUI element of the inserted operation screen information and the GUI element A content information storing step of storing in the content information storage unit in association with the content elements, characterized in that it contained.

  The operation support program according to the present invention includes an acquisition step of acquiring operation screen information including a captured image of an operation screen of application software and structure information relating to a structure of a GUI element displayed on the operation screen; An operation screen information storage step for storing the captured image of the operation screen of the application software acquired in the step and the structure information related to the structure of the GUI element displayed on the operation screen in the operation information storage unit; An operation step of inserting the operation screen information acquired in the step into a content including a content element describing the operation method of the application software, a GUI element of the operation screen information inserted in the insertion step, and the GUI element Conte related to Associating the tool element, characterized in that to execute the content information storing step of storing in the content information storage section, to the computer.

  According to the present invention, there is an effect that information on an operation method of application software and an actual operation screen can be appropriately linked.

FIG. 1 is a schematic configuration diagram showing the overall configuration of the operation support system according to the first embodiment. FIG. 2 is a block diagram showing the configuration of the creation apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of information stored in the operation information storage unit. FIG. 4 is a diagram for explaining the structure information of the GUI element. FIG. 5 is a diagram illustrating an example of a table stored in the content information storage unit. FIG. 6 is a diagram illustrating processing for inserting an operation screen into content. FIG. 7 is a diagram for explaining processing for displaying the boundary of the GUI element displayed on the content. FIG. 8 is a diagram illustrating processing for combining a content element and a GUI element. FIG. 9 is a block diagram showing the configuration of the user terminal according to the first embodiment. FIG. 10 is a diagram for describing processing for specifying an operation screen similar to the operation screen being operated from the content. FIG. 11 is a diagram illustrating a process of associating a GUI element in the operation screen being operated with a GUI element in the content. FIG. 12 is a diagram for explaining processing for displaying a content element on the operation screen during operation. FIG. 13 is a diagram for describing processing for checking information input to a GUI element on the operation screen during operation. FIG. 14 is a diagram illustrating a process of checking information input to a GUI element on the operation screen being operated. FIG. 15 is a diagram illustrating a process of automatically inputting a GUI element on the operation screen being operated. FIG. 16 is a diagram illustrating a process of recording information input to a GUI element on the operation screen being operated. FIG. 17 is a flowchart showing a flow of a process of associating a GUI element and a content element by the creating apparatus according to the first embodiment. FIG. 18 is a flowchart showing a flow of content element display processing on the operation screen by the user terminal according to the first embodiment. FIG. 19 is a diagram illustrating a computer that executes an operation support program.

  Embodiments of an information processing apparatus, an operation support method, and an operation support program according to the present application will be described below in detail with reference to the drawings. Note that the information processing apparatus, the operation support method, and the operation support program according to the present application are not limited by this embodiment.

[First embodiment]
In the following embodiment, the configuration of the operation support system according to the first embodiment, the configuration of the creation device and the user terminal, the processing flow of the creation device and the user terminal will be described in order, and finally the first embodiment The effect by form is demonstrated.

[Configuration of Operation Support System According to First Embodiment]
First, the operation support system according to the first embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram showing the overall configuration of the operation support system according to the first embodiment.

  As shown in FIG. 1, the operation support system is connected to the creation device 10 and the user terminal 20 via a network 30. The network 30 only needs to be configured so that the connected devices can communicate with each other. For example, the network 30 can be configured with the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like. In the example of FIG. 1, the number of the creation apparatuses 10 and the user terminals 20 is one, but the present invention is not limited to this, and there may be a plurality of creation apparatuses 10 and user terminals 20. The creation device 10 and the user terminal 20 are information processing devices such as a personal computer and a smartphone.

  The creation device 10 is an information processing device that creates content describing how to operate application software, and software for creating content is installed. Here, the content in which the operation method of the application software is described is, for example, digital content (an electronic file of document creation software) including an explanatory text in which the operation method of the application software is described.

  The creation apparatus 10 acquires operation screen information including a capture image of the operation screen of the application software and structure information regarding the structure of the GUI element displayed on the operation screen. The acquired capture image and the operation screen information Are stored in the operation information storage unit 14a, which will be described later, in association with structure information related to the structure of a GUI (Graphical User Interface) element displayed on the operation screen. Here, the operation screen information is log information of the operation screen captured when the operator of the creation apparatus 10 actually displays the operation screen of the application and operates the application.

  Further, the creation device 10 inserts the operation screen information into content including a content element in which an operation method of the application software is described, and includes a GUI element of the inserted operation screen information and a content element related to the GUI element. The content information is stored in the content information storage unit 14b, which will be described later. Here, the content element is, for example, a drawing object such as a graphic or a text document included in the digital content.

  The user terminal 20 is an information processing apparatus that displays an operation screen of application software, and software for linking the application software and application software operation screen information in content is installed. The user terminal 20 stores digital content created by the creation device 10. Furthermore, it is assumed that information that associates the operation screen information constructed by the creation device 10 with the structure information of the GUI element and information that associates the GUI element and the content element are stored.

  The user terminal 20 displays the content object (content element) included in the content on the actual application software operation screen, automatically operates the actual operation screen using the rules and procedures described in the content, or conversely The information on the application software operation screen is recorded in the content.

  As described above, the creation device 10 side can associate the GUI element displayed on the operation screen with the content element such as the graphic or text document included in the content, so that the user terminal 20 can execute the digital content and the execution. Application software can be linked to support operation of the application software in various forms (one-source multi-use).

[Configuration of creation device]
Next, the configuration of the creation apparatus 10 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the creation apparatus according to the first embodiment. As illustrated in FIG. 2, the creation device 10 includes an input unit 11, an output unit 12, a control unit 13, and a storage unit 14. The processing of each of these units will be described below.

  The input unit 11 receives various operations from the operator of the creation apparatus 10 and includes a mouse, a keyboard, and the like. Here, the operator is, for example, a person who creates content. The output unit 12 displays digital contents and the like, and includes a monitor (or display, touch panel) and a speaker.

  The storage unit 14 stores data and programs necessary for various processes performed by the control unit 13, and the operation information storage unit 14 a and the content information storage unit 14 b are particularly closely related to the present invention. Have. For example, the storage unit 14 is a semiconductor memory device such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

  The operation information storage unit 14a stores operation screen information obtained by capturing an operation screen of application software and structure information related to the structure of the GUI element displayed on the operation screen of the operation screen information in association with each other. That is, as illustrated in FIG. 3, the operation information storage unit 14 a is displayed on the operation screen and a captured image when the operator of the creation apparatus 10 actually displays the operation screen of the application and operates the application. The GUI element structure information is stored.

  Here, the structure information of the GUI element will be described using the example of FIG. FIG. 4 is a diagram for explaining the structure information of the GUI element. As illustrated in FIG. 4, a text box, a check box, and a button are displayed as GUI elements on the operation screen (window) displayed in the captured image of the operation screen. The operation information storage unit 14a stores the structure information of the text box, the check box, and the button in association with the captured image. For example, the operation information storage unit 14a stores the position and size, the type of GUI component, and the input value as the structure information.

  Specifically, referring to the example of FIG. 4, for example, the operation information storage unit 14a, as the text box structure information, “boundingRectangleH” indicating the height of the text box is “15”, and indicates the width of the text box. “BoundingRectangleW” is “347”, “boundingRectangleX” indicating the upper left X coordinate of the text box is “202”, “boundingRectangleY” indicating the upper left Y coordinate of the text box is “372”, and “ClassName” indicating the type of the GUI component “TextBox”, “name” indicating the name of the text box is “remarks column”, and “Value” indicating the value input to the text box is “AAA”.

  The content information storage unit 14b stores the GUI element of the operation screen information obtained by capturing the operation screen of the application software and the content element in which the description of the operation method of the application software is described in association with each other. For example, as illustrated in FIG. 5, the content information storage unit 14 b stores a “content object ID” that uniquely identifies a content element and a “GUI component identification ID” that uniquely identifies a GUI element in association with each other. To do. In the example of FIG. 5, the content object ID “1111” and the GUI component identification ID “2222” are stored in association with each other. This means that the content element with the content object ID “1111” and the GUI element with the GUI component identification ID “2222” are associated with each other and stored.

  The control unit 13 has an internal memory for storing a program that defines various processing procedures and necessary data, and performs various processes by using these programs. Particularly, as closely related to the present invention, It has the acquisition part 13a, the operation screen information storage part 13b, the insertion part 13c, the display part 13d, and the content information storage part 13e. Here, the controller 13 is an electronic circuit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  The acquisition unit 13a acquires operation screen information including a capture image of the operation screen of the application software and structure information regarding the structure of the GUI element displayed on the operation screen. Here, the captured image of the operation screen of the application software is, for example, log information of the operation screen captured when the operator of the creation apparatus 10 actually displays the operation screen of the application and operates the application. It is assumed that it is stored in the storage unit 14 in advance. For example, this operation screen is captured and stored as log information when an instruction from the operator is received, when the screen is switched, or at a predetermined time interval. Further, for example, the acquisition unit 13a operates when the content creator opens an electronic file of the document creation software when creating the content or when an instruction to select a captured image on the operation screen to be inserted into the content is received. Get screen information.

  The operation screen information storage unit 13b associates the captured image of the operation screen of the application software acquired by the acquisition unit 13a with the structure information regarding the structure of the GUI element displayed on the operation screen in the operation information storage unit 14a. Store. For example, in the example of FIG. 4 described above, the operation screen information storage unit 13b stores the structure information of the text box, the check box, and the button in association with the captured image.

  The insertion unit 13c inserts the operation screen information acquired by the acquisition unit 13a into content including a content element describing an operation method of application software. Here, processing for inserting an operation screen into content will be described with reference to FIG. FIG. 6 is a diagram illustrating processing for inserting an operation screen into content. For example, as illustrated in FIG. 6, the acquisition unit 13 a inserts into the content from information stored in the operation information storage unit 14 a when the operator displays the content and creates a procedure manual or the like. When an instruction to select a captured image on the operation screen to be received is received, structure information of the captured image is inserted into the content together with the selected captured image. Note that the “description” in FIG. 6 is a document input into the content by the operator. In the example of FIG. 6, a capture screen is inserted on the right side of the description.

  As described above, when a captured image of the operation screen is inserted into the content, the structure information of the operation screen displayed in the captured image of the operation screen to be inserted (GUI such as a text box or a button displayed on the operation screen). Information about the display position of the element and state information such as input information and display information) is stored in the operation information storage unit 14a.

  When the display unit 13d accepts selection of the GUI element of the operation screen information inserted by the insertion unit 13c, the display unit 13d selects using the coordinates displayed on the operation screen included in the structure information of the selected GUI element. The boundary between the area where the displayed GUI element is displayed and another area is displayed.

  Here, the process of displaying the boundary of the GUI element displayed on the content will be described with reference to FIG. FIG. 7 is a diagram for explaining processing for displaying the boundary of the GUI element displayed on the content. As shown in FIG. 7, the display unit 13d displays the coordinates ((2) in FIG. 7) when the mouse click position on the content (see (1) in FIG. 7) is actually displayed in the past. The GUI element (see (3) in FIG. 7) including this value is retrieved from the GUI element structure information, and the retrieved GUI element (“text box” in the example of FIG. 7) The coordinates of the boundary value (see (4) in FIG. 7) are converted into coordinates on the content display system, and the GUI element selected by clicking with the mouse is displayed in red and rectangular (see (5) in FIG. 7).

  Explaining a specific processing example, when the content is a power point, the display unit 13d acquires the display position of the slide, the display position of the inserted image, and the clicked coordinate position from the OS, and the image inserted from the coordinate position. Get the relative coordinates of the clicked position at. Then, the display unit 13d acquires the display magnification of the inserted image, multiplies the above-described coordinates by the display magnification, and acquires the coordinates on the operation screen capture. Subsequently, the display unit 13d searches the coordinate information in the structure information of the GUI element displayed on the operation screen capture, and acquires the subordinate GUI element (“text box” in the example of FIG. 7). To do. Then, the display unit 13d acquires the coordinate information of the subordinate GUI element, multiplies the acquired GUI element by the display magnification, and further adds the display position of the image to acquire the coordinates on the content of the GUI element. A red rectangle indicating the boundary between the area where the GUI element is displayed on the content and the other area is displayed.

  In this way, mutual coordinate conversion is performed between the coordinate system on the content when the operation screen capture image is displayed as the content and the coordinate system on the operation screen capture image. Then, the coordinates on the operation screen capture image corresponding to the arbitrary points on the capture image of the operation screen displayed on the content display system are calculated by coordinate conversion, and the coordinates of the boundary of each GUI element are compared. Determine which GUI element the point is on.

  The content information storage unit 13e stores the GUI element of the operation screen information inserted by the insertion unit 13c in association with the content element related to the GUI element in the content information storage unit 14b. Specifically, when the selection of the content element related to the GUI element is received, the content information storage unit 13e stores the selected GUI element and the content element in association with each other in the content information storage unit 14b.

  Referring to the example of FIG. 8, for example, when the description 1 and the text box are selected, the content object ID “1111” of the description 1 and the identification ID of the text box (GUI component identification of FIG. 5). (Corresponding to ID) “2222” in association with each other and stored in the content information storage unit 14b.

  In this way, from the point designated by the content creator by mouse click or the like, the GUI element in the operation screen capture image displayed on the content system is selected, while the content element displayed in the content is selected, The correspondence relationship is stored in a table in the content information storage unit 14b that holds the correspondence between the content element and the GUI element.

[User terminal configuration]
Next, the configuration of the user terminal 20 shown in FIG. 1 will be described with reference to FIG. FIG. 9 is a block diagram showing the configuration of the user terminal according to the first embodiment. As illustrated in FIG. 9, the user terminal 20 includes an input unit 21, an output unit 22, a control unit 23, and a storage unit 24. The processing of each of these units will be described below. It is assumed that the user terminal 20 stores the content created by the creation device 10 as a procedure manual for application software.

  The input unit 21 receives various operations from a user who owns the user terminal 20, and includes a mouse, a keyboard, and the like. Here, the user is a person who uses application software. The output unit 22 displays an operation screen of application software and the like, and includes a monitor (or display, touch panel) and a speaker.

  In addition, the storage unit 24 stores data and programs necessary for various processes by the control unit 23, and in particular, those closely related to the present invention include an operation information storage unit 24 a, a content information storage unit 24 b, And a GUI element storage unit 24c. For example, the storage unit 24 is a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

  The operation information storage unit 24a is associated with each operation screen of the application software inserted in the content including the content element describing the operation method of the application software, and displays each GUI displayed on the operation screen of the operation screen information. (Graphical User Interface) Stores structure information on the structure of each element. Here, the operation information storage unit 24a stores the same information as the operation information storage unit 14a described above. Here, each operation screen of the application software inserted in the content is a screen displayed on the creation device 10 when the content creator actually operates, and is inserted into the content by the creation device 10. It is the screen that was made.

  In addition, the content information storage unit 24b stores content elements in which operation methods of application software related to the GUI elements are described in association with the GUI elements. The content information storage unit 24b stores the same information as the content information storage unit 14b described above. That is, the operation information storage unit 24a and the content information storage unit 24b store information that associates the operation screen information constructed by the creation device 10 with the structural information of the GUI element, and information that associates the GUI element with the content element. Suppose you are.

  The GUI element storage unit 24c associates each GUI element of the operation screen inserted into the content by the content creator actually operating the application with each GUI element of the operation screen currently displayed on the user terminal 20. Attached and remembered. For example, the GUI element storage unit 24c stores the identifier of the text box of the operation screen inserted into the content and the identifier of the text box of the operation screen currently displayed on the user terminal 20 in association with each other. Further, for example, check box and button identifiers of operation screens inserted in the content and check boxes and button identifiers of operation screens currently displayed on the user terminal 20 are stored in association with each other.

  The control unit 23 has an internal memory for storing a program that defines various processing procedures and the necessary data, and performs various processes by using these programs. Particularly, as closely related to the present invention, It has a specifying unit 23a, a GUI element storage unit 23b, a content display unit 23c, a notification unit 23d, an automatic input unit 23e, and a recording unit 23f. Here, the control unit 23 is an electronic circuit such as a CPU or MPU, or an integrated circuit such as an ASIC or FPGA.

  The specifying unit 23a extracts structure information related to the structure of the GUI element of the currently displayed operation screen, and uses the similarity between the extracted structure information and the structure information stored in the operation information storage unit 24a as content. Comparison is made for each inserted operation screen, and the operation screen inserted in the content corresponding to the structure information having the highest similarity among the pieces of structure information stored in the operation information storage unit 24a is specified.

  Here, a process for specifying an operation screen similar to the operation screen being operated from the content will be described with reference to FIG. FIG. 10 is a diagram for describing processing for specifying an operation screen similar to the operation screen being operated from the content. As illustrated in FIG. 10, the specifying unit 23a compares the GUI element structure information associated with the operation screen that is actually operated with the GUI element structure information associated with the operation screen in the content, and evaluates the degree of similarity. To do. The identifying unit 23a identifies an operation screen displayed in the past in the content corresponding to the structural information having the highest similarity.

  For example, in the example of FIG. 10, the specifying unit 23a searches the content for an operation screen on which text boxes, check boxes, and buttons that are GUI elements of the operation screen that is actually operated are displayed. When there are a plurality of operation screens on which text boxes, check boxes, and buttons are displayed, the specifying unit 23a displays information (character strings and values) input to the text boxes, check box states (on / on Off) may be evaluated as having a high degree of similarity, and, for example, GUI elements that are more similar in size (height and length) are similar as structural information of each GUI element. You may evaluate as a high degree. Then, the specifying unit 23a specifies the operation screen inserted in the content corresponding to the structural information having the highest similarity. The method for evaluating the degree of similarity described above is an example, and any method may be used, and the method may be changed as appropriate.

  As described above, the operation screen capture image and the structure information of the GUI element such as the button and text displayed on the operation screen are extracted from the operation screen being executed, and inserted into the content using the structure information of the extracted GUI element. A similar operation screen capture image and set of GUI element structure information are retrieved from a set of the set operation image capture image and GUI element structure information set. For this reason, for example, text strings and check boxes that are displayed without inputting terms using the application software operation screen information that is actually operated on the application software operation screen in the content as a search condition. The search can be performed in consideration of the type and state of the GUI element such as the position of the slider.

  The GUI element storage unit 23b compares the structure information of each GUI element of the operation screen specified by the specifying unit 23a with the structure information of each GUI element of the currently displayed operation screen, and is specified by the specifying unit 23a. Each GUI element on the operation screen and the GUI element on the currently displayed operation screen corresponding to each GUI element are stored in the GUI element storage unit 24c in association with each other.

  For example, as illustrated in FIG. 11, the text box, check box, and button of the operation screen inserted in the content, and the text box of the operation screen currently displayed on the user terminal 20 (see (1) in FIG. 11). , Check boxes (see (2) in FIG. 11) and buttons (see (3) in FIG. 11) are associated with each other and stored in the GUI element storage unit 24c. As described above, the captured image of the inserted operation screen is compared with the operation screen being executed, and both of the GUI elements in the operation screen being executed corresponding to the GUI element in the captured image of the inserted operation screen are displayed. Using the structure information of the operation screen, it identifies and associates.

  The content display unit 23c acquires each GUI element of the operation screen inserted into the content corresponding to each GUI element of the currently displayed operation screen from the GUI element storage unit 24c, and corresponds to each acquired GUI element The content element to be acquired is acquired from the content information storage unit 24b, and the content element is displayed on the currently displayed operation screen.

  For example, as illustrated in FIG. 12, the content display unit 23c displays “text box”, “check box”, and “button” that are GUI elements of the currently displayed operation screen (the screen on the right side of FIG. 12). The “text box”, “check box”, and “button”, which are the GUI elements of the operation screen (the screen on the left side in FIG. 12) inserted in the content corresponding to “”, are acquired from the GUI element storage unit 24c. Then, the content display unit 23c stores the content elements “description A”, “description B”, and “description C” corresponding to the acquired GUI elements “text box”, “check box”, and “button”. Obtained from the information storage unit 24b. Then, the content display unit 23c displays “description A” near the “text box” on the currently displayed operation screen, and “description” near the “check box” on the currently displayed operation screen. B ”is displayed, and“ description C ”is displayed in the vicinity of the“ button ”on the currently displayed operation screen.

  As described above, the user terminal 20 acquires the GUI element displayed in the operation screen capture image inserted in the content corresponding to the GUI element in the operation screen being executed, from the content information storage unit 24b. The content element corresponding to the acquired GUI element is acquired, and the whole or a part is displayed in the vicinity of the GUI element in the operation screen being executed. Thereby, the user terminal 20 can refer to necessary information in the content without knowing an appropriate term for searching for necessary information in the content. In addition, since the search is performed using the structure information of the GUI element, the type of GUI element such as the text string displayed on the operation screen and the check box on / off, which cannot be realized only by the image search, Information necessary for the user in the content can be displayed as a search condition such as the state.

  When the predetermined information is input to the GUI element of the currently displayed operation screen, the notifying unit 23d displays the operation screen inserted into the content corresponding to the GUI element of the currently displayed operation screen. When the condition for the predetermined information input to the GUI element is set in advance, it is determined whether the input predetermined information satisfies the condition. If it is determined that the condition is not satisfied, the input is performed. A warning for the information is sent. Note that the condition set for each GUI element is set in advance by the content creator when creating the content. If no condition is set for the GUI element, the notification unit 23d executes Not.

  Here, a process of checking information input to the GUI element on the operation screen being operated will be described with reference to the examples of FIGS. 13 and 14. FIG. 13 and FIG. 14 are diagrams for describing processing for checking information input to a GUI element on the operation screen being operated. For example, in the example of FIG. 13, it is set that “AAA” is input as the input condition of the text box on the operation screen inserted in the content. In the example of FIG. 13, it is assumed that “BBB” is input in the text box on the operation screen being operated corresponding to the text box on the operation screen inserted into the content.

  In this case, the notification unit 23d determines that the input condition is not satisfied because “BBB” input in the text box of the operation screen being operated is not “AAA”, and issues a warning for the input information. Inform. Here, as a warning, for example, the character “BBB” input in the text box may be red, or a message may be displayed indicating that “BBB” input in the text box is not appropriate. . Note that “AAA” is information input on the application software operation screen inserted into the content, and the application software operation screen inserted into the content and the operation screen currently being operated are compared to display different portions. be able to.

  For example, in the example of FIG. 14, “2 <X <5” is set as the input condition of the text box on the operation screen inserted in the content. In the example of FIG. 14, it is assumed that “7” is input in the text box of the operation screen being operated corresponding to the text box of the operation screen inserted in the content. In this case, the notification unit 23d determines that the set input condition “2 <X <5” is not satisfied because “7” input in the text box of the operation screen being operated is greater than 5, and the input A warning for the information is sent.

  In this way, using the conditions set for each GUI element, the value set in the GUI element, such as the character string in which the corresponding GUI element is set in the text box or the check state of the check box, is evaluated. be able to. That is, the user terminal 20 can perform an input check on the GUI element of the screen that is actually being operated using the conditions related to the GUI element of the application software operation screen inserted into the content.

  The automatic input unit 23e is set when information to be input to the GUI element of the operation screen inserted in the content corresponding to the GUI element of the currently displayed operation screen is set in advance. Information is input to the GUI element of the currently displayed operation screen. For example, as illustrated in FIG. 15, when “AAA” is preset as information to be entered in the text box of the operation screen inserted in the content, the automatic input unit 23 e is currently displayed. "AAA" is automatically entered into the text box on the operation screen.

  When predetermined information is input to the GUI element of the currently displayed operation screen, the recording unit 23f records the input information in the content. For example, as illustrated in FIG. 16, when “CCC” is input in the text box of the currently displayed operation screen, the input “CCC” corresponds to the currently displayed operation screen. Record on the slide of the operation screen inserted in the content.

[Example of creation device processing]
Next, processing of the creation apparatus 10 will be described with reference to FIG. FIG. 17 is a flowchart showing a flow of a process of associating a GUI element and a content element by the creating apparatus according to the first embodiment.

  As illustrated in FIG. 17, the acquisition unit 13 a of the creation apparatus 10 acquires operation screen information including a captured image of an operation screen of application software and structure information regarding the structure of the GUI element displayed on the operation screen. (Step S101). For example, the acquisition unit 13a operates the operation screen information when the content creator opens an electronic file of the document creation software when the content is created, or when an instruction to select a captured image of the operation screen to be inserted into the content is received. To get.

  Then, the operation screen information storage unit 13b associates the captured image of the operation screen of the application software acquired by the acquisition unit 13a with the structure information regarding the structure of the GUI element displayed on the operation screen, and stores the operation information storage unit 14a (step S102).

  Subsequently, the insertion unit 13c inserts the operation screen information acquired by the acquisition unit 13a into the content including the content element describing the operation method of the application software (step S103). For example, the acquisition unit 13a selects the captured image of the operation screen to be inserted into the content from the information stored in the operation information storage unit 14a when the operator displays the content and creates a procedure manual or the like. When the instruction is received, the structure information of the captured image is inserted into the content together with the selected captured image.

  When the selection of the GUI element of the operation screen information inserted by the insertion unit 13c is accepted (Yes at Step S104), the selection is performed using the coordinates displayed on the operation screen included in the structure information of the selected GUI element. The boundary between the area where the GUI element is displayed and another area is displayed (step S105).

  Thereafter, when the content information storage unit 13e receives selection of a content element related to the GUI element (Yes in Step S106), the selected GUI element and the content element are associated with each other and stored in the content information storage unit 14b ( Step S107).

[Example of user terminal processing]
Next, the process of the user terminal 20 is demonstrated using FIG. FIG. 18 is a flowchart showing a flow of content element display processing on the operation screen by the user terminal according to the first embodiment.

  As illustrated in FIG. 18, when the specifying unit 23 a of the user terminal 20 displays the operation screen of the application software (Yes in step S <b> 201), an operation in which an operation screen similar to the displayed operation screen is inserted into the content is displayed. Identification is made from the screen (step S202). Specifically, the specifying unit 23a extracts structure information related to the structure of the GUI element of the currently displayed operation screen, and the similarity between the extracted structure information and the structure information stored in the operation information storage unit 24a. Are compared for each operation screen inserted in the content, and the operation screen inserted in the content corresponding to the structural information having the highest similarity among the pieces of structural information stored in the operation information storage unit 24a is specified. .

  Then, the GUI element storage unit 23b associates the GUI element in the operation screen being operated with the GUI element in the past operation screen inserted in the content (step S203). Specifically, the GUI element storage unit 23b compares the structure information of each GUI element of the operation screen specified by the specifying unit 23a with the structure information of each GUI element of the currently displayed operation screen, and specifies The GUI elements of the operation screen specified by the unit 23a and the GUI elements of the currently displayed operation screen corresponding to the GUI elements are stored in the GUI element storage unit 24c in association with each other.

  Subsequently, the content display unit 23c displays the content element on the operation screen being operated (step S204). Specifically, the content display unit 23c acquires each GUI element of the operation screen inserted into the content corresponding to each GUI element of the currently displayed operation screen from the GUI element storage unit 24c, and acquires the acquired GUI element. The content element corresponding to each GUI element is acquired from the content information storage unit 24b, and the content element is displayed on the currently displayed operation screen.

  Then, the automatic input unit 23e determines whether information to be input to the GUI element of the operation screen inserted in the content corresponding to the GUI element of the currently displayed operation screen is set in advance (Step S205). As a result, when it is determined that the automatic input unit 23e is set (Yes at Step S205), the set information is input to the GUI element of the currently displayed operation screen (Step S206). ), The process proceeds to step S207. If the automatic input unit 23e determines that it has not been set (No at Step S205), the process directly proceeds to Step S207.

  Subsequently, the notification unit 23d determines whether or not predetermined information has been input for the GUI element of the currently displayed operation screen (step S207). As a result, when the notification unit 23d determines that the predetermined information has been input (Yes in step S207), the notification unit 23d displays the GUI element of the operation screen inserted into the content corresponding to the GUI element of the currently displayed operation screen. On the other hand, if the condition for the predetermined information input is set in advance, it is determined whether the input predetermined information satisfies the condition (step S208).

  As a result, when the notification unit 23d determines that the condition is not satisfied (No at Step S209), the notification unit 23d notifies a warning to the input information (Step S210), returns to Step S208, and returns to the predetermined input that has been input again. Determine if the information meets the condition. Further, when it is determined that the condition is satisfied (Yes at Step S209), the recording unit 23f displays the input information when predetermined information is input to the GUI element of the currently displayed operation screen. Record in the content (step S211).

[Effect of the first embodiment]
As described above, in the operation support system according to the first embodiment, the creation apparatus 10 includes the captured image of the operation screen of the application software and the structure information regarding the structure of the GUI element displayed on the operation screen. The operation screen information is acquired, and the acquired captured image and the structure information regarding the structure of the GUI element displayed on the operation screen are associated with each other and stored in the operation information storage unit 14a. Then, the creation device 10 inserts the acquired operation screen information into the content including the content element in which the operation method of the application software is described, and relates to the GUI element of the inserted operation screen information and the GUI element. The content elements are associated with each other and stored in the content information storage unit 14b. For this reason, the information regarding the operation method of the application software can be appropriately linked with the actual operation screen. In other words, it is possible to associate the GUI element of the operation screen information inserted into the content and the content element describing the operation method of the application software, thereby linking the digital content and the running application software, The operation support can be performed in various forms. In addition, the user terminal 20 can be used as online help, check tool contents, or an automatic operation tool as one-source multi-use that performs operation support of application software in various forms.

  In addition, the user terminal 20 associates each GUI displayed on the operation screen with the operation screen information in association with each operation screen of the application software inserted in the content including the content element describing the operation method of the application software. An operation information storage unit 24a that stores structure information related to the structure of each element, and a content information storage that stores content elements that describe an operation method of application software related to each GUI element in association with each GUI element Part 24b. Then, the user terminal 20 extracts structure information related to the structure of the GUI element of the currently displayed operation screen, and determines the similarity between the extracted structure information and the structure information stored in the operation information storage unit 24a. Comparison is made for each operation screen inserted in the content, and the operation screen inserted in the content corresponding to the structural information having the highest similarity among the pieces of structural information stored in the operation information storage unit 24a is specified. Subsequently, the user terminal 20 compares the structure information of each GUI element of the specified operation screen with the structure information of each GUI element of the currently displayed operation screen, and each GUI element of the specified operation screen. Are stored in the GUI element storage unit 24c in association with the GUI elements of the currently displayed operation screen corresponding to the GUI elements.

  Thereby, the information regarding the operation method of the application software can be appropriately linked with the actual operation screen. For example, the application software operation screen information that is actually operated on the application software operation screen in the content is used as a search condition, and text strings and check boxes that are displayed without entering terms are turned on / off. A search can be performed in consideration of the type and state of a GUI element such as a position. This makes it possible to refer to a necessary part in the content without knowing an appropriate term for searching. In addition, search of necessary parts in the content, such as the text string displayed on the operation screen and the type and status of GUI elements such as check box ON / OFF, which cannot be realized by searching only for images, Reference is possible.

  Further, the user terminal 20 acquires each GUI element of the operation screen inserted in the content corresponding to each GUI element of the currently displayed operation screen from the GUI element storage unit 24c, and stores the acquired GUI element in each GUI element acquired. The corresponding content element is acquired from the content information storage unit 24b, and the content element is displayed on the currently displayed operation screen. For this reason, the content element regarding the GUI element of the application software operation screen inserted in the content can be displayed on the actual application software operation screen.

  In addition, the user terminal 20 performs an operation inserted into content corresponding to the GUI element of the currently displayed operation screen when predetermined information is input to the GUI element of the currently displayed operation screen. When the condition for the predetermined information input to the GUI element of the screen is set in advance, it is determined whether the input predetermined information satisfies the condition, and when it is determined that the condition is not satisfied A warning for the input information is notified. For this reason, it is possible to perform an input check on the GUI element of the screen that is actually being operated using the conditions related to the GUI element of the application software operation screen inserted into the content.

(System configuration etc.)
Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Further, all or any part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  In addition, among the processes described in this embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed All or a part of the above can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

(program)
It is also possible to create a program in which processing executed by the creation device 10 and the user terminal 20 according to the above-described embodiment is described in a language that can be executed by a computer. In this case, the same effect as the above-described embodiment can be obtained by the computer executing the program. Further, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer and executed to execute the same processing as in the above embodiment. Below, an example of the computer which performs the operation assistance program which implement | achieves the function similar to the production apparatus 10 or the user terminal 20 is demonstrated.

  FIG. 19 is a diagram illustrating a computer that executes an operation support program. As illustrated in FIG. 19, the computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. For example, a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050. For example, a display 1130 is connected to the video adapter 1060.

  Here, as shown in FIG. 19, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each table described in the above embodiment is stored in, for example, the hard disk drive 1090 or the memory 1010.

  Further, the operation support program is stored in the hard disk drive 1090 as a program module in which a command executed by the computer 1000 is described, for example. Specifically, a program module describing each process executed by the creation apparatus 10 or the user terminal 20 described in the above embodiment is stored in the hard disk drive 1090.

  Data used for information processing by the operation support program is stored as program data, for example, in the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1090 to the RAM 1012 as necessary, and executes the above-described procedures.

  The program module 1093 and the program data 1094 related to the operation support program are not limited to being stored in the hard disk drive 1090. For example, the program module 1093 and the program data 1094 are stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. May be issued. Alternatively, the program module 1093 and the program data 1094 related to the operation support program are stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and are transmitted via the network interface 1070. May be read by the CPU 1020.

DESCRIPTION OF SYMBOLS 10 Creation apparatus 11, 21 Input part 12, 22 Output part 13, 23 Control part 13a Acquisition part 13b Operation screen information storage part 13c Insertion part 13d Display part 13e Content information storage part 14, 24 Storage part 14a, 24a Operation information storage part 14b, 24b Content information storage unit 20 User terminal 23a Identification unit 23b GUI element storage unit 23c Content display unit 23d Notification unit 23e Automatic input unit 23f Recording unit 24c GUI element storage unit 30 Network

Claims (7)

An acquisition unit that acquires operation screen information including a captured image of an operation screen of the application software and structure information on a structure of a GUI (Graphical User Interface) element displayed on the operation screen;
An operation screen information storage unit that stores the captured image of the operation screen of the application software acquired by the acquisition unit and the structure information related to the structure of the GUI element displayed on the operation screen in association with each other in the operation information storage unit;
An insertion unit for inserting the operation screen information acquired by the acquisition unit into content including a content element in which an operation method of the application software is described;
When the selection of the GUI element of the operation screen information inserted by the insertion unit is received, the selected GUI is used by using the coordinates displayed on the operation screen included in the structure information of the selected GUI element. A display for displaying the boundary between the area where the element is displayed and another area;
When accepting the selection of content element associated with the GUI element, a content information storage unit for storing the content information storage unit in association with the previous SL content elements and that the selected GUI element,
An information processing apparatus comprising: Structure information related to the structure of each GUI element displayed on each operation screen is stored in association with each operation screen of the application software inserted in the content including the content element describing the operation method of the application software. An operation information storage unit;
A content information storage unit that stores content elements in which operation methods of application software related to the GUI elements are described in association with the GUI elements;
The structure information regarding the structure of the GUI element of the currently displayed operation screen is extracted, and the similarity between the extracted structure information and the structure information stored in the operation information storage unit is inserted into the content. Compared for each screen, among the structure information stored in the operation information storage unit, a specifying unit for specifying the operation screen inserted in the content corresponding to the structure information having the highest similarity,
The structure information of each GUI element of the operation screen specified by the specifying unit is compared with the structure information of each GUI element of the currently displayed operation screen, and each GUI of the operation screen specified by the specifying unit is compared. A GUI element storage unit that associates an element with each GUI element of the currently displayed operation screen corresponding to each GUI element, and stores the element in the GUI element storage unit;
An information processing apparatus comprising: Each GUI element of the operation screen inserted in the content corresponding to each GUI element of the currently displayed operation screen is acquired from the GUI element storage unit, and a content element corresponding to the acquired GUI element is obtained. The information processing apparatus according to claim 2 , further comprising: a content display unit that is obtained from the content information storage unit and displays the content element on the currently displayed operation screen. When predetermined information is input to the GUI element of the currently displayed operation screen, the GUI element of the operation screen inserted in the content corresponding to the GUI element of the currently displayed operation screen 3. The method according to claim 2 , further comprising: a determination notification unit that determines whether the input predetermined information satisfies the condition when a condition for the predetermined information input is set in advance. Information processing device. When information to be input to the GUI element of the operation screen inserted into the content corresponding to the GUI element of the currently displayed operation screen is set in advance, the set information is The information processing apparatus according to claim 2 , further comprising an automatic input unit configured to input to a GUI element of an operation screen currently displayed. An operation support method executed by an information processing apparatus,
An acquisition step of acquiring operation screen information including a captured image of the operation screen of the application software and structure information regarding the structure of the GUI element displayed on the operation screen;
An operation screen information storage step of storing the captured image of the operation screen of the application software acquired by the acquisition step and the structure information regarding the structure of the GUI element displayed on the operation screen in association with each other in the operation information storage unit;
An insertion step of inserting the operation screen information acquired by the acquisition step into content including a content element describing an operation method of the application software;
When the selection of the GUI element of the operation screen information inserted by the insertion step is accepted, the selected GUI is used by using the coordinates displayed on the operation screen included in the structure information of the selected GUI element. A display process for displaying a boundary between the area where the element is displayed and another area;
When accepting the selection of content element associated with the GUI element, a content information storing step of storing in the content information storage unit in association with the previous SL content elements and that the selected GUI element,
An operation support method characterized by comprising An acquisition step of acquiring operation screen information including a captured image of an operation screen of the application software and structure information regarding a structure of a GUI element displayed on the operation screen;
An operation screen information storage step for storing the captured image of the operation screen of the application software acquired in the acquisition step and the structure information related to the structure of the GUI element displayed on the operation screen in the operation information storage unit;
An insertion step of inserting the operation screen information acquired in the acquisition step into content including a content element describing an operation method of the application software;
When the selection of the GUI element of the operation screen information inserted by the insertion step is accepted, the selected GUI is used by using the coordinates displayed on the operation screen included in the structure information of the selected GUI element. A display step for displaying a boundary between the area where the element is displayed and another area;
When accepting the selection of content element associated with the GUI element, a content information storing step of storing in the content information storage unit in association with the previous SL content elements and that the selected GUI element,
An operation support program for causing a computer to execute.

Images