JP2013211030A - Customization method of unser interface operation integration system, terminal device, computer program and information recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a customization method of a user interface operation integration system, a terminal device, a computer program and an information recording medium that can facilitate new addition of processing of an extraction and recording of an operation content corresponding to addition so as not to affect processing of other User Interface (UI) type or a format thereof when an UI type is added.SOLUTION: A terminal device 10 integrally operating a plurality of types of applications comprises: an event message monitor part 142 that monitors operation information of a target application by an operator 1; a teaching part 131 that stores the operation information received from the event message monitor part 142 in a scenario file 232 and a work data file 231; a process control part 132 that transmits the operation information read from the scenario file 232 and the work data file 231; and an operation execution part 141 that executes an operation of the targe application on the basis of the operation information received from the process control 132.

Description

  The present invention belongs to a technique for operating a plurality of applications operating on a client terminal device in an integrated manner.

  Conventionally, techniques for efficiently coordinating a plurality of systems have been studied in order to improve business efficiency such as network operation.

  For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2004-38759) provides a technique for operating various types of application server software in an integrated manner. This technology consists of a script processing engine that performs common processing that does not depend on differences between applications, and an application adapter that performs individual processing related to data value acquisition / setting in response to differences between applications. ing. The script processing engine executes a script created in advance according to the content of the information to be displayed in a coordinated manner, and connects both functional units with a common interface (hereinafter simply referred to as IF) that does not depend on the application type. As a result, development at a low cost and in a short period of time is realized compared to the normal development method.

  On the other hand, for example, software tools for automating application operations are released and distributed as free software. The operation automation tool monitors events of input / output devices such as a mouse and keyboard as well as messages of user interface (hereinafter simply referred to as UI) elements such as text fields and buttons when the user actually operates the target application. It is possible to extract the necessary information and save them in a file, and generate them (emulation) later to execute the same operation contents any number of times. That is, the contents of the intended application operation can be indirectly described without having detailed knowledge of the program.

  Alternatively, for example, in Patent Document 2 (Japanese Patent Application Laid-Open No. 6-301525), regarding an execution control method in a multi-window system, a process of linking a plurality of windows can be executed without spending a great deal of labor. Also in this technique, the object is achieved by the same internal configuration as the above free software.

JP 2004-38759 A JP-A-6-301525

  In the technique shown in Patent Document 1, the processing handled in cooperation is specialized in keeping data acquisition and updating up-to-date. In actual operations such as network operations, it is desirable to be able to execute various processes such as data retrieval and external device control. Further, in Patent Document 1, development can be performed without compiling using a script editor within the range of data update acquisition processing. This can be developed at a low cost and in a short period of time, compared to the normal development method, but without the development itself (coding and programming), the business skill person himself can define and execute the application specifications. A simpler and simpler method is desirable.

  From this point of view, as with the application operation automation tool, it is effective to extract and record the operation content from the demonstration of the user's operation and execute the same content afterwards an arbitrary number of times. However, the method of the tool can deal only with a predetermined UI type. Actual network operation support systems are often different systems depending on the equipment to be managed, considering that they are implemented on different window systems and different graphic user interface (hereinafter simply referred to as GUI) platforms. There is a need to. In particular, in the current situation where a new system is developed in a cycle of several years according to a network (hereinafter simply referred to as NW) equipment with many technological innovations, it is necessary to be able to cope with it. That is, when a UI type is added, the processing and format for extracting / recording the operation contents corresponding to the UI type must be newly added so as not to affect the processing and format of other UI types, It was difficult to realize.

  In view of the above-described problems, the object of the present invention is that when a UI type is added, the processing and format of extraction / recording of the operation content corresponding to the added UI type does not affect the processing and format of other UI types. It is to provide a user interface operation integration system customization method, a terminal device, a computer program, and an information recording medium that can be easily newly added.

The present invention is a method for customizing a user interface operation integration system in a computer terminal device that operates a plurality of types of applications in an integrated manner . A user interface operation / monitoring interface providing unit having an event message monitoring unit connectable to an adapter and a business record / execution interface providing unit having a teaching unit and a process control unit are integrated into the internal architecture of the user interface operation integrated system. And providing an adapter type identification / distribution function unit between the operation execution unit and the process control unit and between the event message monitoring unit and the teaching unit, The teaching unit includes a plurality of additional expandable teaching modules, the process control unit includes a plurality of additional expandable replay modules, and the event message monitoring unit receives a predetermined teaching module from the teaching unit. An instruction is received, an event for a specific window or component is monitored, and when the event occurs, the content of the event or message is notified to the teaching module that has received the instruction, and the teaching module that has received the notification is sent from the event message monitoring unit. The event content or message content is received, and a work data file describing the work execution content based on the event content or message content and a scenario file describing the work execution order are stored in the storage unit. The predetermined replay module reads the work data file and the scenario file stored in the storage unit, and sends the event contents or message contents to the operation execution unit based on the contents of the work data file and the scenario file. The operation execution unit receives the event content or message content transmitted from the replay module, executes a predetermined user interface operation, and replays with the teaching module corresponding to the target application according to the addition of the target application. Add a module.

  According to the present invention, by using a plurality of additionally expandable teaching modules, the types of monitoring information that can be used in the teaching unit can be additionally expanded without significantly affecting the conventional processing, and further expansion is possible. By using such a plurality of replay modules, the types of execution means that can be used in the process control unit can be additionally expanded without significantly affecting the conventional processing.

The figure which shows one form of the system configuration | structure in one Embodiment of this invention. The figure which shows the other form of the system configuration | structure in one Embodiment of this invention. The figure which shows the other form of the system configuration | structure in one Embodiment of this invention. The functional block diagram which shows the system terminal device in one Embodiment of this invention The flowchart which shows the procedure of the process which the system terminal device in one Embodiment of this invention performs The flowchart which shows the procedure of the process which the system terminal device in one Embodiment of this invention performs The flowchart which shows the procedure of the process which the system terminal device in one Embodiment of this invention performs The internal block diagram of the teaching part in one Embodiment of this invention The figure which shows the work history preserve | saved at the scenario file in one Embodiment of this invention The figure which shows the work data preserve | saved at the work data file in one Embodiment of this invention 1 is a functional block diagram illustrating an IF distribution method for an event message monitoring unit according to an embodiment of the present invention. FIG. 3 is a sequence diagram illustrating an IF distribution method for an event message monitoring unit according to an embodiment of the present invention. The figure which shows the internal structure of the process control part which performs automatically based on the file created in the teaching part in one Embodiment of this invention The functional block diagram explaining the IF allocation method with respect to the operation execution part in one Embodiment of this invention FIG. 3 is a sequence diagram illustrating an IF distribution method for an operation execution unit according to an embodiment of the present invention. The figure which shows the function call relationship in the case of implementing the know-how regarding the external control of a target application in one Embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  1 to 3 are system configuration diagrams according to an embodiment of the present invention. In these drawings, reference numeral 1 denotes an operator (user) of the system terminal device 10. A plurality of application UIs (11) and an integrated system 12 are provided in a system terminal device 10 that is a computer device that operates according to a predetermined computer program. The operator 1 can operate the application UI (11) of the system terminal apparatus 10 to operate the system terminal apparatus 10 and the existing system 20 connected to the system terminal apparatus 10. At this time, the integrated system 12 controls 14 the operation of the application UI (11) in response to the operation 13.

  The integrated system 12 in the present embodiment basically resides in the system terminal device 10 and monitors and substitutes the operation of the target application UI (11). The usage mode shown in FIG. 1 shows a case where an arbitrary UI operation sequence is automatically executed only by pressing the button of the operator 1. In the usage mode shown in FIG. 2, a case is shown in which interactive processing is performed such as confirming data contents in the middle of the UI operation sequence or prompting an input from the operator 1. This can be realized without changing the basic function of the integrated system 12 by separately developing a dummy UI (15) for the configuration shown in FIG. In this case, however, the operation includes advanced contents such as input to the read data input column of the data display on the UI. Details will be described later. The usage mode shown in FIG. 3 shows a case where a target job cannot be realized only by acquiring and setting data from a plurality of application UIs (11). This is a simple helper application 16 that executes processing contents that are insufficient in the form shown in FIG. 2 (for example, check logic for combinations of data values, judgment logic for changing execution contents based on predetermined rules, etc.) Can be realized by developing separately. In the present embodiment, the helper application 16 includes a business logic 17 including the check logic and the determination logic.

  As described above, the system according to the present embodiment is a technology that deals with the integration related to the operation of the user interface, and the integration related to the display can be freely combined with that established as a separate technology. This means that the display integration technique independent of the mechanism of the present invention can be linked in a codingless manner as shown in FIG. 3 by the teaching function described later. Therefore, it constitutes a user interface operation integration technology in a broad sense (meaning to handle both operation and display).

  In this embodiment, as will be described later, the automatic execution content is held in the form of a work data file and a scenario file. Specifically, it does not perform hard coding as conventional program code, or externally store the program logic as script code (Patent Document 1 or free software). Only the definition of the unit elements (defined by the func tag of the function part of the scenario file) and the execution order (specified by the sequence of the func tag of the scenario part) are retained, and the detailed processing contents required to actually execute them Is hidden in the replay module of the process control unit. Also, in the teaching module of the teaching unit (the flow shown in FIG. 6), the work data definition (work data file) is clearly defined separately from the scenario.

  Further, in the present embodiment, the automatic execution function (process control function) and the teaching function are integrated into the internal architecture of the user interface operation integration system without being a separate independent program. As will be described later, the operation execution unit and the event message monitoring unit, the teaching unit and the process control unit are paired with the UI operation / monitoring IF providing unit and the business record / execution IF providing unit, respectively. Are connected by a common adapter type identification / distribution function.

  Furthermore, in this embodiment, as will be described later, the inside of the business record / execution IF providing unit is configured by modules (teaching module, replay module) that can be additionally expanded. Further, as a common function, a file saving / reading function, a work data object, and a work history (scenario) object are provided.

  FIG. 4 is a functional block diagram showing the system terminal apparatus 10 in the present embodiment. As shown in the figure, a computer program of Windows (registered trademark) OS (210) and operation GUI (220) is installed in the system terminal device 10 in addition to the integrated system 12, and a work data file is stored in the storage unit 230. 231 and a scenario file 232 are stored.

  The integrated system 12 includes a business record / execution IF providing unit 130, a UI operation / monitoring IF providing unit 140, and an adapter unit 150. The business record / execution IF providing unit 130 includes a teaching unit 131 and a process control unit 132, and the UI operation / monitoring IF providing unit 140 includes an operation execution unit 141 and an event message monitoring unit 142. The adapter unit 150 includes, for example, an Internet Explorer (hereinafter simply referred to as IE), a dot net framework (hereinafter simply referred to as .NET), a Microsoft Foundry provided in the Windows (registered trademark) OS (210). IE adapter 151,... For the day class (hereinafter simply referred to as MFC), mouse and keyboard. A NET adapter 152, an MFC adapter 153, a mouse adapter 154, and a keyboard adapter 155 are provided. Each adapter 151-155 of the adapter unit 150 can be connected to both the operation execution unit 141 and the event message monitoring unit 142, and other adapters can be additionally expanded to the adapter unit 150 as desired.

  The teaching unit 131 of the business record / execution IF providing unit 130 receives from the event message monitoring unit 142 a mouse and keyboard event and a UI element message when the operator 1 (user) is actually operating the target application. The contents are saved as a scenario file 232 and a work data file 231. The process control unit 132 reads the scenario file 232 and the work data file 231 and transmits an event or message to the operation execution unit 141 based on the contents.

  The event message monitoring unit 142 of the UI operation / monitoring IF providing unit 140 receives an instruction from the teaching unit 131, monitors an event for a specific window or component, and notifies the teaching unit 131 of the contents when the event occurs. The operation execution unit 141 receives data and execution content designation from the business record / execution IF providing unit 130 and executes a predetermined UI operation. These are indicated by two solid arrows in FIG. The other two dashed arrows represent auxiliary function calls, details of which will be described later.

  5 to 7 are flowcharts showing a procedure of processing executed by the system terminal apparatus 10 according to this embodiment.

  First, the procedure will be described.

  When the operator 1 starts up the system terminal device 10 (software is started) in the present embodiment (S101), a window holding the operation GUI 220 is displayed, and the system terminal device 10 performs the operation of the operator 1 on the “operation GUI 220”. Wait (S103). The operator 1 can change the internal setting and mode of the system terminal device 10 by using the teaching setting GUI 224, the replay setting GUI 223, and the operation mode change selector 222 of the operation GUI 220.

  First, the operator 1 operates the teaching setting GUI 224 (S103), specifies the window to be operated on the system terminal device 10 (starts and displays the application to be operated in advance), and sets the replay setting. The GUI 223 is operated (S105), the replay setting contents are changed, and the operation mode change selector 222 is operated to set the operation mode (S109). At this time, the event message monitoring unit 142 automatically determines the monitoring means to be used for teaching / replay. For example, when the GUI included in the designated window is developed on a GUI platform called “.NET”, the monitoring means is determined to be “.NET”. At this time, when it is desired to monitor / execute an input event with a mouse instead of directly monitoring / executing the GUI platform, the operator 1 can designate “mouse” as the monitoring means.

  When the operator 1 presses the start / stop button 221 (S107), the system terminal apparatus 10 starts work. At this time, if the operation mode is set to teaching (S111), teaching work is started. In teaching work, the teaching unit 131 of the system terminal apparatus 10 displays a confirmation dialog as to whether or not to initialize (empty) the work data object and the work scenario object, and performs processing according to the input of the operator 1 (S113). ). However, when it is desired to additionally insert teaching contents into previous teaching contents, the process can proceed to the next process without initialization. This is realized by allowing the operator 1 to make a decision through a confirmation dialog at the time of execution of this processing. Subsequently, the system terminal device 10 refers to “• monitoring means” and “• monitoring target window” in the contents of the teaching setting, and uses “• type identifier” and “• monitoring request content” as arguments to “event” A monitoring request is issued to the message monitoring unit 142 "(S115), and an event notification waiting state is entered.

  When the teaching unit 131 receives the notification in the event notification waiting state (S119), it first receives the type of the GUI to be operated from the adapter unit 150 of the event message monitoring unit 142, determines the type (S123), If it is a button (S125), operation content information (platform dependent) such as GUI ID and class name is stored in the work scenario object (S127), and the process proceeds to S117 and waits for event notification again. Become. On the other hand, if it is a text field (S129), at the time of the operation, the system terminal device 10 displays a dialog prompting the operator 1 to input an “input item ID” (unique character string that does not overlap with others). (S131). The operator 1 inputs a name (for example, “birth date”) desired to be given to the data input to the GUI element of the target window immediately before as the input item ID. The system terminal device 10 that has received it sets the input item ID and the input value as a set (for example, as a key and value of a hash table) and stores them in the work data object (S133). Further, as in the case of the button, operation content information such as GUI ID and class name is stored in the work scenario object (S135), and the process shifts to the processing of S117 and again enters the event notification waiting state.

  When the start / stop button 221 is pressed while waiting for an event notification, the teaching unit 131 writes the work data file 231 to the storage unit 230 based on the work data object (S137), and the scenario file based on the work scenario object. 232 is written in the storage unit 230 (S139), the process proceeds to the process of S103, and the process returns to the operation waiting state for the operation GUI 220.

  The operator 1 can cause the system terminal apparatus 10 to start a replay operation by changing the operation mode from teaching to replay and pressing the start / stop button 221. In the replay operation, when the process control unit 132 of the system terminal device 10 determines that the operation mode is replay by pressing the start / stop button 221 (S141), a predetermined work data file stored in the storage unit 230 231 and the scenario file 232 are read into the process control unit 132, and the process control unit 132 that has read them generates a work data object and a work scenario object corresponding to each of them (S143, S145). Then, the process control unit 132 interprets the func tags of the work scenario object in a predetermined order from the top, and transmits an execution request to the operation execution unit 141 (S147). If the next func tag of the work scenario object exists, the process control unit 132 interprets and executes the request (S149), and repeats this until there is no next func tag. Finally, when there is no next func tag, the process control unit 132 stops the scenario execution (S151), and the system terminal apparatus 10 proceeds to the process of S103 and returns to the operation waiting state of the “operation GUI 220”. .

  5 to 7, when the teaching operation is performed on the mouse event, the process of S123 “identify the type of the GUI to be operated” is replaced with another process. . In other words, the coordinates of the mouse coordinates are calculated based on the designated teaching setting contents, and then the processing is stored in the work scenario object.

  In addition, in FIG. 5 to FIG. 7, the acquisition of the value of the text field can be realized by the implementation method depending on the platform in the teaching module for each GUI platform. Furthermore, processing can be similarly implemented for setting and acquiring selections for GUI elements such as check buttons, list boxes, and check boxes.

  FIG. 8 is an internal configuration diagram of the teaching unit 131.

Implementation code teaching process for each UI type is incorporated as a teaching module 311, each time a UI type corresponding in the system is added, Teach ring module 311 is also created and added to the new. In addition, the UI type of the target application during teaching and object instances such as windows displayed during teaching are held as the type designation 321 and instance designation 322 of the teaching target designation function unit 320, respectively. These designations are designated via the teaching setting GUI 224 of the operation GUI 220 in FIG. The teaching module 311 corresponding to the type designation is activated, and the module 311 indirectly calls the adapter 150 in the event message monitoring unit 142, and Windows (registered trademark) 0S (210) (for example, Windows XP (registered trademark)) Monitor events and messages issued from). The obtained events and messages are stored in the work scenario object 340. Also, values such as input data and selection data accompanying the message are stored in the work data object 330. The work data object 330 and the work scenario object 340 are stored in the work data file 231 and the scenario file 232 in the storage unit 230 by the file storage function unit 350.

  FIG. 9 and FIG. 10 are specific examples of the format when the work history and work data stored in the stored work scenario object 340 and work data object 330 are saved in a file. FIG. 9 shows the work history stored in the scenario file, and FIG. 10 shows the work data stored in the work data file.

  A work history is stored in the scenario file 232. The work history is composed of a scenario part (XML scenario tag) and a function part (XML function tag). The period from the start to the end of the operation in the teaching mode is accumulated as a unit, and a function identifier is given. This is stored as the name attribute of the function tag in XML as in a specific example, for example. The function identifier is input and designated by the teaching operator 1 who performs teaching via the teaching setting GUI 224.

  Work data is stored in the work data file 231. The work data is accumulated when a value is input to the TextField in the UI operation, and a parameter identifier is given. This is stored, for example, as the name attribute of a param tag in XML as in a specific example. The work data is input and designated by the operator 1 via the child dialog window of the teaching setting GUI 224 or the like every time the operator 1 inputs a value into the UI.

  Note that mouse events may be recorded as a variation of FIGS. In this case, the coordinate value and mouse click type (right, left) in the work scenario object 340 are encoded into a character string using a comma and a semicolon, and are recorded in the file in the format of the func tag in FIGS. . At this time, by encoding a plurality of mouse events into one character string, it is possible to suppress an increase in the size of the scenario file 232 for a problem that tends to accumulate a large amount of low-level events. The keyboard event is recorded in the same manner by character-encoding the key type and the operation type (UP, DOWN).

  FIG. 11 is a functional block diagram illustrating an IF distribution method for the event message monitoring unit 142 as a distribution method of the UI operation / monitoring IF providing unit 140.

  The monitoring request from the teaching module 311 is interpreted by the adapter type identification / distribution function unit 421 through the common IF of the event message monitoring unit 142. Here, according to the adapter type identifier included in the request content, the teaching module 311 identifies which adapters 151 to 155 provided in the adapter unit 150 can make the request by the procedure shown in the sequence diagram of FIG. Specifically, the type identifier exchange procedure (611) is performed for each of the adapters 151 to 155 in order to make the adapters 151 to 155 return their own adapter type identifiers, and the corresponding adapters 151 to 155 are determined. Next, the teaching module 311 makes a monitoring request to the corresponding adapters 151 to 155 (612), and the adapters 151 to 155 that have received the request send an event notification to the teaching module 311 at that time (613).

  The “monitoring request content” here includes, for example, the following information.

(1) Whether to specify the range to be monitored Limit the range of events to be monitored to be unlimited (monitor all events on the terminal device), or to the event source being a specific window Select the mode.

(2) Specifying the range to be monitored If the range to be monitored is specified, the title character string of the window representing that range.

(3) Specifying the monitoring result recording method In the case of an event related to the mouse, the mouse click position is the relative position when the origin is the top right, bottom right, top left, or bottom left of the window (the desktop if no window is specified). Mode selection as well as the position coordinate unit (whether it is recorded as a pixel value or a ratio when the movable range is 0 to 100%).

  The contents of this monitoring request may vary depending on the receiving adapter 151-155. (1) (2) is information commonly referred to by most adapters 151-155, while (3) is information transmitted only to the mouse adapter 154 that handles mouse events.

  FIG. 13 is a diagram showing an internal configuration of the process control unit 132 that performs automatic execution based on the file created by the teaching unit 131.

  The created work data file 231 and scenario file 232 are expanded into a work data object 530 and a work scenario object 540 by the file reading function unit 550. When execution is started by pressing the start / stop button 221 of the operation GUI 220, the scenario interpretation function unit 520 sets the execution position pointer 521 in the first func tag in the scenario part in the work scenario object 540 and executes from there. To start. Based on the character string value of the name attribute of the func tag at the pointer position, search for the same name attribute value from the set of func tags under the function tag of the function part, and based on the attribute value of the obtained func tag Perform specific operations. At this time, which replay module 511 is requested to execute is determined according to the type attribute (UI platform type). Upon receiving the request, the replay module 511 refers to the content of the func tag, and calls the function of the operation execution unit 141 using the targetID (UI element instance ID) as an argument. At that time, if necessary, based on the param attribute (argument value), the argument value necessary for execution is obtained from the work data object 530, passed to the operation execution unit 141, and after execution is completed, the characters in the text field In the case of an operation with a return value such as column acquisition, the value is received and stored in the work data object 530. In the param attribute, a character string in the name attribute of the intended param tag in the work data object 530 is designated. By matching these character strings, the scenario and work data are linked. This character string is an “input item ID” in the processing of S131 and S133 shown in FIG.

  FIG. 14 is a functional block diagram for explaining an IF distribution method for the operation execution unit 141 as a distribution method of the UI operation / monitoring IF providing unit 140.

  The execution request from the play module 511 is interpreted by the adapter type identification / distribution function unit 411 through the common IF of the operation execution unit 141. Here, according to the adapter type identifier included in the request content, the replay module 511 specifies which adapters 151 to 155 provided in the adapter unit 150 can make the request by the procedure shown in the sequence diagram of FIG. Specifically, a type identifier exchange procedure (711) is performed for each adapter 151 to 155 in order to make each adapter 151 to 155 reply its own adapter type identifier, and the corresponding adapter 151 to 155 is determined. Next, the replay module 511 makes an execution request to the corresponding adapters 151 to 155 (712), and the adapters 151 to 155 that have received the request send an execution completion notification to the replay module 511 when the execution is completed (713).

  FIG. 16 is a diagram showing a function call relationship when know-how related to external control of the target application is implemented in the configuration of FIG.

  The know-how related to external control of the target application means that the OS operation and GUI platform message are generated in a pseudo manner from the execution process different from the target application when the operation is executed, and the UI operation is performed on behalf of the operator 1. It is a detailed peripheral process that is necessary in relation to execution from software. Specifically, the internal state of the target application is checked in advance to determine whether it is in the context where the event or message is accepted and executed as intended, or the actual execution result is completed and reflected as intended This refers to processing such as checking the status after the fact, or polling the check processing for each and waiting for a predetermined time for the internal state of the target application to transition.

Situations that require event / message monitoring in replay For example, when a data value is set in a text field, the text field may be disabled. Depending on the window system (GUI component library), when the data value setting operation is performed, the execution result is not returned and the processing exception does not take effect, so the consistency with the subsequent operation is lost and normal cooperation cannot be performed. . Therefore, in this embodiment, it is conceivable to improve the reliability of cooperation by combining three measures.
A) It is checked immediately before whether the operation target is valid, and if it is invalid, the process waits until it changes to valid.
B) Until the operation is actually finished, it is continuously monitored whether the change event of the disable / enable of the GUI component is activated.
C) After the operation is completed, a value is read from the GUI component to check whether the value set immediately before is actually reflected.

  Furthermore, in FIG. 2 and FIG. 3, in order to cooperate with the situation of the dialogue operation between the dummy AP or helper AP and the user, the above control function can be used indirectly. In other words, processing inside the AP is created so that specific text fields (GUI parts) in these APs are invalidated in advance and enabled if necessary information is input from the user, and disabled otherwise. Keep it. If the operation of reading the value of the text field at a desired timing location on the script of the present embodiment is taught, the dialogue operation between the dummy AP or helper AP and the user can be performed by using the countermeasure of the “A”. The processing can be made to wait in response to the situation. That is, the user's operation at these APs can be coordinated with an external operation on another system.

  When implementing such know-how, it is desirable that not only the operation execution unit 141 but also the function of the event message monitoring unit 142 can be called as appropriate. In this configuration, function calls between the business record / execution IF providing unit 130 and the UI operation / monitoring IF providing unit 140 are unified, and the necessary calls to the adapters 151-155 are performed by hard coding as shown in FIG. There is also a mechanism that can accommodate additional adapter expansion. Accordingly, various execution / monitoring means in the operation execution unit 141 and the event message monitoring unit 142 can be called from the replay module 511.

  Similar know-how may be necessary for teaching. This is a more active sensing function that not only monitors events but also performs operation execution behind the scenes without affecting the operation of the operator 1 and obtains the execution result to grasp the internal state of the operation target. It corresponds to. This makes it possible to determine a suitable operation type and record it in a scenario file even for a context hidden from the outside that is unknown only by passively monitoring an event.

For example situations requiring operations performed in teaching, such as third-party GUI components, the focus is deviated beginning began around the focus to it (reflected destination user keyboard input is the GUI component) Although the end point (the state has ended) can be known through event monitoring, it may not be known depending on the monitoring whether there is a change in the value set therein. In such a case, the value held by the GUI component is read at the start time when the focus starts to be read, and the value is read again at the end time when the focus is off, and the change event is indirectly detected by comparing the change before and after. It is conceivable to monitor.

  In FIG. 4, the teaching unit 131, the process control unit 132, the operation execution unit 141, and the event message monitoring unit 142 are connected to each other by four arrows, two of which are indicated by broken lines. This is because these two wavy arrows represent an auxiliary function call for implementing the above know-how.

  As described above, according to the present embodiment, by using a plurality of teaching modules 311 that can be additionally expanded, the types of monitoring information that can be used in the teaching unit 131 can be additionally expanded without significantly affecting the conventional processing. In addition, by using a plurality of replay modules 511 that can be additionally expanded, the types of execution means that can be used in the process control unit 132 can be additionally expanded without significantly affecting the conventional processing.

  Furthermore, according to the terminal device 10 of the UI operation integrated system of the present embodiment, the following operational effects can be obtained.

  -With this configuration, when defining the automatic execution processing of work that you want to perform in the user interface operation integration system, you can specify the target ID, type, and argument of the operation, and determine the execution order. There is no need to describe the know-how individually. As a result, data is not related to the processing logic (the target operation procedure and external control know-how are mixed together), which is equivalent to a variable in a programming language as in the past. As an argument, it is possible to hold data with high abstraction that conceals external control know-how. Thereby, it is possible to safely and simply implement data input work of different values by arbitrarily changing the contents of the work data. In addition, by creating work data and scenarios separately, it is possible to simplify the implementation of higher-level functions, such as changing the data contents from the outside and calling the functions.

  -With this configuration, multiple different abstract levels of automation, such as a method of automatically sending messages to UI elements to perform automatic operations and a method of automatically sending events to GUI platforms and OSs to perform automatic operations The operation can be executed by combining the operation means. As a result, even for GUI platforms and applications that cannot be partially supported by only one type of automatic operation method (or it takes time to find and implement the corresponding means), it is easy to take alternative means for the corresponding part, There are effects such as expansion of response range and improvement of responsiveness. The “abstract level” is an abstracted representation of an operation procedure obtained by stepwise decomposition and analysis of an input operation by the operator 1 and how the operation procedure is interpreted by a program. For example, in the case of a mouse button press, this is an abstraction of the operation procedure analyzed by analyzing input operations at different levels of interpretation, such as pressing and pressing the mouse button, clicking the mouse button, and double clicking the mouse button. .

  -With this configuration, not only the automatic execution function but also the teaching function can be provided. Teaching is a function that the user interface operation integration system automatically extracts and stores by automatically showing the contents to be automatically executed as an example of the operation of the operator 1 (user) itself. For this purpose, it is necessary to monitor an event or message in accordance with a desired automatic operation means (abstract level) and execute processing for extracting necessary information therefrom. These are realized by cooperation between the teaching unit and the event monitoring unit, and generation of a work data file and a scenario file.

  -With this configuration, a teaching function can be realized by referring to a plurality of types of monitoring information, and the types of monitoring information that can be used can be additionally expanded without significantly affecting conventional processing. That is, the monitoring function depending on the GUI platform that can be supported (additionally expanded) in the system is individually implemented as a pluggable (optionally connectable) adapter in the event message monitoring unit 142, and information for each abstract level. Extraction processing is individually implemented as a teaching module 311 that can be loaded into the teaching unit 131 as a loadable (can be read and executed as appropriate) and connected by the adapter type identification / distribution function 421 to increase mutual independence. There is.

  -With this configuration, automatic execution can be realized by combining multiple types of operation execution means and monitoring information, and the types of execution means and monitoring information that can be used can be further expanded without significantly affecting conventional processing. . In other words, in the execution of operations, it is necessary to create know-how related to external control of the target application. At that time, by using multiple types of execution means and monitoring information comprehensively, the opposing AP (application program) and GUI It becomes easy to achieve a high degree of stability with respect to manufacturing and a high degree of stability against the contingency of execution timing. However, if each replay module 511 handles the execution means and the monitoring information individually, a lot of similar execution codes are generated and the quality and maintainability of the program are deteriorated. The above feature of this configuration has an effect of eliminating these conflicting problems.

  In the present embodiment, the integrated system 12 is configured by a computer program. However, the integrated system 12 of the present embodiment can be realized by both a computer (hardware) and a computer program, and the computer program can be recorded on a recording medium. It goes without saying that computer programs can also be provided through a network.

  DESCRIPTION OF SYMBOLS 1 ... Operator (user), 10 ... System terminal device, 11 ... Application UI, 12 ... Integrated system, 15 ... Dummy UI, 16 ... Helper application, 17 ... Business logic, 20 ... Existing system, 130 ... Business record and execution IF Providing unit 131 ... Teaching unit 132 ... Process control unit 140 ... UI operation / monitoring IF providing unit 141 ... Operation executing unit 142 ... Event message monitoring unit 150 ... Adapter 151 ... IE adapter 152 .... NET adapter, 153 ... MFC adapter, 154 ... mouse adapter, 155 ... keyboard adapter, 210 ... Windows (registered trademark) OS, 220 ... operation GUI, 221 ... start / stop button, 222 ... operation mode change selector, 223 ... replay setting GUI, 224 ... Teaching setting GUI, 230 ... Storage unit, 231 ... Work data file, 232 ... Scenario file, 311 ... Teaching module, 320 ... Teaching target designation function unit, 321 ... Type designation, 322 ... Instance designation, 330 ... Work Data object, 340 ... Work scenario object, 350 ... File storage function unit, 411,421 ... Adapter type identification / distribution function unit, 511 ... Replay module, 520 ... Scenario interpretation function unit, 521 ... Execution position pointer, 530 ... Work data object, 540 ... Work scenario object, 550 ... File reading function part.

Claims (7)

A method for customizing a user interface operation integration system in a computer terminal device that operates a plurality of types of applications in an integrated manner,
A user interface operation / monitoring interface providing unit having an operation execution unit and an event message monitoring unit that can be connected to a plurality of adapters that do not depend on individual monitoring target applications and have a monitoring function; a teaching unit; and a process control unit. A business record / execution interface provider is provided in the internal architecture of the user interface operation integration system.
An adapter type identification / distribution function unit connects between the operation execution unit and the process control unit and between the event message monitoring unit and the teaching unit,
The teaching unit has a plurality of additional expandable teaching modules, and the process control unit has a plurality of additional expandable replay modules,
The event message monitoring unit receives an instruction from a predetermined teaching module of the teaching unit, monitors an event for a specific window or component, and notifies an event content or message content to the teaching module that has received the instruction when an event occurs. And steps to
The teaching module that has received the notification receives the event content or message content from the event message monitoring unit, and the work data file describing the execution content of the work based on the event content or the message content and the execution order of the work Saving the described scenario file in the storage unit;
A predetermined replay module reads the work data file and the scenario file stored in the storage unit, and transmits event content or message content to the operation execution unit based on the content of the work data file and the scenario file. Steps,
The operation execution unit receives event contents or message contents transmitted from the replay module, and executes a predetermined user interface operation;
A method for customizing a user interface operation integration system, comprising: adding a teaching module and a replay module corresponding to the target application in response to the addition of the target application. The event content is a content of an event of a mouse or a keyboard when a user actually operates a target application, and the message content is a message of a user interface element. How to customize the user interface operation integration system. In terminal devices that operate multiple types of applications in an integrated manner,
An event message monitoring unit that monitors the operation information of the target application by the user and can be connected to a plurality of adapters that do not depend on each monitoring target application and has a monitoring function;
A teaching unit that receives the operation information from the event message monitoring unit and stores the operation information in a scenario file and a work data file;
A process control unit that reads the operation information from the scenario file and the work data file, and transmits the read operation information to an operation execution unit;
An operation execution unit that executes an operation of the target application based on operation information received from the process control unit. The teaching unit is individually implemented as a teaching module that can arbitrarily read and execute an information extraction processing function for each abstract level,
The said event message monitoring part is provided with the adapter type identification and distribution function part which connects the said adapter and the said teaching module which can be further extended depending on a graphic user interface platform. Terminal device. The process control unit is individually implemented as a replay module that can arbitrarily read and execute an operation execution processing function for each abstract level,
The terminal according to claim 3, wherein the operation execution unit includes an adapter type identification / distribution function unit that connects the plurality of adapters that can be additionally expanded depending on a graphic user interface platform and the replay module. apparatus. A user interface operation / monitoring interface providing unit having an operation execution unit and an event message monitoring unit that can be connected to a plurality of adapters that do not depend on individual monitoring target applications and have a monitoring function; a teaching unit; and a process control unit. A business record / execution interface providing unit provided in the internal architecture of the user interface operation integration system, and an adapter between the operation execution unit and the process control unit and between the event message monitoring unit and the teaching unit. Connected by a type identification / distribution function unit, the teaching unit has a plurality of additional expandable teaching modules, and the process control unit has a plurality of additional replay modules that can be expanded. Comprising a user interface operation integrated system, a computer program of the user interface operating integrated system in a computer terminal device for operating a plurality of kinds of applications integrated manner,
A computer program comprising the steps according to claim 1.   The computer-readable recording medium which recorded the computer program of Claim 6.