WO2022195784A1 - 分類装置、分類方法、および、分類プログラム - Google Patents
分類装置、分類方法、および、分類プログラム Download PDFInfo
- Publication number
- WO2022195784A1 WO2022195784A1 PCT/JP2021/010932 JP2021010932W WO2022195784A1 WO 2022195784 A1 WO2022195784 A1 WO 2022195784A1 JP 2021010932 W JP2021010932 W JP 2021010932W WO 2022195784 A1 WO2022195784 A1 WO 2022195784A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- operation event
- classification
- unit
- captured image
- event
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 54
- 230000008859 change Effects 0.000 claims abstract description 9
- 238000000605 extraction Methods 0.000 claims description 13
- 239000000284 extract Substances 0.000 claims description 4
- 238000003860 storage Methods 0.000 description 36
- 230000008569 process Effects 0.000 description 29
- 238000012545 processing Methods 0.000 description 28
- 238000010586 diagram Methods 0.000 description 21
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000010365 information processing Effects 0.000 description 3
- 238000005520 cutting process Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004801 process automation Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 238000004040 coloring Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000011017 operating method Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present invention relates to a classification device, a classification method, and a classification program.
- the present invention has been made to solve the above-described problems of the prior art, and it is an object of the present invention to easily specify an operated GUI component and its type without depending on the application execution environment of a terminal device.
- the classification device of the present invention provides an acquisition unit that acquires captured images of an operation screen of a terminal device before and after an operation event occurs, and a generation unit that uses the captured image to generate, as a differential image, changes that occur on the operation screen before and after the occurrence of the operation event; and a classifying unit for classifying the types of GUI components operated in.
- a classification method of the present invention is a classification method executed by a classification device, comprising: an acquisition step of acquiring captured images of an operation screen before and after an operation event of a terminal device occurs; a generation step of generating, as a difference image, a change that occurred on the operation screen before and after the occurrence of the operation event, using the captured image obtained by the operation event; and a classification step of classifying the types of GUI parts operated by the event.
- the classification program of the present invention includes an acquisition step of acquiring captured images of an operation screen of a terminal device before and after an operation event has occurred; a generation step of generating, as a difference image, a change that occurred on an operation screen before and after the occurrence of a; and a classification step of classifying the .
- FIG. 1 is a block diagram showing the configuration of a classification device according to Embodiment 1.
- FIG. 2 is a diagram illustrating an example of a difference image generated from captured images before and after operation of a radio button GUI component.
- FIG. 3 is a diagram illustrating an example of a difference image generated from captured images before and after an operation of a check box GUI component.
- FIG. 4 is a diagram showing an example of a difference image generated from captured images before and after operation of a GUI component of a pull-down menu.
- FIG. 5 is a diagram illustrating an example of a difference image generated from captured images before and after operation of the GUI component of the text box.
- FIG. 6 is a diagram illustrating an example of a difference image generated from captured images before and after operation of a button GUI component.
- FIG. 7 is a diagram showing an example of a difference image generated from captured images of the entire screen before and after the operation of the GUI component.
- FIG. 8 is a diagram showing an example of variation of input data for a trained model.
- FIG. 9 is a diagram illustrating a process of classifying the types of operated GUI parts by inputting a captured image and a difference image into a trained model.
- 10 is a flowchart illustrating an example of processing for storing captured images for each operation event in the classification device according to Embodiment 1.
- FIG. 11 is a flowchart illustrating an example of processing for extracting an operation event for a GUI component from a captured image in the classification device according to Embodiment 1.
- FIG. 12 is a flowchart illustrating an example of processing for generating a differential image in the classification device according to Embodiment 1.
- FIG. 13 is a flowchart illustrating an example of processing for classifying GUI components from a captured image for each operation event in the classification device according to Embodiment 1.
- FIG. 14 is a diagram showing an example of the flow of processing for obtaining, inputting, and determining operation information in the classification device according to Embodiment 1.
- FIG. FIG. 15 is a diagram showing a computer executing a classification program.
- Embodiment 1 the configuration of the classification device according to Embodiment 1 and the processing flow of the classification device will be described in order, and finally the effects of Embodiment 1 will be described.
- FIG. 1 is a block diagram showing the configuration of a classification device according to Embodiment 1.
- FIG. 1 is a block diagram showing the configuration of a classification device according to Embodiment 1.
- the classification device 10 is connected to the terminal device 20 via a network (not shown), which may be wired or wireless.
- a network not shown
- the configuration shown in FIG. 1 is merely an example, and the specific configuration and the number of each device are not particularly limited.
- the terminal device 20 is an information processing device operated by an operator.
- the terminal device 20 is a desktop PC, a notebook PC, a tablet terminal, a mobile phone, a PDA, or the like.
- the classification device 10 has a communication section 11 , a control section 12 and a storage section 13 . Processing of each unit of the classification device 10 will be described below.
- the communication unit 11 controls communication regarding various information.
- the communication unit 11 controls communication regarding various types of information exchanged between the terminal device 20 and an information processing device connected via a network.
- the communication unit 11 receives, from the terminal device 20, operation event information regarding an operation event that occurs when a mouse or keyboard is operated.
- the operation event information is various kinds of information including, for example, operation event occurrence time (time), occurrence position, event type (mouse click, keyboard input), and cursor information.
- the storage unit 13 stores data and programs necessary for various processes by the control unit 12, and has a captured image storage unit 13a, a learned model storage unit 13b, and an operation log storage unit 13c.
- the storage unit 13 is a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk.
- the captured image storage unit 13a stores captured images acquired by the acquiring unit 12a described later at regular time intervals (for example, one second). For example, the captured image storage unit 13a stores the captured time (time) and the captured image in association with each other. Note that the captured image storage unit 13a may store a captured image of the entire operation screen, or may store a partially extracted captured image on the operation screen.
- the learned model storage unit 13b stores learned models that classify the types of GUI components operated by operation events.
- the learned model stored in the learned model storage unit 13b includes, for example, a captured image at or after the occurrence of an operation event, and a difference image showing changes that occurred on the operation screen before and after the occurrence of the operation event.
- the type of GUI component operated by the operation event is output.
- the data input to the trained model is not limited to the captured image and the difference image, and includes an image obtained by synthesizing the cursor image and the captured image, cursor information including values indicating the cursor state, etc. It's okay. It is assumed that the learned model stored in the learned model storage unit 13b has been learned in advance by an external device.
- the learned model stored in the learned model storage unit 13b is not limited to one learned by an external device, and may be one learned by the classification device 10, for example.
- the classification device 10 further includes a learning unit that performs machine learning, and the learning unit performs the above-described learning process in advance to generate a trained model.
- the operation log storage unit 13c stores the captured images saved in the captured image storage unit 13a by the acquisition unit 12a in association with the time of occurrence as captured images before, during, and after the occurrence of the operation event. For example, the operation log storage unit 13c links captured images before, during, and after the occurrence of an operation event, the difference image generated by the generation unit 12c, and the types of GUI components classified by the classification unit. memorize it. Further, the operation log storage unit 13c may associate and store a part of the operation event information including the cursor information and the occurrence position.
- the operation log storage unit 13c may store logs of all operation events performed on the terminal device 20, or may store only logs of predetermined operation events.
- the operation log storage unit 13c stores not only operation logs of operation events related to a specific business system, but also operation event logs of business operations using various applications such as email, web browsers, and Office applications such as Word, Excel, and PowerPoint. may be stored at the same time, or a log may be stored for each operation event of a single application.
- the control unit 12 has an internal memory for storing programs defining various processing procedures and required data, and executes various processing using these.
- the control unit 12 has an acquisition unit 12a, an extraction unit 12b, a generation unit 12c, and a classification unit 12d.
- the control unit 12 is an electronic circuit such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).
- the acquisition unit 12a acquires captured images of the operation screen before and after the operation event of the terminal device occurs. For example, the acquiring unit 12a periodically acquires captured images at regular intervals, and stores the acquired captured images in the captured image storage unit 13a.
- the acquisition unit 12a may acquire three types of captured images from the captured image storage unit 13a at the timing when the operator's operation event occurs, when the operation event occurs, before the occurrence, and after the occurrence.
- the acquisition unit 12a may acquire three types of captured images from the captured image storage unit 13a at the timing when the operator's operation event occurs, when the operation event occurs, before the occurrence, and after the occurrence.
- three types of captured images are acquired when an operation event occurs, before the occurrence, and after the occurrence will be described as a main example.
- the acquisition unit 12a acquires a captured image at regular time intervals regardless of whether an operation event has occurred, as a method for acquiring a capture before an operation event occurs.
- the captured image acquired before the occurrence of the operation event (predetermined time before) is stored in the operation log storage unit 13c as the captured image before the occurrence of the operation event.
- the acquisition unit 12a acquires a capture image after a certain period of time has elapsed after the occurrence of the operation event, and acquires the operation log storage unit 13c as a capture after the occurrence of the operation event.
- the acquisition unit 12a may compare the acquisition time of the capture acquired at regular intervals with the time of occurrence of the operation event, and later link the images as captured images when the operation event occurs, before the occurrence, and after the occurrence of the operation event. good.
- the acquisition unit 12a may also acquire information on the cursor displayed on the operation screen, and use the information on the cursor to specify the shape of the cursor. For example, the acquisition unit 12a acquires a cursor handle when an operation event occurs, and compares it with a predefined cursor handle to identify the shape of the cursor.
- the acquisition unit 12a acquires the event occurrence time, the event occurrence position, and the event type for the user's operation event. For example, when an operation event occurs, the acquisition unit 12a acquires from the terminal device 20 event type information that specifies the operation content such as a click operation or key input, and information on the occurrence time of the operation event. Further, for example, when a click operation is performed, the acquisition unit 12a may acquire information on the position where the operation event occurs, and when key input is performed, the information on the operated key may be acquired. You may get the type information.
- the extraction unit 12b compares the captured image before and after the occurrence of the operation event, and extracts the operation event if there is a difference. For example, the extracting unit 12b compares captured images at the time, before, and after the occurrence of a given operation event, and if there is a difference in any of them, extracts the operation event. Extract as an operation event that may have had a meaningful operation.
- an operation event in which a meaningful operation may have been performed hereinafter referred to as a “meaningful operation event” means an operation event in which an operation on a GUI part may have been performed. do.
- the extraction unit 12b may use a captured image of the entire screen as the captured image to be compared, or may use an image obtained by cutting out the vicinity of the position where the operation event occurs from the captured image.
- the generation unit 12c uses the captured image acquired by the acquisition unit 12a to generate, as a difference image, changes that occurred on the operation screen before and after the operation event occurred. Specifically, the generation unit 12c generates, as a difference image, changes that occur on the operation screen before and after the operation event extracted by the extraction unit 12b.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation event determined to be the operation event for the GUI component by the extraction unit 12b, and converts the absolute value of the difference into image data. By doing so, a difference image representing the difference as an image is generated.
- FIGS. 2 to 6 examples of difference images generated from captured images before and after operation of GUI parts are shown. 2 to 6 respectively illustrate the case of a captured image without a cursor and the case of a captured image with a cursor.
- FIG. 2 is a diagram showing an example of a difference image generated from captured images before and after operation of a radio button GUI component.
- the acquisition unit 12a acquires a pre-operation captured image in which a radio button labeled “number of transfers” is checked as a pre-operation captured image.
- the acquisition unit 12a removes the check mark from the radio button labeled "Number of transfers” and removes the check mark from the radio button labeled "Fare ” radio button is checked to obtain a captured image after the operation.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation, converts the absolute value of the difference into image data, and generates a difference image including the round marks of the two radio buttons. Generate.
- FIG. 3 is a diagram showing an example of a difference image generated from captured images before and after operation of a check box GUI component.
- the acquiring unit 12a acquires a captured image in which the edge of the check box labeled “Express Line” is displayed with a thick border, as a captured image before the operation.
- the acquisition unit 12a removes the check mark from the check box labeled "express train”, Acquire a captured image in which the edge of the check box labeled "route bus” is displayed with a thick frame and a check mark is displayed.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation, converts the absolute value of the difference into image data, and converts the square edge of the two check boxes and the check mark of "route bus" into image data. Generate a difference image containing
- FIG. 4 is a diagram showing an example of a difference image generated from the captured images before and after the operation of the GUI component of the pull-down menu.
- the acquisition unit 12a acquires a captured image in which a pull-down menu labeled “2019” is selected as a pre-operation captured image.
- the acquisition unit 12a selects the pull-down menu labeled "2019”.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation, and converts the absolute value of the difference into image data, thereby creating a pull-down menu labeled “2019”, Then, a difference image is generated that includes a pull-down menu labeled "November” and selections for all other months.
- FIG. 5 is a diagram showing an example of a difference image generated from the captured images before and after the operation of the text box GUI component.
- the acquiring unit 12a acquires a captured image in which a GUI component of a text box labeled "web search" is displayed as a pre-operation captured image.
- the acquisition unit 12a causes the characters in the text box labeled "web search” to disappear. , to obtain a captured image after the operation in which the cursor is displayed on the GUI component of the text box.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation, and converts the absolute value of the difference into image data to generate the characters of the text "web search". , and the cursor displayed in the text box GUI component.
- FIG. 6 is a diagram showing an example of a difference image generated from the captured images before and after the button GUI component is operated.
- the acquiring unit 12a acquires a captured image in which an "OK" button is displayed on a tab labeled "Arriving station not found” as a captured image before operation.
- the acquisition unit 12a removes the tab labeled "Arriving station not found.” and displays the original screen. Get the captured image after the operation.
- the generation unit 12c calculates the difference between the pixel values of the captured image before and after the operation, and converts the absolute value of the difference into image data, so that "arrival station not found.” generates a difference image that includes the tabs that are hidden by the tabs and the original screen that was hidden by the tabs.
- FIG. 7 is a diagram showing an example of a difference image generated from captured images of the entire screen before and after the operation of a GUI component.
- a case was described in which an image obtained by cutting out the vicinity of the position where the operation event occurred from the captured image was used.
- the images may be used to generate a difference image.
- the classification unit 12d classifies the types of GUI components operated by operation events using the difference image generated by the generation unit 12c.
- the classification unit 12d classifies the types of GUI components and determines whether or not the event is a meaningful operation event.
- the GUI parts that can be classified include, for example, "radio button”, “check box”, “pull-down menu”, “text box”, and “button”. and "link”.
- the classification unit 12d classifies it as "not a meaningful operation event”.
- the classification unit 12d receives the captured image and cursor information acquired by the acquisition unit 12a, and the difference image generated by the generation unit 12c, and classifies the type of GUI component operated by the operation event.
- a trained model may be used to classify the types of GUI components operated by each operation event.
- This trained model is a trained model stored in the trained model storage unit 13b, and is learned using a predetermined machine learning algorithm with the relationship between input data and operated GUI parts as teacher data. It is a finished model.
- classification unit 12d classifies the type of GUI component operated by the operation event using the difference image generated by the generation unit 12c and the shape of the cursor specified by the acquisition unit 12a.
- the classification unit 12d may use information about operation events performed before and after the operation event targeted for classification for classification.
- the target operation event is a mouse click intended to focus on a text box
- the next operation event is likely to be a key input such as a character. Therefore, by using the information that the next operation event is a key input, it is expected that the classification accuracy of the target operation event will be improved.
- the classification unit 12d inputs the operation event performed before and after the operation event in addition to the captured image and the difference image to the learned model. Classify the types of GUI parts.
- the classification unit 12d may use the window identification information for classification. For example, if the target operation event is link depression, there is a high possibility that the operation event will cause a page transition. Therefore, if the acquisition unit 12a can obtain window identification information indicating that a page transition has occurred after an operation event, the use of the window identification information can be expected to improve the classification accuracy of target operation events. In this case, the classifying unit 12d classifies the type of GUI component operated by the operation event by inputting the identification information of the window in addition to the captured image and the difference image to the learned model. do.
- FIG. 8 is a diagram showing an example of variation of input data for a trained model.
- the classification unit 12d inputs the captured image acquired by the acquisition unit 12a and the difference image generated by the generation unit 12c to the trained model.
- the classification unit 12d may input a cursor image in addition to the captured image and the differential image to the learned model.
- the classification unit 12d may input information on the shape of the cursor specified by the acquisition unit 12a to the learned model. Further, the classification unit 12d may input the information of the occurrence position of the operation event acquired by the acquisition unit 12a to the trained model, in addition to the captured image and the difference image.
- FIG. 9 is a diagram illustrating a process of classifying the types of operated GUI parts by inputting a captured image and a difference image into a trained model.
- the CNN has a hierarchical structure and includes a convolutional layer, a pooling layer, a fully connected layer, and an output layer.
- an external device that performs learning can robustly acquire the type of an unknown GUI component from limited learning data. You may use Dropout to deactivate some of the nodes in the layer.
- an external device that performs learning can generate a learning model with high classification accuracy by using a trained model for another related task.
- the external device that performs learning acquires the types of unknown GUI parts robustly from limited learning data, when learning the relationship between input data and operated GUI parts, the GUI Transfer learning or fine tuning may be performed using a model that has learned in advance the relationship between component images and the types of GUI components.
- FIG. 10 is a flowchart illustrating an example of processing for storing captured images for each operation event in the classification device according to Embodiment 1.
- FIG. 10 is a flowchart illustrating an example of processing for storing captured images for each operation event in the classification device according to Embodiment 1.
- the acquisition unit 12a determines whether the user has stopped processing or turned off the terminal device 20 (step S101). As a result, when the acquiring unit 12a determines that the operator has stopped the process or turned off the power of the terminal device (step S101 Yes), the process of this flow ends. Further, when the acquiring unit 12a determines that the operator has not stopped the processing and has not turned off the power of the terminal device 20 (step S101 No), the captured image storage unit 13a stores the captured image at regular intervals. (step S102).
- the acquisition unit 12a determines whether or not an operation event has occurred (step S103). As a result, if an operation event has occurred (step S103 Yes), the acquisition unit 12a acquires operation event information (step S104). For example, the acquisition unit 12a acquires the event occurrence time, the event occurrence position, and the event type for the user's operation event, and stores them in the operation log storage unit 13c in association with the captured image at the time of the event occurrence. Also, if no operation event has occurred (step S103 NO), the process returns to step S101.
- the acquiring unit 12a acquires a pre-occurrence captured image of the operation event based on the occurrence time from the captured images temporarily stored in the captured image storage unit 13a in step S102 (step S105). Subsequently, the acquiring unit 12a acquires a captured image as a captured image after occurrence of an operation event after a certain period of time has elapsed (step S106). Then, the acquisition unit 12a associates the captured images before, during, and after the occurrence of the operation event based on the occurrence time from the acquired captured images, and stores them in the operation log storage unit 13c (step S107). After that, the process returns to step S101 and repeats the above process flow.
- the acquisition unit 12a may store the captured image in the classification device according to the first embodiment by linking the captured image to the operation event later.
- the acquisition unit 12a independently acquires a captured image and acquires an operation event, and after accumulating a certain amount of captured image data, associates the operation event with the captured image based on the operation event occurrence time.
- FIG. 11 is a flowchart illustrating an example of processing for extracting an operation event for a GUI component from a captured image in the classification device according to Embodiment 1.
- FIG. 11 is a flowchart illustrating an example of processing for extracting an operation event for a GUI component from a captured image in the classification device according to Embodiment 1.
- the extraction unit 12b determines whether all operation events are targeted (step S201). As a result, when the extraction unit 12b determines that all the operation events are targets (step S201 Yes), the process of this flow is terminated. In addition, when not all the operation events are targeted (step S201 No), the extraction unit 12b determines the operation events to be targeted (step S202).
- the extraction unit 12b determines whether there is a difference in any of the captured images when the operation event occurs, before the occurrence, or after the occurrence (step S203). As a result, when the extracting unit 12b determines that there is no difference in any of the captured images before and after the occurrence of the operation event (step S203 No), the process returns to step S201.
- step S203 determines that there is a difference in any of the captured images before and after the occurrence of the operation event.
- the extraction unit 12b determines that the target operation event is a meaningful operation. (step S204). After that, the process returns to step S201 and repeats the above process flow.
- FIG. 12 is a flowchart illustrating an example of processing for generating a differential image in the classification device according to Embodiment 1.
- FIG. 12 is a flowchart illustrating an example of processing for generating a differential image in the classification device according to Embodiment 1.
- the generation unit 12c determines whether all the operation events are targeted (step S301). As a result, when the generation unit 12c determines that all the operation events are targets (step S301 Yes), the process of this flow is terminated. Further, when not all the operation events are targeted (step S301 No), the generation unit 12c determines the operation events to be targeted (step S302).
- the generation unit 12c determines whether or not the target operation event is an operation event extracted as a meaningful operation event (step S303). As a result, if the operation event is not extracted as a meaningful operation event (step S303 No), the generating unit 12c returns to the process of step S301.
- step S303 Yes when the generation unit 12c determines that the target operation event is an operation event extracted as a meaningful operation event (step S303 Yes), when the operation event occurs, before the occurrence, and after the occurrence From the captured image, the difference occurring on the screen is generated as an image (step S304). For example, when an operation event occurs, the generation unit 12c calculates the difference between the pixel values of the captured image before and after the occurrence, and converts the absolute value of the difference into image data to generate a difference image. After that, the process returns to step S301 and repeats the above process flow.
- FIG. 13 is a flowchart illustrating an example of processing for classifying GUI components from a captured image for each operation event in the classification device according to Embodiment 1.
- FIG. 13 is a flowchart illustrating an example of processing for classifying GUI components from a captured image for each operation event in the classification device according to Embodiment 1.
- the classification unit 12d determines whether all operation events are targeted (step S401). As a result, when the classification unit 12d determines that all the operation events are targets (step S401 Yes), the processing of this flow ends. Further, when not all the operation events are targeted (step S401 No), the classification unit 12d determines the operation event to be targeted (step S402).
- the classification unit 12d determines whether or not the target operation event is an operation event extracted as a meaningful operation event (step S403). As a result, if the operation event is not extracted as a meaningful operation event (step S403 No), the classification unit 12d returns to the process of step S401.
- the classification unit 12d determines that the target operation event is an operation event extracted as a meaningful operation event (step S403 Yes)
- the captured image, difference image, cursor shape, operation event occurrence Using the information such as location, the types of operated GUI parts are classified (step S404).
- the classification unit 12d classifies an operation event that does not correspond to a meaningful operation on a GUI component into the category of "no meaningful operation event". After that, the process returns to step S401 and repeats the above process flow.
- the classification device 10 acquires captured images of the operation screen before and after the operation event of the terminal device 20 occurs. Then, the classification device 10 uses the acquired captured image to generate, as a difference image, changes that occurred on the operation screen before and after the operation event occurred. Subsequently, the classification device 10 classifies the type of the operated GUI component using the generated difference image. As a result, the classification device 10 can easily identify the manipulated GUI component and its type without depending on the application execution environment of the terminal device 20 .
- the GUI part where the operation was performed is specified and the type thereof is determined using the changed operation location and the appearance of the operation location. It is possible.
- the shape of the GUI component changes when the cursor is placed on it, the shape of the GUI component changes when the mouse is down, or the screen is displayed after clicking. It is possible to specify the operated GUI component and determine its type by using the operation location and the appearance of the operation location where there was a change in the screen difference that occurred before and after the operation event. is.
- a standard arrow is used as the shape when the cursor is in a place where no GUI component exists, or an I
- the shape of a beam or when the cursor is on a button is the shape of a hand with a finger raised. It is possible to identify and determine its type.
- the capture image acquired by the acquisition unit 12a and the difference image generated by the generation unit 12c are input, and the type of GUI component operated by the operation event is determined.
- the classification device 10 or an external device learns features common to GUI parts for a trained model, and from limited learning data, when GUI parts change or unknown GUI parts On the other hand, it becomes possible to acquire the type robustly.
- the classification device 10 can collect reference data for creating an RPA scenario and improving the scenario by identifying the type of GUI component that has been operated.
- RPA Robot Process Automation
- the operating procedures for the system to provide the same services and products are determined for each service and product, and the person in charge is in charge according to manuals, etc. shared with others.
- GUI component attribute values differs for each application, in order to comprehensively grasp the work execution status of the person in charge, it is necessary to acquire the GUI component attribute values according to the execution environment of all the applications used in the work. Although it is necessary to develop a mechanism for identifying the acquisition and change points of the information, the development cost is actually very high and it is not realistic. Even if a specific application is developed, if the specification changes due to the version upgrade of the target application, there is a problem that it is necessary to modify it accordingly.
- the classification device 10 of the present embodiment acquires only screen captures and mouse/keyboard operation information by specifying an operation log using a captured image of the operation screen of the terminal device 20. It can be used even in environments where it is not possible. In addition, even if different browsers, websites, and applications are used for each terminal device 20, it is possible to determine even unknown data by having the CNN learn the captured image and the difference image. Therefore, in the classification device 10 of the present embodiment, regardless of the execution environment of the application of the terminal device 20, it is possible to acquire the type of GUI component of the operation event by the operator and the flow of the operation in general.
- GUI part by specifying the type of the operated GUI part, it is possible to visualize the number of operations for each GUI part operated by the user, such as text boxes and radio buttons. This can be used as a reference for improving the design of the system, such as changing to a select box when there are many text box inputs and can be patterned.
- each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated.
- the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.
- each processing function performed by each device may be implemented in whole or in part by a CPU and a program analyzed and executed by the CPU, or implemented as hardware based on wired logic.
- program It is also possible to create a program in which the processing executed by the classification device 10 according to the above embodiment is described in a computer-executable language. In this case, the same effects as those of the above embodiments can be obtained by having the computer execute the program. Further, such a program may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read by a computer and executed to realize processing similar to that of the above embodiments.
- An example of a computer that executes a classification program that implements the same functions as the classification device 10 will be described below.
- FIG. 15 is a diagram showing a computer that executes a classification program.
- computer 1000 has memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070, for example. These units are connected by a bus 1080 .
- the memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 .
- the ROM 1011 stores a boot program such as BIOS (Basic Input Output System).
- BIOS Basic Input Output System
- Hard disk drive interface 1030 is connected to hard disk drive 1090 .
- Disk drive interface 1040 is connected to disk drive 1041 .
- a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041, for example.
- a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050, for example.
- a display 1130 is connected to the video adapter 1060 .
- the hard disk drive 1090 stores an OS 1091, application programs 1092, program modules 1093 and program data 1094, for example.
- Each table described in the above embodiment is stored in the hard disk drive 1090 or the memory 1010, for example.
- the classification program is stored in the hard disk drive 1090 as a program module in which commands to be executed by the computer 1000 are written, for example.
- the hard disk drive 1090 stores a program module 1093 that describes each process executed by the classification device 10 described in the above embodiment.
- data used for information processing by the classification program is stored in, for example, the hard disk drive 1090 as program data.
- CPU 1020 reads out program module 1093 and program data 1094 stored in hard disk drive 1090 to RAM 1012 as necessary, and executes each procedure described above.
- program module 1093 and program data 1094 related to the classification program are not limited to being stored in the hard disk drive 1090.
- they may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like.
- the program module 1093 and program data 1094 related to the classification program are stored in another computer connected via a network such as a LAN (Local Area Network) or WAN (Wide Area Network), and via the network interface 1070 It may be read by CPU 1020 .
- LAN Local Area Network
- WAN Wide Area Network
- classification device 11 communication unit 12 control unit 12a acquisition unit 12b extraction unit 12c generation unit 12d classification unit 13 storage unit 13a captured image storage unit 13b learned model storage unit 13c operation log storage unit 20 terminal device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- User Interface Of Digital Computer (AREA)
- Debugging And Monitoring (AREA)
Abstract
Description
[実施の形態1]
まず、最初に、図1を用いて、実施の形態1に係る分類装置の構成を説明する。図1は、実施の形態1に係る分類装置の構成を示すブロック図である。
次に、図10~図13を用いて、第1の実施形態に係る分類装置1による処理手順の例を説明する。
このように、実施の形態1に係る分類装置10は、端末装置20の操作イベントの発生前および発生後の操作画面のキャプチャ画像を取得する。そして、分類装置10は、取得したキャプチャ画像を用いて、操作イベントの発生前と発生後において操作画面に起こった変化を差分画像として生成する。続いて、分類装置10は、生成した差分画像を用いて、操作が行われたGUI部品の種類を分類する。これにより、分類装置10は、端末装置20のアプリケーションの実行環境に依らず、操作されたGUI部品とその種類を容易に特定することができる。
また、図示した各装置の各構成要素は機能概念的なものであり、必ずしも物理的に図示の如く構成されていることを要しない。すなわち、各装置の分散・統合の具体的形態は図示のものに限られず、その全部または一部を、各種の負荷や使用状況などに応じて、任意の単位で機能的または物理的に分散・統合して構成することができる。さらに、各装置にて行なわれる各処理機能は、その全部または任意の一部が、CPUおよび当該CPUにて解析実行されるプログラムにて実現され、あるいは、ワイヤードロジックによるハードウェアとして実現され得る。
また、上記実施形態に係る分類装置10が実行する処理をコンピュータが実行可能な言語で記述したプログラムを作成することもできる。この場合、コンピュータがプログラムを実行することにより、上記実施形態と同様の効果を得ることができる。さらに、かかるプログラムをコンピュータ読み取り可能な記録媒体に記録して、この記録媒体に記録されたプログラムをコンピュータに読み込ませて実行することにより上記実施形態と同様の処理を実現してもよい。以下に、分類装置10と同様の機能を実現する分類プログラムを実行するコンピュータの一例を説明する。
11 通信部
12 制御部
12a 取得部
12b 抽出部
12c 生成部
12d 分類部
13 記憶部
13a キャプチャ画像記憶部
13b 学習済みモデル記憶部
13c 操作ログ記憶部
20 端末装置
Claims (6)
- 端末装置の操作イベントの発生前および発生後の操作画面のキャプチャ画像を取得する取得部と、
前記取得部によって取得された前記キャプチャ画像を用いて、前記操作イベントの発生前と発生後において操作画面に起こった変化を差分画像として生成する生成部と、
前記生成部によって生成された前記差分画像を用いて、前記操作イベントで操作が行われたGUI部品の種類を分類する分類部と、
を備えることを特徴とする分類装置。 - 前記取得部は、前記キャプチャ画像を取得するとともに、操作画面上に表示されるカーソルの情報を取得し、該カーソルの情報を用いてカーソルの形状を特定し、
前記分類部は、前記生成部によって生成された前記差分画像と、前記取得部によって特定されたカーソルの形状とを用いて、前記操作イベントで操作が行われたGUI部品の種類を分類することを特徴とする請求項1に記載の分類装置。 - 前記分類部は、前記取得部によって取得された前記キャプチャ画像と前記生成部によって生成された前記差分画像とを入力として、前記操作イベントで操作が行われたGUI部品の種類を分類する学習済みモデルを用いて、各操作イベントで操作が行われたGUI部品の種類を分類することを特徴とする請求項1に記載の分類装置。
- 前記操作イベントの発生前のキャプチャ画像と発生後のキャプチャ画像とを比較し、差分が発生している場合には、当該操作イベントを抽出する抽出部をさらに有し、
前記生成部は、前記抽出部によって抽出された操作イベントの発生前と発生後において操作画面に起こった変化を差分画像として生成することを特徴とする請求項1に記載の分類装置。 - 分類装置が実行する分類方法であって、
端末装置の操作イベントの発生前および発生後の操作画面のキャプチャ画像を取得する取得工程と、
前記取得工程によって取得された前記キャプチャ画像を用いて、前記操作イベントの発生前と発生後において操作画面に起こった変化を差分画像として生成する生成工程と、
前記生成工程によって生成された前記差分画像を用いて、前記操作イベントで操作が行われたGUI部品の種類を分類する分類工程と
を含んだことを特徴とする分類方法。 - 端末装置の操作イベントの発生前および発生後の操作画面のキャプチャ画像を取得する取得ステップと、
前記取得ステップによって取得された前記キャプチャ画像を用いて、前記操作イベントの発生前と発生後において操作画面に起こった変化を差分画像として生成する生成ステップと、
前記生成ステップによって生成された前記差分画像を用いて、前記操作イベントで操作が行われたGUI部品の種類を分類する分類ステップと
をコンピュータに実行させることを特徴とする分類プログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2021/010932 WO2022195784A1 (ja) | 2021-03-17 | 2021-03-17 | 分類装置、分類方法、および、分類プログラム |
US18/281,641 US20240153241A1 (en) | 2021-03-17 | 2021-03-17 | Classification device, classification method, and classification program |
JP2023506610A JP7517590B2 (ja) | 2021-03-17 | 2021-03-17 | 分類装置、分類方法、および、分類プログラム |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2021/010932 WO2022195784A1 (ja) | 2021-03-17 | 2021-03-17 | 分類装置、分類方法、および、分類プログラム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022195784A1 true WO2022195784A1 (ja) | 2022-09-22 |
Family
ID=83322055
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/010932 WO2022195784A1 (ja) | 2021-03-17 | 2021-03-17 | 分類装置、分類方法、および、分類プログラム |
Country Status (3)
Country | Link |
---|---|
US (1) | US20240153241A1 (ja) |
JP (1) | JP7517590B2 (ja) |
WO (1) | WO2022195784A1 (ja) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010073099A (ja) * | 2008-09-22 | 2010-04-02 | Nec Corp | 評価装置、評価方法およびプログラム |
JP2011159018A (ja) * | 2010-01-29 | 2011-08-18 | Fujitsu Ltd | 情報処理装置、情報処理プログラム、および情報処理方法 |
JP2017162120A (ja) * | 2016-03-08 | 2017-09-14 | 三菱電機株式会社 | 情報処理装置、情報処理方法及び情報処理プログラム |
WO2020250320A1 (ja) * | 2019-06-11 | 2020-12-17 | 日本電信電話株式会社 | 操作ログ取得装置、操作ログ取得方法および操作ログ取得プログラム |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2021128402A (ja) | 2020-02-12 | 2021-09-02 | 富士通株式会社 | 摂動プログラム、摂動方法、および情報処理装置 |
-
2021
- 2021-03-17 US US18/281,641 patent/US20240153241A1/en active Pending
- 2021-03-17 WO PCT/JP2021/010932 patent/WO2022195784A1/ja active Application Filing
- 2021-03-17 JP JP2023506610A patent/JP7517590B2/ja active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010073099A (ja) * | 2008-09-22 | 2010-04-02 | Nec Corp | 評価装置、評価方法およびプログラム |
JP2011159018A (ja) * | 2010-01-29 | 2011-08-18 | Fujitsu Ltd | 情報処理装置、情報処理プログラム、および情報処理方法 |
JP2017162120A (ja) * | 2016-03-08 | 2017-09-14 | 三菱電機株式会社 | 情報処理装置、情報処理方法及び情報処理プログラム |
WO2020250320A1 (ja) * | 2019-06-11 | 2020-12-17 | 日本電信電話株式会社 | 操作ログ取得装置、操作ログ取得方法および操作ログ取得プログラム |
Also Published As
Publication number | Publication date |
---|---|
JP7517590B2 (ja) | 2024-07-17 |
JPWO2022195784A1 (ja) | 2022-09-22 |
US20240153241A1 (en) | 2024-05-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7398068B2 (ja) | ソフトウェアテスト | |
CN106290378B (zh) | 缺陷分类方法和缺陷检查系统 | |
WO2021120186A1 (zh) | 分布式产品缺陷分析系统、方法及计算机可读存储介质 | |
US10810018B2 (en) | Device with extensibility | |
CN105320589B (zh) | 云测试环境中测试脚本自动解析系统及其实现方法 | |
CN110442510A (zh) | 一种页面属性获取方法、装置及计算机设备、存储介质 | |
US20060020931A1 (en) | Method and apparatus for managing complex processes | |
KR102543064B1 (ko) | Rpa 기반 제조환경 모니터링 서비스 제공 시스템 | |
US20220350857A1 (en) | Methods and systems for browser extension used to generate relative xpath, absolute xpath and/or css selectors | |
WO2021140594A1 (ja) | 操作ログ取得装置および操作ログ取得方法 | |
CN112506778A (zh) | Web用户界面自动化测试方法、装置、设备和存储介质 | |
US11615016B2 (en) | System and method for executing a test case | |
Jaganeshwari et al. | an Automated Testing Tool Based on Graphical User Interface With Exploratory Behavioural Analysis | |
WO2021087818A1 (zh) | 软件知识捕捉方法、装置和系统 | |
WO2022195784A1 (ja) | 分類装置、分類方法、および、分類プログラム | |
CN112667517A (zh) | 自动化测试脚本的获取方法、装置、设备及存储介质 | |
CN105303082B (zh) | 伺服器、使用者装置及终端装置 | |
CN115035044A (zh) | 一种应用于工业质检的智能ai平台 | |
Korstanje | Machine Learning for Streaming Data with Python: Rapidly build practical online machine learning solutions using River and other top key frameworks | |
CN110309047B (zh) | 一种测试点生成方法、装置及系统 | |
CN115917446A (zh) | 用于机器人过程自动化的系统和方法 | |
US20230111999A1 (en) | Method and system of creating clusters for feedback data | |
Sarakiotis | Human-centered AI: Challenges and opportunities | |
Novaliendry et al. | Development of a Digital Twin Prototype for Industrial Manufacturing Monitoring System Using IoT and Augmented Reality. | |
CN115495057B (zh) | 实现windows和HDFS通信的方法和系统 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21931535 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023506610 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18281641 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21931535 Country of ref document: EP Kind code of ref document: A1 |