JP2019197504A - Information processing apparatus and information processing program - Google Patents

Abstract

To provide an information processing apparatus and an information processing program that can easily collect images of an object application without correcting the object application.SOLUTION: An information processing apparatus according to one embodiment comprises: a processor; an operation interface connected to an operating member; and a memory. The memory stores a device driver that causes the processor to perform detection of an event from a signal input to the operation interface, an application that causes the processor to make a transition of a screen on the basis of the event, and a screen information collection program for causing the processor to execute generating first image data on the basis of a screen of the application before the transition based on the event, and generating second image data on the basis of the screen of the application after the transition based on the event.SELECTED DRAWING: Figure 2

Description

  Embodiments described herein relate generally to an information processing apparatus and an information processing program.

  An automatic operation script that causes an application to execute a predetermined process by recognizing a screen displayed in the application has been put into practical use. In the automatic operation script, an image (determination image) is associated with a process to be executed by an application. The information processing apparatus that executes the automatic operation script and the application causes the application to execute processing associated with the determination image when an image that matches the determination image is displayed in the application. Such an automatic operation script is used for checking the operation of an application.

  For example, when the image displayed in the application is changed, it is necessary to reset the determination image of the automatic operation script. However, there is a problem that it is troublesome to collect the determination image while operating the target application.

JP2013-73428A

  The problem to be solved by the present invention is to provide an information processing apparatus and an information processing program capable of easily collecting images of a target application without correcting the target application.

  An information processing apparatus according to an embodiment includes a processor, an operation interface connected to an operation member, and a memory. The memory makes a transition according to the event, a device driver that causes the processor to detect an event from a signal input to the operation interface, an application that causes the processor to make a screen transition based on the event, and Causing the processor to execute generation of first image data based on the previous application screen and generation of second image data based on the application screen after transition by the event; And a screen information collecting program.

FIG. 1 is an explanatory diagram for describing a configuration example of an information processing apparatus according to an embodiment. FIG. 2 is an explanatory diagram for describing an example of a program stored in the memory of the information processing apparatus according to the embodiment. FIG. 3 is an explanatory diagram for explaining an example of the operation of the information processing apparatus that has executed the screen information collection program according to the embodiment. FIG. 4 is an explanatory diagram for explaining an example of screen information registered in the storage. FIG. 5 is an explanatory diagram for explaining an example of the operation of the information processing apparatus that has executed the filter driver. FIG. 6 is an explanatory diagram for explaining an example of the operation of the information processing apparatus that has executed the event acquisition program. FIG. 7 is an explanatory diagram for explaining an example of the operation of the information processing apparatus that has executed the image acquisition auxiliary program.

Hereinafter, an information processing apparatus and an information processing program according to an embodiment will be described with reference to the drawings.
FIG. 1 is an explanatory diagram for describing an example of an information processing apparatus 1 according to an embodiment.

  The information processing apparatus 1 is, for example, a personal computer, a smartphone, or a tablet PC. The information processing apparatus 1 is an apparatus that executes an automatic operation script that causes an application to execute a predetermined process by recognizing a screen displayed in the application. In addition, the information processing apparatus 1 generates and saves application screen data (screen information). This screen information is used to generate an image (determination image) used for determination in the automatic operation script.

  The information processing apparatus 1 includes a display 11, an operation interface 12, a communication interface 13, a storage 14, and a control unit 15.

  The display 11 displays a screen according to a video signal input from the control unit 15 or a display control unit such as a graphic controller (not shown). The information processing apparatus 1 may be configured to include an output terminal that outputs a video signal instead of the display 11.

  The operation interface 12 has various operation members. The operation interface 12 supplies an operation signal corresponding to the operation of the operation member to the control unit 15. The operation member is, for example, a touch sensor, a numeric keypad, a power key, a paper feed key, various function keys, or a keyboard. The touch sensor is, for example, a resistive touch sensor or a capacitive touch sensor. The touch sensor acquires information indicating a specified position in a certain area. The touch sensor is configured as a touch panel integrally with the display 11, thereby inputting a signal indicating the touched position on the screen displayed on the display 11 to the control unit 15. The operation interface 12 may be an interface that receives an operation signal from the operation member and supplies the operation signal to the control unit 15.

  The communication interface 13 is an interface for communicating with devices other than the information processing apparatus 1.

  The storage 14 is a storage medium capable of storing various information. The storage 14 stores a program and data used in the program. The storage 14 is, for example, a solid state drive (SSD), a hard disk drive (HDD), or another storage device. The information processing apparatus 1 may be configured to include an interface capable of communicating with an external storage device instead of the storage 14.

  The storage 14 has a database (screen information database) in which screen information described later is registered. The storage 14 sequentially registers the screen information supplied from the control unit 15 in the screen information database. Further, the storage 14 has a screen information database (automatic operation script database) in which an image used for determination in an automatic operation script (determination image) and processing to be executed by an application are associated with each other.

  The control unit 15 controls the information processing apparatus 1. The control unit 15 includes, for example, a processor 16 and a memory 17.

  The processor 16 is an arithmetic element that executes arithmetic processing. The processor 16 is configured as a CPU, for example. The processor 16 executes various processes by executing a program stored in the memory 17 or the storage 14.

  The memory 17 has a ROM that stores a program and data used in the program. The memory 17 has a RAM that temporarily stores (decompresses) data being processed by the processor.

  FIG. 2 is an explanatory diagram for explaining an example of a program stored in the memory 17. The memory 17 stores an operating system 21, a device driver 22, an application 23, and a screen information collection program 24.

  The operating system 21 manages various programs executed by the processor 16, controls input / output between programs, controls input / output to / from hardware, and manages files stored in the storage 14. It is a program for doing it. The operating system 21, device driver 22, application 23, and screen information collection program 24 operate on the operating system 21 executed by the processor 16.

  The device driver 22 is a program for causing the processor 16 to control the hardware connected to the information processing apparatus 1 by the operating system 21. For example, the processor 16 that has executed the device driver 22 analyzes an event generated by operating the operation interface 12 and supplies the analyzed event to the operating system 21. The processor 16 that has executed the device driver 22 supplies a video signal to the display 11. The processor 16 that has executed the device driver 22 acquires an event from the filter driver 31 described later. Further, the processor 16 that has executed the device driver 22 may acquire an event directly from the operation interface 12. In other words, the device driver 22 is a program that causes the processor 16 to detect an event from a signal input to the operation interface 12.

  The application 23 is a program stored in the memory 17 so as to be executable by the processor 16. The application 23 is a GUI-based application program that can be operated by an operation member such as a mouse or a touch panel. The processor 16 that has executed the application 23 generates video signals of various screens based on processing on the application and displays them on the display 11. For example, when an operation for selecting an object is performed on a screen on which a selectable object is displayed, the application 23 shifts the display to a screen corresponding to the selected object. That is, the application 23 is a program that causes the processor 16 to perform screen transition based on the event detected by the device driver 22.

  The screen information collection program 24 is a program stored in the memory 17 so as to be executable by the processor 16. The screen information collection program 24 is a program for causing the processor 16 to perform processing for saving the screen generated by the application 23. For example, the screen information collection program 24 generates image data based on the screen of the application 23 before the transition by the event, and generates image data based on the screen of the application 23 after the transition by the event. Is a program that causes the processor 16 to execute. The screen information collection program 24 includes a filter driver 31, an event acquisition program 32, an image acquisition auxiliary program 33, and the like.

  The filter driver 31 is a driver that acquires an event (operation event) in the operation interface 12 and causes the processor 16 to perform processing for notifying the event acquisition program 32 and the device driver 22 of the event. For example, the processor 16 arranges the filter driver 31 in the lower layer of the device driver 22. For example, the filter driver 31 acquires an operation event by acquiring report descriptor information. When the device driver 22 is configured to acquire an event directly from the operation interface 12, the processor 16 that has executed the filter driver 31 may not notify the device driver 22 of the acquired event.

  The event acquisition program 32 is a program for causing the processor 16 to analyze the operation event notified from the filter driver 31. The event acquisition program 32 causes the processor 16 to analyze an operation event, for example, by acquiring report descriptor information. If the operation event notified from the filter driver 31 is a preset event, the processor 16 that has executed the event acquisition program 32 notifies the image acquisition auxiliary program 33 of a screen information acquisition request. Specifically, the processor 16 that has executed the event acquisition program 32 notifies the image acquisition auxiliary program 33 of a screen information acquisition request when the operation event notified from the filter driver 31 is a preset input event. To do.

  The input event is an event that occurs when a predetermined input operation is performed by an operation member provided in the operation interface 12. For example, when the operation interface 12 is configured as a mouse, the click event corresponds to the input event. When the operation interface 12 is configured as a touch sensor, the touch event corresponds to an input event.

  Note that the input event is not limited to the above example. For example, when the application 23 is a POS application, a device that executes the application 23 may be connected to or built in a scanner, a magnetic card reader, an IC card reader, or other various devices. Presumed. In such a configuration, an event (code input event) for reading a code (bar code, two-dimensional code, etc.) by a scanner, an event for reading a magnetic card by a magnetic card reader (magnetic card input event), or an IC card reader An IC card reading event (IC card input event) may be set as the input event.

  The image acquisition auxiliary program 33 is a program for causing the processor 16 to acquire the image data of the screen generated by the application 23 when a screen information acquisition request is received. When receiving the screen information acquisition request, the processor 16 that has executed the image acquisition auxiliary program 33 acquires the screen image data generated by the application 23. For example, the processor 16 generates image data based on a function generated by the operating system 21 (for example, a screen shot) based on the screen generated by the application 23.

  FIG. 3 is an explanatory diagram for explaining the processing of the processor 16 that has executed the screen information collection program 24. Here, description will be made assuming that the event acquisition program 32 is configured to acquire a preset input event.

  For example, as shown in FIG. 3, it is assumed that the application 23 is generating a first screen 41 having selectable objects. Further, it is assumed that when an input event for selecting a selectable object occurs on the first screen 41, the application 23 generates a second screen 42 corresponding to the selected object. In other words, the first screen 41 is a screen of the application 23 before transition due to an input event (pre-transition screen). The first screen 41 is a screen (post-transition screen) of the application 23 after transition by an input event.

  In such an application 23, the processor 16 that has executed the screen information collection program 24 detects an input event on the first screen 41 by the filter driver 31 and the event acquisition program 32.

  For example, when an input event is detected, the processor 16 is generated by the application 23 at a timing before the input event is reflected in the application 23 and at a timing after the input event is reflected in the application 23. Screen image data (for example, a raster image) is acquired. For example, the processor 16 acquires the first screen 41 of the application 23 as the first image data 43 at a timing before the input event is reflected in the application 23. Further, for example, the processor 16 acquires the second screen 42 of the application 23 as the second image data 44 at a timing after the input event is reflected in the application 23. That is, when one input event is detected, the processor 16 detects the first image data 43 that is an image of the first screen of the application 23 in which the input event has occurred, and the first image data 43 that has transitioned due to the input event. Two image data with the second image data 44 that is an image of the second screen 42 are generated.

  Specifically, when the processor 16 detects an operation event by the filter driver 31 and the event acquisition program 32 determines that the operation event is an input event, the filter driver 31 supplies the input event to the device driver 22. Before the first image data 43 is acquired. That is, the processor 16 that has executed the screen information collection program 33 detects an event from the signal input to the operation interface 12 before the device driver 22, and based on the screen of the application 23, acquires the first image data 43. Generate.

  Further, the processor 16 obtains the second image data 44 after a predetermined time has elapsed since the input event was supplied from the filter driver 31 to the device driver 22. If the first image data 43 can be acquired at the timing before the input event is reflected in the application 23, the processor 16 can acquire the device driver from the filter driver 31 before acquiring the first image data 43. 22 may be supplied with an input event.

  Furthermore, the processor 16 generates first screen information 45 using the acquired first image data 43. Further, the processor 16 generates the second screen information 46 using the acquired second image data 44. That is, when one input event is detected, the processor 16 generates two pieces of screen information, that is, the first screen information 45 and the second screen information 46. For example, the first screen information 45 and the second screen information 46 are generated by adding additional information to the first image data 43 and the second image data 44, respectively. The processor 16 registers the generated first screen information 45 and second screen information 46 in the screen information database of the storage 14.

  FIG. 4 is an explanatory diagram for explaining an example of screen information registered in the storage 14. 4 shows an example in which the first screen information 45 and the second screen information 46 are registered without distinction, but the first screen information 45 and the second screen information 46 are: It may be registered in another screen information database.

  The first screen information 45 is information having image data (first image data 43), identification information, event occurrence date / time, event occurrence coordinates, event type, and image address. The second screen information is information having image data (second image data 44), identification information, event occurrence date / time, event occurrence coordinates, event type, pre-event image, and image address. is there. That is, the first screen information 45 is based on the fact that the image data is the second image data 44 generated based on the screen of the application 23 after the input event and the point having the pre-event image. Different from 45.

  The identification information is information for identifying screen information. The identification information may be any information as long as it does not overlap with at least other screen information.

  The event occurrence date and time is information indicating timing such as date and time when image data is generated. The processor 16 generates an event occurrence date and time based on the input event acquired by the filter driver 31 and the event acquisition program 32. The event occurrence date and time may be information indicating the date and time (for example, the timing when the filter driver 31 detects the event) when the event that triggered the generation of the image data has occurred.

  The event occurrence coordinates are information indicating at which position (coordinates) on the image data the event has occurred. The processor 16 generates event occurrence coordinates based on the input event acquired by the filter driver 31 and the event acquisition program 32. For example, the event occurrence coordinates indicate a position where an input operation such as click or touch is performed on the screen of the application 23.

  The event type is information indicating the type of event. As described above, the input event includes various events such as a click event, a touch event, a code input event, a magnetic card input event, and an IC card input event. The event type indicates one of these events. That is, the event type indicates what kind of event caused the screen information to be generated.

  The image address is information indicating an address on a storage area where image data is stored. For example, when the image data is stored in the storage 14, the image address indicates an area on the storage 14 where the image data is stored.

  The pre-event image is information indicating the screen of the application 23 in which the input event has occurred. For example, the pre-event image is identification information of screen information generated based on the screen of the application 23 in which the input event has occurred. That is, the pre-event image is identification information of the first screen information generated based on the screen of the application 23 before the transition by the input event.

  Next, the processing of the processor 16 when executing each of the filter driver 31, the event acquisition program 32, and the image acquisition auxiliary program 33 of the screen information collection program 24 will be described.

  FIG. 5 is a flowchart for explaining the processing of the processor 16 that executes the filter driver 31.

  The processor 16 determines whether or not the screen acquisition mode is on (ACT11). For example, the processor 16 determines that the screen acquisition mode is in effect when the screen information collection program 24 is being executed. When the screen information collection program 24 is being executed, the processor 16 determines that the screen acquisition mode is on.

  When the processor 16 determines that the screen acquisition mode is on (ACT 11, YES), the processor 16 determines whether an event has occurred (ACT 12). If the processor 16 determines that no event has occurred (ACT12, NO), the processor 16 proceeds to the processing of ACT11. That is, the processor 16 repeatedly determines whether an event has occurred until the event occurs.

  If the processor 16 determines that an event has occurred (ACT 12, YES), it notifies the event acquisition program 32 of the operation event (ACT 13).

  Next, the processor 16 notifies an operation event to the device driver 22 (ACT 14). The processor 16 may be configured to notify the device driver 22 of the operation event after a predetermined time has elapsed after notifying the event acquisition program 32 of the operation event. Further, the processor 16 may be configured to determine whether or not the acquisition of the first image data 43 by the image acquisition auxiliary program 33 is completed, and to notify the device driver 22 of an operation event when the acquisition is completed.

  Next, the processor 16 notifies the event acquisition program 32 of the operation event again (ACT 15). The processor 16 may be configured to notify the operation event to the event acquisition program 32 again after a predetermined time has elapsed after notifying the device driver 22 of the operation event. Further, the processor 16 may be configured to determine whether or not the screen transition has been completed in the application 23 and to notify the event acquisition program 32 of the operation event again when it is completed. For example, the processor 16 compares the first image data 43 and the screen currently displayed in the application 23, and determines whether or not the screen transition of the application 23 is completed based on the comparison result. That is, when a difference occurs between the first image data 43 and the screen of the application 23, the processor 16 determines that the transition of the screen of the application 23 has been completed.

  The processor 16 determines whether or not the screen acquisition mode is turned off (ACT 16). When the screen acquisition mode is turned off (ACT 16, YES), the processor 16 ends the process. On the other hand, when the screen acquisition mode is not turned off (ACT16, NO), the processor 16 proceeds to the process of ACT12. As a result, the processor 16 waits for the next operation event to occur.

  FIG. 6 is a flowchart for explaining the processing of the processor 16 that has executed the event acquisition program 32.

  The processor 16 determines whether or not an operation event has been received from the filter driver 31 (ACT 21). When the processor 16 determines that an operation event has not been received (ACT21, NO), the processor 16 proceeds to processing of ACT21. That is, the processor 16 repeatedly determines whether or not an operation event has been received until the operation event is received.

  When it is determined that the operation event has been received (ACT21, YES), the processor 16 determines the type of the operation event (ACT22).

  The processor 16 determines whether or not the operation event is an input event (ACT 23). That is, the processor 16 determines whether or not the operation event is a preset event. When the processor 16 determines that the operation event is not an input event (ACT 23, NO), the processor 16 proceeds to the processing of ACT 21. That is, the processor 16 waits for reception of the next operation event.

  When the processor 16 determines that the operation event is an input event (ACT 23, YES), the processor 16 notifies the image acquisition auxiliary program 33 of the input event (ACT 24).

  FIG. 7 is a flowchart for explaining the processing of the processor 16 that has executed the image acquisition auxiliary program 33.

  The processor 16 determines whether or not an input event has been received from the event acquisition program 32 (ACT 31). When the processor 16 determines that an input event has not been received (ACT 31, NO), the processor 16 proceeds to processing of ACT 31. That is, the processor 16 repeatedly determines whether or not an input event has been received until the input event is received.

  When the processor 16 determines that an input event has been received (ACT 31, YES), the processor 16 generates screen information based on the screen of the application 23 in which the input event has occurred (ACT 32). That is, the processor 16 obtains image data of the screen of the application 23 and generates screen information by adding additional information such as identification information, event occurrence date / time, event occurrence coordinates, and image address to the image data.

  When receiving an input event, the processor 16 determines whether the screen of the application 23 is a pre-transition screen or a post-transition screen. For example, the processor 16 refers to the processing contents of the filter driver 31 and the event acquisition program 32 to determine whether the screen of the application 23 is a pre-transition screen or a post-transition screen. The processor 16 may be configured to determine that the screen of the application 23 is a post-transition screen when the same input event is received twice in succession. Further, the filter driver 31 may add information for distinguishing the operation event notified in ACT 13 in FIG. 5 and the operation event notified in ACT 15 in FIG. 5 to the operation event.

  When the processor 16 determines that the screen of the application 23 is the pre-transition screen, the screen information including the image data generated from the pre-transition screen, identification information, event occurrence date and time, event occurrence coordinates, and image address (first Screen information 45) is generated. If the processor 16 determines that the screen of the application 23 is a post-transition screen, the processor 16 further transitions to the image data, identification information, event occurrence date / time, event occurrence coordinates, and image address generated from the post-transition screen. Screen information (second screen information 46) having a pre-event image indicating first screen information 45 having image data of the previous screen is generated.

  The processor 16 calculates the similarity between the generated image data of the screen information and the image data of the screen information in the screen information database (ACT33). When a plurality of pieces of screen information are registered in the screen information database, the processor 16 calculates a similarity for each piece of screen information. Note that the similarity may be calculated by any method.

  The processor 16 determines whether or not the calculated similarity is greater than or equal to a threshold value (ACT 34). For example, the processor 16 determines whether or not the highest similarity among the plurality of calculated similarities is greater than or equal to a threshold value.

  When the processor 16 determines that the calculated similarity is not equal to or greater than the threshold (ACT 34, YES), the screen information generated by the ACT 32 is registered in the screen information database of the storage 14 (ACT 35), and the process proceeds to ACT 31. Waits for the next input event.

On the other hand, when the processor 16 determines that the calculated similarity is equal to or greater than the threshold (ACT 34, YES), the image data of the screen information is replaced with a shortcut (link) (ACT 36), and the process proceeds to ACT 35. For example, the processor 16 generates a shortcut to the image data having the highest similarity calculated in ACT 33, and replaces the image data of the screen information generated in ACT 32 with the shortcut. The processor 16 registers the screen information obtained by replacing the image data with the shortcut in the screen information database of the storage 14, moves to the processing of ACT 31, and waits for the next input event.
That is, the processor 16 calculates the similarity between the first image data 43 and the image data registered in the database in advance, and when the calculated similarity is less than a preset threshold, First screen information 45 including data 43 is registered in the database. Further, the processor 16 calculates the similarity between the second image data 44 and the image data registered in the database in advance, and if the calculated similarity is less than a preset threshold, Second screen information 46 including data 44 is registered in the database. Further, the processor 16 calculates the similarity between the first image data 43 and the image data registered in the database in advance, and if the calculated similarity is equal to or greater than a preset threshold, the first image Instead of the data 43, the first screen information 45 including a link to the image data used for calculating the similarity is registered in the database. Further, the processor 16 calculates the similarity between the second image data 44 and the image data registered in the database in advance, and if the calculated similarity is equal to or greater than a preset threshold, Instead of the data 44, the second screen information 46 including a link to the image data used for calculating the similarity is registered in the database.

  According to the configuration described above, the processor 16 of the information processing apparatus 1 executes the application 23 and also executes a screen information collection program that acquires image data of the screen of the application 23. The processor 16 monitors an input from the operation interface and detects an operation event by the screen information collection program 24. When the operation event is a predetermined event (for example, an input event), the processor 16 generates the first image data 43 from the screen of the application 23 at a timing before the input event is reflected on the application 23, and inputs the first image data 43. The second image data 44 is generated from the screen of the application 23 at a timing after the event is reflected in the application 23. The processor 16 registers the generated first image data 43 and second image data 44 in the screen information database of the storage 14 respectively.

  Thus, the processor 16 can generate the first image data 43 and the second image data 44 by the screen information collection program 24 every time an input event occurs for the application 23. That is, the first image data 43 and the second image data 44 can be collected without modifying the application 23 and performing a special operation. This facilitates the collection of determination images used in automatic operation scripts.

  The screen information collection program 24 includes a filter driver 31 that detects an operation event before the device driver 22 that reflects the operation event generated by the operation of the operation interface 12 in the application 23 is notified of the operation event. Thus, the processor 16 that has executed the screen information collection program 24 can generate the first image data 43 from the screen of the application 23 at a timing before the input event is reflected in the application 23. In addition, the processor 16 can generate the second image data 44 by acquiring an image of the screen of the application 23 after a predetermined time has elapsed since the operation event was detected by the filter driver 31. In this way, the processor 16 that has executed the screen information collection program 24 can generate two pieces of image data after transition and before transition based on one input event.

  Further, the processor 16 adds information such as event occurrence date and time and event occurrence coordinates to the first image data 43 and the second image data 44, respectively, and the first screen information 45 and the second screen information 46 are added. Generate and register in the screen information database of the storage 14. This makes it possible to grasp the event occurrence date and time later. It is also possible to grasp which position on the screen of the application 23 is clicked (touched) when an event occurs.

  Further, when the second image data 44 is stored in the storage, the processor 16 stores second information (pre-event image) indicating the screen before the transition of the screen of the application 23 used to generate the second image data 44. To the image data 44. This makes it possible to grasp which second image data 44 is generated based on the screen from which the application 23 has transitioned.

  The processor 16 refers to a plurality of image data already registered in the storage 14 before storing the newly generated image data in the storage 14. The processor 16 calculates the similarity between the newly generated image data and a plurality of image data registered in the storage 14, and when image data having a similarity equal to or greater than a threshold is stored, a shortcut to the image is stored. Is stored instead of image data. This can prevent similar image data from being stored in the plurality of storages 14.

  It should be noted that the functions described in the above embodiments are not limited to being configured using hardware, but can be realized by causing a computer to read a program describing each function using software. Each function may be configured by appropriately selecting either software or hardware.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Information processing apparatus, 11 ... Display, 12 ... Operation interface, 13 ... Communication interface, 14 ... Storage, 15 ... Control part, 16 ... Processor, 17 ... Memory, 21 ... Operating system, 22 ... Device driver, 23 ... Application 24 ... Screen information collection program, 31 ... Filter driver, 32 ... Event acquisition program, 33 ... Image acquisition auxiliary program.

Claims (5)

A processor;
An operation interface connected to the operation member;
A device driver for causing the processor to detect an event from a signal input to the operation interface, an application for causing the processor to perform screen transition based on the event, and the application before the transition due to the event Screen information collection for causing the processor to generate the first image data based on the screen of the image and to generate the second image data based on the screen of the application after the transition by the event. A memory storing a program,
An information processing apparatus comprising:   The processor that has executed the screen information collection program detects the event from a signal input to the operation interface before the device driver, and generates the first image data based on the screen of the application. The information processing apparatus according to claim 1. The processor that has executed the screen information collection program includes:
When the similarity between the first image data and image data registered in the database in advance is calculated, and the calculated similarity is less than a preset threshold, the first image data is stored in the database. Register,
When the similarity between the second image data and the image data registered in the database in advance is calculated, and the calculated similarity is less than a preset threshold, the second image data is Register in the database,
When the similarity between the first image data and the image data registered in the database in advance is calculated, and the calculated similarity is equal to or greater than a preset threshold value, the similarity is used instead of the first image data. Register a link to the image data used to calculate the degree in the database,
When the similarity between the second image data and the image data registered in the database in advance is calculated, and the calculated similarity is equal to or greater than a preset threshold value, instead of the second image data The information processing apparatus according to claim 1, wherein a link to image data used for calculating similarity is registered in the database.   The processor that has executed the screen information collecting program adds information indicating a position where the event has occurred on the first image data to the first image data and the second image data, respectively. The information processing apparatus according to claim 3, wherein the information processing apparatus is registered in a database. A processor, an operation interface connected to the operation member, a device driver that causes the processor to detect an event from a signal input to the operation interface, and a screen transition to the processor based on the event An information processing program that is executed in an information processing apparatus comprising:
Generating first image data based on a screen of the application before transition to the processor due to the event;
Generating second image data based on the screen of the application after transition by the event;
Information processing program that executes