WO2023238356A1 - 生成装置、生成方法及び生成プログラム - Google Patents

生成装置、生成方法及び生成プログラム Download PDF

Info

Publication number
WO2023238356A1
WO2023238356A1 PCT/JP2022/023374 JP2022023374W WO2023238356A1 WO 2023238356 A1 WO2023238356 A1 WO 2023238356A1 JP 2022023374 W JP2022023374 W JP 2022023374W WO 2023238356 A1 WO2023238356 A1 WO 2023238356A1
Authority
WO
WIPO (PCT)
Prior art keywords
captured image
generation
time
screen
captured
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/023374
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
美沙 深井
将志 田所
晴夫 大石
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NTT Inc
Original Assignee
Nippon Telegraph and Telephone Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nippon Telegraph and Telephone Corp filed Critical Nippon Telegraph and Telephone Corp
Priority to PCT/JP2022/023374 priority Critical patent/WO2023238356A1/ja
Priority to JP2024526178A priority patent/JPWO2023238356A1/ja
Publication of WO2023238356A1 publication Critical patent/WO2023238356A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment

Definitions

  • the conventional technology has a problem in that it may not be possible to generate high-quality operation logs.
  • operation logs are acquired based on operation events such as clicks and key inputs, it is difficult to acquire logs for items that are displayed only by cursor movement (such as menus displayed by mouseover).
  • the present invention has been made in view of the above, and an object of the present invention is to generate high-quality operation logs.
  • FIG. 1 is a diagram showing an example of the configuration of a generation device according to the first embodiment.
  • FIG. 2 is a diagram illustrating a method for deleting a captured image.
  • FIG. 3 is a diagram showing an example of a difference image.
  • FIG. 4 is a diagram illustrating extraction of differences.
  • FIG. 5 is a diagram illustrating extraction of differences.
  • FIG. 6 is a diagram showing an example of the characteristics of the change.
  • FIG. 7 is a diagram illustrating a method for deleting an operation.
  • FIG. 8 is a diagram illustrating a method for adding operations.
  • FIG. 9 is a flowchart illustrating an example of the flow of acquisition processing.
  • FIG. 10 is a flowchart showing an example of the flow of extraction processing.
  • FIG. 11 is a flowchart showing an example of the flow of extraction processing.
  • FIG. 12 is a flowchart illustrating an example of the flow of generation processing.
  • FIG. 13 is a diagram illustrating an example of a computer
  • the work in the embodiment includes input work to a terminal device such as a PC (Personal Computer).
  • a terminal device such as a PC (Personal Computer).
  • business operations using a terminal device are performed by operating an input device such as a mouse, a keyboard, or a touch panel.
  • the operation log is data that associates a screen capture image with an operation event, which will be described later.
  • operation log generated by the generation device can be used, for example, in procedures such as business analysis, particularly process visualization and timeline visualization, as described in Non-Patent Document 1.
  • the operation log generated by the generation device may be used as data for learning a model (for example, a neural network) that automatically performs operations on the terminal device that imitate human input operations.
  • a model for example, a neural network
  • FIG. 1 is a diagram illustrating an example of the configuration of a generation device according to the first embodiment.
  • the generation device 10 is connected to a terminal device 20.
  • the terminal device 20 is a device such as a PC for a worker to perform work. Further, the generation device 10 does not need to be directly connected to the terminal device 20, as long as it can acquire information regarding operations on the terminal device 20 by any means.
  • the generation device 10 includes a communication section 11, a storage section 12, and a control section 13.
  • the storage unit 12 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. Note that the storage unit 12 may be a data-rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory).
  • a RAM Random Access Memory
  • a flash memory or an NVSRAM (Non Volatile Static Random Access Memory).
  • the storage unit 12 stores an OS (Operating System) and various programs executed by the generation device 10.
  • the storage unit 12 stores operation log information 121 and learning information 122.
  • the operation log information 121 includes an operation log.
  • the operation log is information that associates screen capture images with operation events. The method of generating the operation log will be described later.
  • the control unit 13 controls the entire generation device 10.
  • the control unit 13 includes, for example, electronic circuits such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate). Array) etc. It is an integrated circuit.
  • control unit 13 has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. Further, the control unit 13 functions as various processing units by running various programs.
  • control unit 13 includes an acquisition unit 131, an extraction unit 132, and a generation unit 133.
  • the acquisition unit 131 acquires multiple operation events that occur in response to operations on the screen of the terminal device 20 and multiple captured images of the screen.
  • the acquisition unit 131 can acquire operation events and captured images via the communication unit 11.
  • the operation event corresponding to the mouse operation includes information identifying the operated button or wheel, and the type of operation (click, press down, push up, scroll, etc.). Further, the operation event corresponding to the keyboard operation includes the type of the operated key.
  • the acquisition unit 131 captures a screen capture image from the terminal device 20, the time at which the captured image was acquired (time stamp), and the mouse cursor position at the time, at regular time intervals, regardless of the occurrence of an operation event. Get location.
  • the extraction unit 132 extracts, from among the plurality of captured images acquired by the acquisition unit 131, a captured image that satisfies a condition indicating that it is related to an operation.
  • the extraction unit 132 determines whether the captured image satisfies a condition indicating that it is related to the operation. Specifically, the extraction unit 132 determines whether a change related to the operation appears in the captured image to be extracted.
  • the extraction unit 132 deletes captured images in which only changes that are unrelated to the above-mentioned operations or have a small degree of association appear.
  • the extraction unit 132 may perform the extraction on the original captured image (the captured image 311 or 312 in the example of FIG. 3) or the generated difference image (the difference image 313 in the example of FIG. 3). You can also go to
  • FIG. 4 is a diagram illustrating extraction of differences.
  • the extraction unit 132 cuts out an image 323 that includes the entire portion where a difference occurs between the captured image 321 and the captured image 322.
  • the extraction unit 132 uses the cut out image to determine the captured image.
  • the extraction unit 132 can perform the determination based on one or more of the characteristics of the changed portion, the relationship between the changed portion and the position of the cursor, and the acquisition time of the captured image.
  • the extraction unit 132 can utilize the features shown in FIG. That is, the extraction unit 132 extracts the extracted capture image, the extracted difference image, the size of the range where the change occurs (for example, a rectangle whose sides are parallel to the X axis and the Y axis), the change position, and the similar change You can use the presence/absence and interval of capture image acquisition time.
  • FIG. 6 is a diagram showing an example of the characteristics of the change.
  • the extracting unit 132 extracts, from among the plurality of captured images, a captured image in which the characteristics of a portion where a change in the screen appears over time satisfies the condition. For example, the extraction unit 132 determines that the condition is satisfied when the characteristics shown in FIG. 6 are similar to the characteristics of changes caused by predetermined operations.
  • the extraction unit 132 may perform the determination using a machine learning model constructed based on the learning information 122. In this case, it is assumed that the machine learning model has already learned whether there is a relationship between features and operations using training data.
  • a machine learning model outputs a score representing the degree of similarity to an operation based on the input features.
  • the extraction unit 132 determines the corresponding captured image as a captured image related to the operation.
  • the extraction unit 132 extracts, from among the plurality of captured images, a captured image in which the distance between the position of the portion where the screen changes over time and the position where the cursor is indicated is equal to or greater than a threshold value.
  • the acquisition unit 131 can acquire the position of the cursor (for example, a mouse cursor and a cursor indicating a character input position) at the time when the captured image was acquired. This makes it possible to identify changes that occur without any operation.
  • the cursor for example, a mouse cursor and a cursor indicating a character input position
  • the cursor position is far from the change position, it is highly likely that the change occurred without any operation.
  • the cursor position is close to the change position, it is considered that the change is likely to have occurred in connection with an operation.
  • the extracting unit 132 extracts a captured image from among the plurality of captured images in which a change in the screen due to the passage of time appears, and in which the difference between the acquired time and the occurrence time of the nearest operation event is less than or equal to a threshold value. Extract images. This makes it possible to identify the relationship between screen changes and operations based on easily obtainable information such as time.
  • a change occurs in a short period of time, it is highly likely that an operation was performed. For example, in the case of a web advertisement, the same screen is displayed for a certain period of time. On the other hand, a worker may perform an operation to switch screens in a short period of time.
  • the generation unit 133 compares the acquisition time of the captured image extracted by the extraction unit 132 with the occurrence time of the operation event, and links the captured image and the operation event. As a result, the operation event and the captured images before and after the operation corresponding to the operation event are linked.
  • FIG. 7 is a diagram illustrating a method for deleting an operation.
  • deletion of an operation event or captured image by the generation unit 133 means excluding it from being added to the operation log.
  • Operation_1, operation_2, and operation_3 in FIG. 7 are operation events.
  • operation events and captured images are arranged in chronological order based on acquisition time or occurrence time.
  • the extraction unit 132 extracts the captured image_1 and the captured image_2, which was acquired after the captured image_1 and in which a change in the screen appeared for the first time after the acquisition time of the captured image_1. Then, the generation unit 133 generates an operation event other than operation_3, which is the operation event with the latest occurrence time, among the plurality of operation events that occurred between the acquisition time of capture image_1 and the acquisition time of capture image_2. Operation_2, which is , is excluded from the operation log. This allows you to exclude meaningless operation events from the operation log.
  • FIG. 8 is a diagram illustrating a method for adding operations.
  • Operation_1, operation_2, and operation_3 in FIG. 8 are operation events.
  • operation events and captured images are arranged in chronological order based on acquisition time or occurrence time.
  • operation_4 is an operation event generated by the generation unit 133.
  • the extraction unit 132 extracts the captured image_2 and the captured image_3, which was acquired after the captured image_2 and in which a change in the screen appeared for the first time after the acquisition time of the captured image_2. Then, if there is no operation event that occurred between the acquisition time of capture image_2 and the acquisition time of capture image_3, the generation unit 133 determines that the occurrence time is the same as the acquisition time of capture image_2 and the acquisition time of capture image_3. (including the acquisition time of capture image_3) is generated in a pseudo manner and added to the operation log.
  • the occurrence time is the acquisition time of capture image_3, the content of the operation is none or a predetermined operation content such as cursor movement, and the cursor position is the cursor coordinates at the time of capture_3 acquisition. Good too.
  • FIG. 9 is a flowchart showing an example of the flow of the acquisition process.
  • the acquisition process is a process performed by the acquisition unit 131.
  • steps S102 to S104 are repeated.
  • dropping the terminal device 20 means shutting down the terminal device 20, turning off the power of the terminal device 20, and the like.
  • the generation device 10 ends the process if the worker stops the process or if the worker drops the terminal device 20 (step S101, Yes).
  • step S104 the generation device 10 acquires the operation event of the terminal device 20.
  • the operation event includes the event type (mouse click, keyboard key press, etc.), the time of occurrence, the mouse cursor position at the time of occurrence, and the like.
  • step S202 the generation device 10 determines a target capture image.
  • step S203 determines whether there is a difference between the target captured image and the continuously acquired captured images.
  • FIG. 11 is a flowchart showing an example of the flow of the extraction process.
  • the extraction process is a process performed by the extraction unit 132.
  • the generation device 10 repeats steps S302 to S306 until all captured images acquired in the acquisition process are targeted (step S301, No).
  • the generation device 10 targets all captured images (step S301, Yes)
  • the generation device 10 ends the process.
  • step S302 the generation device 10 determines a target capture image.
  • the generation device 10 determines whether or not the difference has been extracted for the target captured image (step S303). Note that the extraction of the difference is performed by the process shown in FIG.
  • step S303 If the generation device 10 determines that the difference has been extracted (step S303, Yes), the generation device 10 generates a captured image of the target using information about the characteristics of the changed portion, the relationship between the changed portion and the cursor position, and the timing at which the change occurred. It is determined whether the captured image is related to the operation (step S304).
  • step S305 the generation device 10 determines that the target captured image is not a captured image related to an operation (step S305, No).
  • the process returns to step S301.
  • Step S305 If the generation device 10 determines that the target captured image is a captured image associated with an operation (Step S305, Yes), the generation device 10 extracts the captured image as an image associated with the operation (Step S306).
  • the generation device 10 determines whether or not there is a captured image extracted before the next operation event for the targeted operation event (step S403).
  • step S403, No If the generation device 10 determines that there is no captured image extracted before the next operation event (step S403, No), the process returns to step S401.
  • step S403 If the generation device 10 determines that there is a captured image extracted before the next operation event (step S403, Yes), the generation device 10 generates a captured image extracted at the time closest to the target operation event and the time when the operation event occurs. The captured images are linked (step S404).
  • step S405, No If the generation device 10 determines that two or more captured images do not exist between the target operation event and the next operation event (step S405, No), the process returns to step S401.
  • step S405 If the generation device 10 determines that two or more captured images exist between the target operation event and the next operation event (step S405, Yes), the generation device 10 generates an operation corresponding to the captured image (a pseudo-generated operation event). is added to the operation log (step S406).
  • the generation device 10 can generate operation logs automatically and with high precision. As a result, according to the embodiment, a high-quality operation log can be generated.
  • each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings.
  • the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices may be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.
  • each processing function performed by each device is realized in whole or in part by a CPU (Central Processing Unit) and a program that is analyzed and executed by the CPU, or by hardware using wired logic. It can be realized as Note that the program may be executed not only by the CPU but also by another processor such as a GPU.
  • a CPU Central Processing Unit
  • the generation device 10 can be implemented by installing a generation program that executes the above generation process on a desired computer as packaged software or online software. For example, by causing the information processing device to execute the above generation program, the information processing device can be made to function as the generation device 10.
  • the information processing device referred to here includes a desktop or notebook personal computer.
  • information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHSs (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).
  • the generation device 10 can also be implemented as a generation server device that uses a terminal device used by a user as a client and provides services related to the above generation process to the client.
  • the generation server device is implemented as a server device that provides a generation service that receives an operation event of a terminal device and a captured image as input, and outputs a generated operation log.
  • the generation server device may be implemented as a web server, or may be implemented as a cloud that provides services related to the above generation process through outsourcing.
  • FIG. 13 is a diagram showing an example of a computer that executes the generation program.
  • Computer 1000 includes, for example, a memory 1010 and a CPU 1020.
  • the computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.
  • the memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012.
  • the ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System).
  • Hard disk drive interface 1030 is connected to hard disk drive 1090.
  • Disk drive interface 1040 is connected to disk drive 1100.
  • Serial port interface 1050 is connected to, for example, mouse 1110 and keyboard 1120.
  • Video adapter 1060 is connected to display 1130, for example.
  • the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the generation device 10 is implemented as a program module 1093 in which computer-executable code is written.
  • Program module 1093 is stored in hard disk drive 1090, for example.
  • a program module 1093 for executing processing similar to the functional configuration of the generation device 10 is stored in the hard disk drive 1090.
  • the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
  • the setting data used in the processing of the embodiment described above is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 as necessary, and executes the processing of the embodiment described above.
  • program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like.
  • the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from another computer via the network interface 1070.
  • LAN Local Area Network
  • WAN Wide Area Network

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
PCT/JP2022/023374 2022-06-09 2022-06-09 生成装置、生成方法及び生成プログラム Ceased WO2023238356A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2022/023374 WO2023238356A1 (ja) 2022-06-09 2022-06-09 生成装置、生成方法及び生成プログラム
JP2024526178A JPWO2023238356A1 (https=) 2022-06-09 2022-06-09

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/023374 WO2023238356A1 (ja) 2022-06-09 2022-06-09 生成装置、生成方法及び生成プログラム

Publications (1)

Publication Number Publication Date
WO2023238356A1 true WO2023238356A1 (ja) 2023-12-14

Family

ID=89117810

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/023374 Ceased WO2023238356A1 (ja) 2022-06-09 2022-06-09 生成装置、生成方法及び生成プログラム

Country Status (2)

Country Link
JP (1) JPWO2023238356A1 (https=)
WO (1) WO2023238356A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025169939A1 (ja) * 2024-02-07 2025-08-14 株式会社Brain Magic 情報処理システム、情報処理装置およびプログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014067262A (ja) * 2012-09-26 2014-04-17 Sumitomo Chemical Co Ltd アラーム設定の最適化支援装置および最適化支援プログラム
JP2015049520A (ja) * 2013-08-29 2015-03-16 富士通株式会社 シナリオ生成プログラム、シナリオ実行プログラム、シナリオ生成方法、シナリオ実行方法、シナリオ生成装置、及びシナリオ実行装置
WO2016051479A1 (ja) * 2014-09-29 2016-04-07 株式会社日立製作所 画面仕様分析装置及び方法
WO2021140594A1 (ja) * 2020-01-08 2021-07-15 日本電信電話株式会社 操作ログ取得装置および操作ログ取得方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014067262A (ja) * 2012-09-26 2014-04-17 Sumitomo Chemical Co Ltd アラーム設定の最適化支援装置および最適化支援プログラム
JP2015049520A (ja) * 2013-08-29 2015-03-16 富士通株式会社 シナリオ生成プログラム、シナリオ実行プログラム、シナリオ生成方法、シナリオ実行方法、シナリオ生成装置、及びシナリオ実行装置
WO2016051479A1 (ja) * 2014-09-29 2016-04-07 株式会社日立製作所 画面仕様分析装置及び方法
WO2021140594A1 (ja) * 2020-01-08 2021-07-15 日本電信電話株式会社 操作ログ取得装置および操作ログ取得方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025169939A1 (ja) * 2024-02-07 2025-08-14 株式会社Brain Magic 情報処理システム、情報処理装置およびプログラム
JP2025121562A (ja) * 2024-02-07 2025-08-20 株式会社Brain Magic 情報処理システム、情報処理装置およびプログラム
JP7785302B2 (ja) 2024-02-07 2025-12-15 株式会社Brain Magic 情報処理システム、情報処理装置およびプログラム

Also Published As

Publication number Publication date
JPWO2023238356A1 (https=) 2023-12-14

Similar Documents

Publication Publication Date Title
CN113688288A (zh) 数据关联分析方法、装置、计算机设备和存储介质
CN112817817A (zh) 埋点信息查询方法、装置、计算机设备和存储介质
US20210406104A1 (en) Information processing apparatus and non-transitory computer-readable storage medium for storing api use history display program
CN111400361A (zh) 数据实时存储方法、装置、计算机设备和存储介质
CN113051183B (zh) 一种测试数据的推荐方法、系统、电子设备及存储介质
CN111447507A (zh) 视频制作方法和装置、电子设备和存储介质
WO2023238356A1 (ja) 生成装置、生成方法及び生成プログラム
US11513884B2 (en) Information processing apparatus, control method, and program for flexibly managing event history
US20220365789A1 (en) Storage medium, information processing method, and information processing apparatus
CN111581518A (zh) 信息推送方法和装置
JP6642024B2 (ja) 管理装置，管理方法および管理プログラム
JP7211228B2 (ja) 解析装置、解析方法、及びプログラム
CN112100156A (zh) 基于用户行为构建知识库的方法、装置、介质、系统
US12380071B2 (en) Tracking data lineage and applying data removal to enforce data removal policies
Murtuza et al. A tool for extracting static and volatile forensic artifacts of windows 8. x apps
JP2023106867A (ja) エビデンス管理装置、エビデンス管理方法、プログラム、及び、記録媒体
JP7691482B1 (ja) 情報提供装置、情報提供方法及び情報提供プログラム
JP7845465B2 (ja) 特定装置、特定方法及び特定プログラム
CN103713987A (zh) 基于关键词的日志处理
CN108960433B (zh) 用于运行机器学习建模过程的方法及系统
CN113742501A (zh) 一种信息提取方法、装置、设备、及介质
KR102745736B1 (ko) 데이터 수집 방법 및 이를 수행하기 위한 시스템
Zhou et al. Reconstructing android user behavior through timestamped state models
JP7601259B2 (ja) 情報付与装置、情報付与方法及び情報付与プログラム
CN115859278B (zh) 软件操作行为审计方法、系统、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22945857

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024526178

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18871816

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22945857

Country of ref document: EP

Kind code of ref document: A1