JP2014137631A - Processing program, terminal device, processing system, and processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a processing program capable of effectively recording data related to user's operation applied to a terminal device.SOLUTION: The processing program is a program for making a computer execute the steps of: an operation section disposed in a terminal device (as one example, terminal device 1), receiving an operation performed by a user; an operation data acquiring section disposed in the terminal device, acquiring data related to the operation received by the operation section as data having a format that does not depend on the terminal; the operation data acquiring section disposed in the terminal device, transmitting the acquired data to an external device (as one example, management device 2).

Description

  The present invention relates to a processing program, a terminal device, a processing system, and a processing method.

For example, Android is a platform for terminal devices such as smartphones and tablet personal computers (PCs).
In the following, [Background Art] and [Problems to be Solved by the Invention] will be described by taking Android as an example.

  Non-Patent Document 1 describes commands for getevent and sendevent (see Non-Patent Document 1). By executing a getevent command, it is possible to dump data (for example, list data) of a user (for example, a person) performed in the terminal device from the kernel. Further, by executing a send command using the dumped data as an argument, it is possible to reproduce a result corresponding to a user operation in the terminal device. Here, the user operation data is acquired from the kernel, and is unique to each terminal device.

  Non-Patent Document 2 describes a tool called MonkeyRunner (see Non-Patent Document 2). The user can record (store) operation data by operating the terminal device using the monkey recorder GUI (Graphical User Interface) in a personal computer connected to the terminal device using the monkey recorder attached to the monkey runner. Is possible. By inputting the recorded operation data into the monkeyrunner, it is possible to reproduce the result according to the user operation. Here, in MonkeyRunner, the user uses a personal computer to remotely operate the terminal device. In addition, since MoneyKeyner captures and records the screen of the terminal device with a personal computer, operations such as double taps and long presses are not recognized.

“Android Debug Bridge”, [online], [December 5, 2012 search], Internet, <http: // developer. android. com / tools / help / adb. html> “Monkeyrunner”, [online], [searched on December 5, 2012], Internet, <http: // developer. android. com / tools / help / monkeyrunner_concepts. html>

  However, conventionally, there has been an inadequate technique for recording data relating to operations performed by the user on the terminal device. Specifically, for example, there has been no technique for recording data (terminal-independent data) that does not depend on each terminal device for data relating to operations such as double tap and long press.

  The getevent and senddev commands described in Non-Patent Document 1 described above can accurately record and play back user operations, but the dumped data is data that depends on each terminal device. . For this reason, data dumped by a certain terminal device cannot be applied to other types of terminal devices. Also, it is difficult for the user to directly edit the dumped data.

  In the Monkey Recorder attached to the MonkeyKeyer described in Non-Patent Document 2 described above, intuitive operation recording is possible with the GUI, but the types of operations that can be recorded are limited. For example, double tap It is not possible to record operations such as and long presses. In addition, in the Money Recorder, the terminal device and the personal computer need to be connected and the user needs to operate on the screen of the personal computer, and the user cannot directly record the operation by operating the terminal device directly.

  The present invention has been made in view of such circumstances, and provides a processing program, a terminal device, a processing system, and a processing method capable of effectively recording data relating to an operation performed on a terminal device by a user. The issue is to provide.

  (1) In order to solve the above-described problem, a processing program according to the present invention includes a step in which an operation unit provided in a terminal device accepts an operation performed by a user, and operation data acquisition provided in the terminal device A unit that acquires data relating to an operation received by the operation unit as data in a format independent of a terminal, and the operation data acquisition unit provided in the terminal device externally acquires the acquired data. A program for causing a computer to execute the step of transmitting to the apparatus.

  (2) The present invention provides the processing program according to (1), wherein the operation data acquisition unit provided in the terminal device stores data related to an operation received by the operation unit in a format independent of a terminal. When acquired as data, an operation accepted by the operation unit is switched to a state in which the operation accepted by the operation unit cannot be transmitted to the operation data obtaining unit, and an operation accepted by the operation unit in response to an instruction from the external device. And a step of causing the computer to execute a step of switching to a state that can be transmitted to the operation data acquisition unit.

  (3) In order to solve the above-described problem, a terminal device according to the present invention includes an operation unit that receives an operation performed by a user, and data related to the operation received by the operation unit in a format independent of the terminal. And an operation data acquisition unit that acquires the data and transmits the acquired data to an external device.

  (4) In order to solve the above-described problem, a processing system according to the present invention includes a terminal device and a management device, and the terminal device receives an operation performed by a user, and receives an operation performed by the operation unit. An operation data acquisition unit that acquires data relating to a given operation as data in a format independent of a terminal, and transmits the acquired data to the management device, wherein the management device is received from the terminal device A control unit for starting the operation data acquisition unit is transmitted to the recording unit for recording the operation data and the terminal device, and the operation data acquisition unit is terminated for the terminal device. And an instruction unit that transmits a control signal for this purpose.

  (5) In order to solve the above problems, in the processing method according to the present invention, the instruction unit provided in the management device causes the terminal device to activate the operation data acquisition unit provided in the terminal device. The operation unit provided in the terminal device receives an operation performed by the user, and the operation data acquisition unit provided in the terminal device relates to the operation received by the operation unit. The data is acquired as data in a format independent of the terminal, the acquired data is transmitted to the management device, and the recording unit provided in the management device records the operation data received from the terminal device. The instruction unit provided in the management device transmits a control signal for ending activation of the operation data acquisition unit to the terminal device.

  ADVANTAGE OF THE INVENTION According to this invention, the data regarding the operation performed by the user with respect to a terminal device can be recorded effectively.

It is a block diagram which shows the mode of recording, editing, and reproduction | regeneration of the data regarding the schematic structure and operation of the processing system concerning one Embodiment of this invention. It is a block diagram which shows the schematic structure of the terminal device which concerns on one Embodiment of this invention. It is a block diagram which shows the schematic structure of the management apparatus which concerns on one Embodiment of this invention. It is a sequence diagram which shows an example of the procedure of the process of the recording of the operation data performed in the processing system which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the screen which displays the operation data of the format independent of a terminal which concerns on one Embodiment of this invention. It is a sequence diagram which shows an example of the procedure of the edit process of the operation data performed in the processing system which concerns on 1st Embodiment of this invention. It is a sequence diagram which shows an example of the procedure of the process of the recording of the operation data performed in the processing system which concerns on 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Hereinafter, Android will be described as an example.
Specifically, a processing program, a terminal device, a processing system, and a processing method described below are, for example, those of Android.

[Outline of Processing System According to Embodiment]
FIG. 1 is a block diagram showing recording, editing, and playback of data relating to a schematic configuration and operation of a processing system according to an embodiment of the present invention.
The processing system according to the present embodiment includes a terminal device 1, a management device 2, and a cable 3.

The terminal device 1 is a terminal device such as a smartphone or a tablet personal computer. In the present embodiment, a non-rooted (non-rooted) terminal device is used as the terminal device 1, but it is also possible to apply to a routed terminal device.
The management device 2 is a device external to the terminal device 1, and is an information processing device such as a notebook personal computer, for example.
The cable 3 connects the terminal device 1 and the management device 2 so that they can communicate with each other. In the present embodiment, a wired cable 3 composed of a USB (Universal Serial Bus) cable is used, but as another example, a configuration in which the terminal device 1 and the management device 2 communicate with each other wirelessly may be used. .

In the processing system according to the present embodiment, data related to an operation performed by the user is recorded on the terminal device 1, the recorded content is edited by the user, and reproduction is performed according to the recorded content before or after editing.
(A) At the time of recording, the terminal device 1 and the management device 2 are connected via the cable 3. Then, the user's hand 11 operates the terminal device 1, and data related to the operation is acquired and recorded by the management device 2 via the cable 3.
(B) At the time of editing, the terminal device 1 and the management device 2 may not be connected. Then, the management device 2 is operated with the user's hand 11 to edit the recorded content.
(C) At the time of reproduction, the terminal device 1 and the management device 2 are connected via the cable 3. Then, the management device 2 is operated by the user's hand 11 and the result corresponding to the operation is reproduced on the terminal device 1 according to the recorded content (recorded content before or after editing).
The terminal device 1 and the management device 2 are connected to, for example, an adb (Android Debug Bridge).

[Outline of Terminal Device According to Embodiment]
FIG. 2 is a block diagram showing a schematic configuration of the terminal device 1 according to the embodiment of the present invention.
The terminal device 1 according to the present embodiment includes an operation display unit 31, an operation data acquisition unit 32, an information processing unit 33, a communication unit 34, and a storage unit 35. The operation display unit 31 includes an operation unit 41 and a display unit 42.

The operation unit 41 receives an operation performed by the user.
The display unit 42 is configured using a screen that displays various types of information.
Here, as the operation unit 41, for example, a software unit (for example, a touch panel) that performs an operation by pressing an image such as a key, button, or icon displayed on the screen of the display unit 42 with a user's finger. Alternatively, a hardware device that performs an operation by pressing an actual key or button with a user's finger may be used.

The operation data acquisition unit 32 acquires data (operation data) related to the operation performed by the user and transmits the data to the management device 2 via the cable 3. In the present embodiment, the operation data acquisition unit 32 is configured using a function of an information collection application (an information collection application 103 described later).
The information processing unit 33 performs various types of information processing, for example, performs various types of information processing as a terminal device such as a smartphone or a tablet personal computer. In the present embodiment, the information processing unit 33 has an operation target function (an operation target function 104 described later).
The communication unit 34 communicates various signals with the management device 2. In the present embodiment, the communication unit 34 communicates with the management device 2 via the cable 3.
The memory | storage part 35 is comprised from memory, for example, and memorize | stores various data.

[Outline of Management Device According to Embodiment]
FIG. 3 is a block diagram showing a schematic configuration of the management apparatus 2 according to an embodiment of the present invention.
The management device 2 according to the present embodiment includes an operation unit 61, a display unit 62, a management unit 63, an information processing unit 64, a communication unit 65, and a storage unit 66. The management unit 63 includes an instruction unit 71, a recording unit 72, an editing unit 73, and a playback unit 74.
The operation unit 61 receives an operation performed by the user. The operation unit 61 is configured using, for example, a keyboard or a mouse.
The display unit 62 is configured using a screen that displays various types of information.

The management unit 63 records, edits, and reproduces operation data.
The instruction unit 71 transmits an instruction for the terminal device 1 via the cable 3.
The recording unit 72 performs a process of recording operation data.
The editing unit 73 performs an operation data editing process.
The reproduction unit 74 performs operation data reproduction processing.
In the present embodiment, the management unit 63 is configured using a management program (a management program 102 to be described later) and MonkeyRunner (see Non-Patent Document 2). Specifically, the management unit 63 uses the management program to record the instruction unit 71 and recording. The unit 72 and the editing unit 73 are configured, and the playback unit 74 is configured using MonkeyRunner.

The information processing unit 64 performs various types of information processing, for example, performs various types of information processing as an information processing device such as a notebook personal computer.
The communication unit 65 communicates various signals with the terminal device 1. In the present embodiment, the communication unit 65 communicates with the terminal device 1 via the cable 3.
The memory | storage part 66 is comprised from memory, for example, and memorize | stores various data.
In the present embodiment, the recording unit 72 records (stores) the operation data in the storage unit 66, the editing unit 73 records the edited operation data in the storage unit 66, and the playback unit 74 is recorded in the storage unit 66. Read operation data.

[First Embodiment]
A first embodiment will be described.
<Preparation>
In the terminal device 1 to be operated, an information collection application (information collection application) 103 for setting a transparent layer for capturing (acquiring) data (operation data) related to an operation performed by the user is Installed in the terminal device 1 and prepared.
In the present embodiment, in the terminal device 1 that is an operation target, a function that is originally an operation target is referred to as an operation target function (operation target function 104). Various functions (for example, applications) may be used as the operation target function 104.
As described above, in the present embodiment, the information collection application 103 captures data related to the operation performed on the operation target function 104 by the user.

  Here, the transparent layer set by the information collection application 103 can be displayed at a higher level than the normal screen. For example, WindowManager. TYPE_SYSTEM_ERROR, WindowManager. TYPE_SYSTEM_OVERLAY, WindowManager. TYPE_TOAST, WindowManager. TYPE_PHONE or the like can be used.

In addition, the information collection application 103 uses a transparent layer and a predetermined Android API (Application Programming Interface).
In the present embodiment, this API is used to recognize a terminal-independent operation.
As this API, for example, a GestureDetector class that detects a gesture for a touch event can be used. Examples of the touch event include a single tap (operation of pressing once), a double tap (operation of continuously pressing twice), a long press (operation of pressing for a predetermined time or more), scrolling (for example, a drag operation), and the like.
As this API, for example, the KeyEvent class can be used for key input (here, key input excluding the HOME key and the power key), and Android. intent. action. CLOSE_SYSTEM_DIALOGS broadcast can be used, and Android. intent. action. SCREEN_ON and Android. intent. action. A SCREEN_OFF broadcast can be used.

Thereby, the information collection application 103 can record data related to operations performed by the user on the terminal device 1 in a format independent of the terminal device model (terminal-independent format).
In the present embodiment, the information collection application 103 can record data related to all operations performed by the user on the terminal device 1.
In the present embodiment, the information collection application 103 records, for example, data related to an operation for starting an application (application), an operation (operation while the application is being started), and an operation for ending the application. It is possible to record data on the operation of any app.

As the information collection application 103, an application using a transparent layer is used in the present embodiment, but an opaque layer (non-transparent layer) may be used as another configuration example.
Further, as the information collection application 103, a configuration other than the layer may be used as another configuration example.

In the management apparatus 2, a management program (management program) 102 for executing processing related to operation data is installed and prepared.
In the present embodiment, the management program 102 executes operation data recording processing and operation data editing processing.
In the present embodiment, the operation data reproduction process is executed using a monkey runner (see Non-Patent Document 2), which is a terminal-independent reproduction tool. Various playback tools may be used. Further, the operation program reproduction process may be executed by the management program 102.

<Operation data recording process>
FIG. 4 is a sequence diagram illustrating an example of a procedure for recording operation data performed in the processing system according to the present embodiment.
FIG. 4 shows an example of a processing procedure performed by the user 101, the management program 102 of the management device 2, the information collection application 103 of the terminal device 1, and the operation target function 104 of the terminal device 1.
As shown in FIG. 1, the terminal device 1 and the management device 2 are connected via a cable 3 during recording.

First, the user 101 operates the operation unit 61 of the management apparatus 2 to input an instruction (notification) to start recording operation data to the management program 102 (process T1).
Upon receiving this instruction, the management program 102 (function of the instruction unit 71) sends a control signal (for example, a predetermined message signal) for starting the information collection application 103 of the terminal device 1 from the management device 2 to the cable 3. To the terminal device 1 (process T2).
When this control signal is received, the information collection application 103 is activated in the terminal device 1. As a result, the operation (user operation) performed by the user in the terminal device 1 is changed to a state in which the operation can be transmitted to the information collection application 103 (state T21).

Thereafter, when the user 101 operates the operation unit 41 of the terminal device 1 (process T3), the information collection application 103 captures the content of the operation using the transparent layer and the API. Specifically, the information collection application 103 recognizes and acquires all user operations in a terminal-independent format using an API.
Then, the operation (user operation) performed by the user 101 in the terminal device 1 changes to a state where it cannot be transmitted to the information collection application 103 (state T22).

  Specifically, for example, the information collection application 103 of the terminal device 1 is operated by one unit operation (for example, tap, double tap, drag, predetermined key) with the finger of the user 101 on the operation unit 41 of the terminal device 1. The operation (user operation) performed by the user 101 is detected when it is detected that the finger has completed the operation on the operation unit 41 of the terminal device 1. It changes to a state where it cannot be transmitted to the collection application 103. In this state, the operation of the user 101 with respect to the operation unit 41 of the terminal device 1 is not accepted, and neither recording nor actual operation (operation execution instruction for the operation target function 104) is performed. By performing such state switching, for example, a time lag (for example, a time interval between operations) can be provided for the operation performed by the user 101.

Thereafter, the information collection application 103 transmits (transfers) data including the captured user operation content (an example of operation data) from the terminal device 1 to the management device 2 via the cable 3 (process T4).
Here, in the present embodiment, the information collection application 103 converts the content of the captured user operation into a format that can be played back by a terminal-independent playback tool (function of the playback unit 74), and after the conversion Data including the contents of the user operation (an example of operation data) is transmitted to the management device 2.

  At this stage, the operation performed by the user 101 in the terminal device 1 is captured by the information collection application 103, but no operation is given to the operation target function 104.

The management program 102 (function of the recording unit 72) receives the operation data transmitted by the information collection application 103 of the terminal device 1, and records the received operation data in the storage unit 66 of the management device 2 (process T11). Thereby, the operation data is stored in the storage unit 66 of the management device 2.
Then, the management program 102 (function of the instruction unit 71) sends a control signal (for example, a predetermined message signal) for performing an operation corresponding to the recorded operation data in the terminal device 1 from the management device 2 to the cable 3. To the terminal device 1 (process T5). In the present embodiment, the management program 102 (function of the instruction unit 71) generates this control signal using a reproduction tool (function of the reproduction unit 74).
When this control signal is received, in the terminal device 1, the operation target function 104 executes processing corresponding to the operation corresponding to the received control signal. Thereby, the operation (operation T3) performed by the user 101 in the terminal device 1 is actually performed.

In addition, the management program 102 (function of the instruction unit 71) is used to release a state (screen lock state) in which a user operation cannot be transmitted to the information collection application 103 of the terminal device 1. A control signal (for example, a predetermined message signal) is transmitted from the management device 2 to the terminal device 1 via the cable 3 (process T6).
When this control signal is received, the operation (user operation) performed by the user in the terminal device 1 is changed to a state in which the operation can be transmitted to the information collection application 103 (state T23).

Here, when an operation is further performed by the user in the terminal device 1, the processes T3 to T6 described above are repeatedly performed (process T3 to process T6).
Thereafter, when completing the recording of the operation data, the user 101 operates the operation unit 61 of the management apparatus 2 to instruct (notify) the management program 102 to complete (end) the recording of the operation data. Is input (process T7).
Upon receiving this instruction, the management program 102 (function of the instruction unit 71) sends a control signal (for example, a predetermined message signal) for terminating the information collection application 103 of the terminal device 1 from the management device 2 to the cable 3. Through the terminal device 1 (process T8).
When this control signal is received, the information collection application 103 ends in the terminal device 1. As a result, the transparent layer is deleted.

FIG. 5 is a diagram illustrating an example of a screen 201 that displays operation data in a terminal-independent format according to the present embodiment.
Note that operation data corresponding to information displayed on the screen 201 is recorded in the storage unit 66 of the management apparatus 2.

Here, as the terminal-independent format, in this embodiment, a format that does not depend on the sensitivity of the terminal device model with respect to touch events and key operations is used. Specifically, as a terminal-independent format, mechanical sensitivity (for example, pressure required for operation) and software sensitivity (for example, duration and timing required for operation) of each terminal device are used. The format representing the contents (only) of the event of the operation performed is used without depending on.
The contents of the operation event include “tap”, “double tap”, “long press”, “drag”, “key press”, and the like. For example, information on the time when a screen such as a touch panel or an actual key is pressed (1 second, 10 seconds, etc.) and the timing when the pressing is continuously performed (pressed at intervals of 1 second, etc.) ) Is deleted, and each operation event content is regarded as one operation content.

  In the example of the screen 201 shown in FIG. 5, tap (400, 200), wait (500 ms), dtap (700, 350), wait (5000 ms), drag (400, 200 → 500, 300), wait (500 ms). , Press (HOME), wait (1000 ms),... (Data) are displayed. Then, event content data (information) of operations of tap (400, 200), d tap (700, 350), drag (400, 200 → 500, 300), press (HOME) is displayed.

Here, the value of the horizontal coordinate (for example, pixel position) of the screen of the display unit 42 of the terminal device 1 is represented by X, and the value of the vertical coordinate (for example, pixel position) of the screen is represented by Y.
In the present embodiment, tap (X, Y) is the position of the coordinate (X, Y) on the screen of the display unit 42 of the terminal device 1, that is, the horizontal coordinate value of the screen is X, and the vertical coordinate value. Represents a position where Y is tapped with a finger (or another part) of the user's hand 11. As a specific example, tap (400, 200) represents that the position of the coordinates (400, 200) on the screen is tapped by the user.

  Moreover, dtap (X, Y) is double-tapped by the finger (or other part) of the user's hand 11 at the position of the coordinate (X, Y) on the screen of the display unit 42 of the terminal device 1. Represents that. As a specific example, dtap (700, 350) represents that the position of the coordinates (700, 350) on the screen is double-tapped by the user.

  Also, assuming that (X1, Y1) and (X2, Y2) represent screen coordinates, drag (X1, Y1 → X2, Y2) is the screen coordinates (X1, Y1) of the display unit 42 of the terminal device 1. The position of the start point and the position of the coordinates (X2, Y2) of the screen as the end point, and the user's hand 11 (or another part) may be traced (dragged). Represent. As a specific example, drag (400, 200 → 500, 300) represents that the user has traced from the position of coordinates (400, 200) to the position of coordinates (500, 300).

  Further, assuming that K represents a name that identifies a key or button, press (K) represents that the key or button K has been pressed by the finger of the user's hand 11 (or may be another part). As a specific example, press (HOME) represents that the HOME key has been pressed by the user.

Further, with P as the length of time, wait (P) represents that the waiting time is P. As a specific example, wait (500 ms) represents that the waiting time is 500 ms.
Here, in this embodiment, the time interval at which the operation data is transmitted (transferred) from the terminal device 1 to the management device 2 in (Process T4) shown in FIG.
Specifically, the management program 102 of the management device 2 detects this time interval. That is, the terminal device 1 does not detect the waiting time P, but detects the data of the event contents of the operation and notifies the management device 2 of it.

<Operation data editing process>
FIG. 6 is a sequence diagram showing an example of the procedure of the operation data editing process performed in the processing system according to the present embodiment.
FIG. 6 shows an example of a procedure of processing performed by the user 101 and the management program 102 of the management apparatus 2.
As shown in FIG. 1, the terminal device 1 and the management device 2 do not have to be connected during editing.

First, the user 101 operates the operation unit 61 of the management apparatus 2 to input an instruction (notification) to start editing operation data to the management program 102 (process T51).
Upon receiving this instruction, the management program 102 (function of the editing unit 73) displays the operation data recorded in the storage unit 66 on the screen of the display unit 62 and presents the display contents to the user (processing T52). ). As an example, this screen displays information like a screen 201 shown in FIG.
Note that the management program 102 (the function of the editing unit 73) is, for example, viewed by the user 101 when the operation data recorded in the storage unit 66 cannot be viewed and understood by the user 101. After converting the data to a format that can be understood, it is displayed on the screen.

Then, the user 101 edits the operation data by operating the operation unit 61 of the management apparatus 2 while viewing the display content of the screen of the management apparatus 2 (process T53).
For data editing, for example, a function such as a text editor can be used.
The data editing may be performed manually by the user 101, for example, or may be performed by a device (management device) according to a procedure predetermined by a predetermined program (for example, the management program 102 or another program). It may be performed automatically according to 2).

Thereafter, the user 101 operates the operation unit 61 of the management apparatus 2 to input an instruction (notification) to complete (end) editing of the operation data to the management program 102 (process T54).
Upon receiving this instruction, the management program 102 (the function of the editing unit 73) ends the editing of the operation data, and records the edited operation data in the storage unit 66.
Here, for example, the management program 102 separately stores both the operation data (operation data before editing) received from the terminal device 1 and the operation data after editing that is the result of editing the operation data. The operation data after editing may be overwritten with the operation data before editing, and only the operation data after editing may be stored (recorded) in the storage unit 66.

In the present embodiment, in (process T4) shown in FIG. 4, the information collection application 103 of the terminal device 1 reproduces the contents of the captured user operation using a terminal-independent reproduction tool (function of the reproduction unit 74). Is converted to a format that can be performed, and the data including the contents of the user operation after the conversion (an example of operation data) is transmitted to the management apparatus 2. However, as another configuration example, It is also possible to perform the conversion at another timing.
As another example of such a configuration, in the processing T4 shown in FIG. 4, the information collection application 103 of the terminal device 1 converts data (an example of operation data) including the contents of the captured user operation into a predetermined format (an example of operation data). In this case, the terminal-independent playback tool (format that cannot be played back by the playback unit 74) is transmitted to the management device 2, and the management program 102 (the function of the editing unit 73) of the management device 2 Before or after editing the received operation data, use a configuration in which the operation data is converted into a format that can be played back by a terminal-independent playback tool (function of the playback unit 74) and recorded in the storage unit 66. it can.

<Operation data playback process>
As shown in FIG. 1, the terminal device 1 and the management device 2 are connected via a cable 3 at the time of reproduction.
First, the user 101 operates the operation unit 61 of the management apparatus 2 to input an instruction (notification) for starting reproduction of operation data to the management program 102. At this time, for example, the user 101 designates operation data to be reproduced from one or more pieces of operation data recorded in the storage unit 66 of the management device 2 and the number of times of reproduction of the designated operation data. Can be specified.

Upon receiving this instruction, the management program 102 (function of the playback unit 74) sends a control signal (for example, a predetermined message signal) for performing an operation corresponding to the instructed (designated) operation data in the terminal device 1. Then, the data is transmitted from the management device 2 to the terminal device 1 via the cable 3.
When this control signal is received, in the terminal device 1, the operation target function 104 executes processing corresponding to the operation corresponding to the received control signal. Thereby, in the terminal device 1, the operation described in the operation data (operation of one or more event contents) is actually performed.

  As an example, in the case where operation data such as information displayed on the screen 201 shown in FIG. 5 is reproduced on the terminal device 1, an event content operation “tap (400, 200)” on the terminal device 1, dtap ( 700, 350), an event content operation called drag (400, 200 → 500, 300), an event content operation called press (HOME), and so on. Further, when the number of reproductions of the operation data is set to N (N is an integer equal to or greater than 1), the terminal device 1 performs N reproduction (execution of a series of operations) based on the operation data. Performed (eg continuously).

<Specific Example of Editing Operation Data 1: Adjustment of wait (P)>
Specific example 1 of editing operation data is shown.
Consider a case where the terminal device 1 has a plurality of network interface (network interface) functions. Also, consider a case where the terminal device 1 records, edits, and reproduces operation data of an application that performs communication. Various networks may be used as the plurality of networks. For example, WiFi (Wireless Fidelity), 3GPP (Third Generation Partnership Project), or the like can be used.

In this case, first, operation data of an application that performs communication using one network interface function among a plurality of network interface functions of the terminal apparatus 1 is acquired and transmitted from the terminal apparatus 1 to the management apparatus 2. And record.
Next, the management device 2 edits the recorded operation data. In this example, the user 101 applies the value of wait (P) corresponding to the time interval of each operation included in the recorded operation data to the network interface to which the user 101 applies (here, the recorded network interface is Change according to another network interface).

  As an example, as a comparative communication speed, operation data acquired and recorded when a function of an interface of a network with a low communication speed (for example, 3GPP) is used in the terminal device 1 is recorded as a network with a high communication speed (for example, , WiFi) interface function is considered to be changed (edited) to operation data applied when the terminal device 1 is used. In this case, the value of P of wait (P) (for example, 500 ms) included in the operation data recorded using the network interface function with a low communication speed is set to a shorter time value (for example, 50 ms). ), And a reproduction instruction based on the changed operation data is transmitted from the management device 2 to the terminal device 1 using the network interface function with a high communication speed.

  In this way, the operation data when using the function of one network interface in the terminal device 1 is recorded, and the time interval between the operations included in the recorded operation data (in this example, P of wait (P)) ) To match (adjust) the function of another network interface, the changed operation data can be used for reproduction when the terminal device 1 uses the function of the other network interface. . Therefore, for example, for a plurality of network interfaces, it is possible to eliminate the need to acquire and record operation data for each network interface, and to improve efficiency.

Note that the value P of wait (P) may be set arbitrarily. For example, it is preferable to set a value slightly longer than the actual time interval value.
In addition, the editing process as in this example is not limited to the case where the same terminal device 1 switches and uses a plurality of different network interface functions, but the plurality of terminal devices 1 use different network interface functions. It can also be applied to such cases.

<Specific Example 2 of Editing Operation Data: Adjustment of Coordinates (X, Y)>
Specific example 2 of editing operation data is shown.
Consider a case where operation data composed of the same series of operations (operations of a plurality of event contents) is reproduced in a plurality of types of terminal devices 1. Further, consider a case where the terminal device 1 records, edits, and reproduces operation data of an application that uses coordinates (X, Y) such as tap, double tap, and drag.

In this case, first, the operation data of the application is acquired in one type of terminal device 1 among the plurality of types of terminal devices 1, and is transmitted from the terminal device 1 to the management device 2 and recorded.
Next, the management device 2 edits the recorded operation data. In this example, the user 101 applies a coordinate value to an operation that uses coordinates among the operations included in the recorded operation data (in this case, the terminal device 1 that has performed recording). Change according to the terminal device 1) of another model. Here, as coordinate values, for example, the values of X and Y of tap (X, Y), the values of X and Y of dtap (X, Y), and drag (X1, Y1 → X2, Y2) There are values for X1, Y1, X2, and Y2.

  As an example, the operation data acquired and recorded in the terminal device 1 of the first model having the screen of the first size as a comparative screen size (horizontal length or vertical length), Consider a case where the operation data is changed (edited) to be applied to the terminal device 1 of the second model having the second size screen (a screen size different from the first size screen). In this case, the X and Y values of tap (X, Y) included in the operation data acquired and recorded in the terminal device 1 of the first model having the screen of the first size, dtap (X, Y) X and Y values and drag (X1, Y1 → X2, Y2) X1, Y1, X2, and Y2 values in the terminal device 1 of the second model having the second size screen. The screen is changed (edited) so as to be enlarged or reduced in accordance with the second size of the screen, and a reproduction instruction based on the changed operation data is sent from the management device 2 to the second size screen having the second size. It transmits to the terminal device 1 of 2 types.

As a specific example, the first size of the screen of the terminal device 1 of the first model is 50 × 100 in the coordinate value, and the second size of the screen of the terminal device 1 of the second model. Is the coordinate value of 40 × 80 in the horizontal direction, the value of the horizontal coordinate X (or X1 or X2) included in the operation data acquired and recorded in the terminal device 1 of the first model and The value of the vertical coordinate Y (or Y1 or Y2) is changed (edited) to a value of 0.8 times.
The same applies to the case where the aspect ratio is different between the screen of the terminal device 1 of the first model and the screen of the terminal device 1 of the second model. As a specific example, the first size of the screen of the terminal device 1 of the first model is 60 × 100 in the coordinate value, and the second size of the screen of the terminal device 1 of the second model. Is the horizontal value 72 × vertical value 130, the value of the horizontal coordinate X (or X1 or X2) included in the operation data acquired and recorded in the terminal device 1 of the first model is used. The value is changed (edited) to a value of 1.2 times, and the value Y (or Y1 or Y2) of the vertical coordinate is changed (edited) to a value of 1.3 times.

  As described above, the operation data is recorded for the terminal device 1 of one model in the terminal device 1, and the coordinate value of each operation included in the recorded operation data is adjusted (adjusted) to the terminal device 1 of another model. The operation data after the change can be used for reproduction in the terminal device 1 of the other model. For this reason, for example, it is possible to eliminate the necessity of acquiring and recording operation data for each model of the terminal devices 1 of a plurality of models, and to improve efficiency.

<Specific example of operation data playback>
The terminal device 1 will be described by taking as an example a case where quality inspection such as error occurrence or throughput (communication speed) is performed.
As an example, the same operation data is reproduced many times (for example, 10,000 times) in the same terminal device 1. For example, if a small number of times (for example, two or three times) of errors (for example, different screen displays) occur in the large number of times of reproduction, this is reported. In this case, information when an error (for example, a different screen display) occurs (for example, a captured image of the screen when the error occurs) can be acquired and recorded by the playback tool.
As another example, an operation data composed of a certain combination of a plurality of operations such as a tap and a double tap is measured on an application for measuring throughput in a plurality of different terminal devices 1 (for example, terminal devices 1 of different models). It is possible to measure the throughput in each terminal device 1 by reproducing (executing) with.

<Summary of First Embodiment>
As described above, in the processing system according to the present embodiment, the information collection application 103 of the terminal device 1 captures data (operation data) related to the operation performed by the user 101 by operating the operation unit 41 of the terminal device 1 ( Acquired) and transmitted to the management program 102 of the management apparatus 2, and the management program 102 (function of the recording unit 72) of the management apparatus 2 records the operation data in the storage unit 66. Also, the information collection application 103 of the terminal device 1 or the management program 102 (function of the editing unit 73) of the management device 2 can reproduce the operation data with a terminal-independent reproduction tool (function of the reproduction unit 74). To a different format Further, the management program 102 (function of the editing unit 73) of the management apparatus 2 edits the operation data, and the management program 102 (function of the reproduction unit 74) of the management apparatus 2 reproduces the operation data.

Therefore, according to the processing system concerning this embodiment, the data (operation data) about operation performed by user 101 to terminal unit 1 can be recorded effectively.
Further, according to the processing system according to the present embodiment, the recorded operation data can be edited by the management device 2 and effectively used for reproduction.
Further, according to the processing system according to the present embodiment, for example, the hardware such as the terminal device 1 can be realized by installing software using the same hardware as the conventional one.

  As a specific example, according to the processing system according to the present embodiment, “operation data recorded in a terminal device of one model is edited and the edited operation data is used, which is difficult with the technique according to Non-Patent Document 1. It is possible to easily “play on another type of terminal device”. Thereby, according to the processing system concerning this embodiment, when reproducing the same operation (operation realized by the same operation) in a plurality of different types of terminal devices 1, operation data is recorded for each model. Compared with the structure which is equal, it is possible to reduce a user's effort significantly. Further, according to the processing system according to the present embodiment, even in the terminal device 1 of the same model, when reproduction is performed using a plurality of different network interface functions, operation data is recorded for each network interface. Compared with the structure which is equal, it is possible to reduce a user's effort significantly.

As a specific example, the technique according to Non-Patent Document 2 does not support operation recording such as double tap and long press. However, according to the processing system according to the present embodiment, operation recording such as double tap and long press is also performed. It is possible to record all user operations including. Thereby, according to the processing system concerning this embodiment, recording, editing, and reproduction of various operations are possible.
Furthermore, in the technology according to Non-Patent Document 2, it is necessary to record the operation in the external device connected to the terminal device. However, according to the processing system according to this embodiment, the operation performed by the user 101 in the terminal device 1 is directly performed. For example, the operation can be recorded more accurately, and the user 101 can easily understand the usage method.

  Here, in the present embodiment, as shown in FIG. 4, data (operation data) related to an operation performed by the user 101 in the terminal device 1 is captured by the information collection application 103 of the terminal device 1 and managed by the management device 2. After being transmitted to the program 102 (in the present embodiment, after the operation data is further recorded in the storage unit 66 of the management apparatus 2), the management program 102 of the management apparatus 2 applies to the operation target function 104 of the terminal apparatus 1 Although the configuration is such that an operation execution instruction is issued, as another configuration example, when an operation is performed by the user 101 in the terminal device 1, processing corresponding to the operation is performed by the operation target function 104 of the terminal device 1. In parallel with this, the information collection application 103 of the terminal device 1 captures and manages data (operation data) related to the operation performed by the user 101. It is also possible to use a structure for transmitting to the management program 102 of the device 2.

[Second Embodiment]
A second embodiment will be described.
In the present embodiment, operation data recording processing is different from that in the first embodiment.
Therefore, the operation data recording process according to the present embodiment will be described in detail, and detailed description of the same parts as those in the first embodiment will be omitted. Further, the same processing units as those in the first embodiment will be described with the same reference numerals.

<Operation data recording process>
FIG. 7 is a sequence diagram illustrating an example of a procedure for recording operation data performed in the processing system according to the present embodiment.
FIG. 7 shows an example of a processing procedure performed by the user 101, the management program 102 of the management device 2, the information collection application 103 of the terminal device 1, and the operation target function 104 of the terminal device 1.
As shown in FIG. 1, the terminal device 1 and the management device 2 are connected via a cable 3 during recording.

Here, the procedure of the operation data recording process performed in the processing system according to this embodiment is compared with the procedure of the operation data recording process shown in FIG. 4 according to the first embodiment (state T22). , (State T23) and (process T6) are different in that the state switching process in the terminal device 1 is excluded, and the other processes are the same.
Specifically, in the present embodiment, when the information collection application 103 is activated in the terminal device 1, an operation (user operation) performed by the user 101 can always be transmitted to the information collection application 103. And
As described above, the processing system according to the present embodiment may be implemented.

[Configuration example of the above embodiment]
In the above embodiment, the terminal device 1 is configured using a computer, stores a processing program to be executed by the computer in the storage unit 35, and causes the computer to execute the processing program. Process various steps.

  As one configuration example, an operation unit (the operation unit 41 in the above embodiment) provided in the terminal device (in the above embodiment, the terminal device 1) is performed by the user (in the above embodiment, the user 101). A step of accepting an operation (in the above embodiment, the step of the process T3 shown in FIGS. 4 and 7), and an operation data acquisition unit (in the above embodiment, the operation data acquisition unit 32) provided in the terminal device. ) Is a step of acquiring data related to the operation received by the operation unit as data in a format independent of the terminal (in the above embodiment, the step of the process T3 shown in FIGS. 4 and 7); The operation data acquisition unit provided in the terminal device transmits the acquired data to an external device (the management device 2 in the above embodiment) In the form, the steps) of processing T4 shown in FIGS. 4 and 7, the computer (or more embodiments, a processing program to be executed by a computer) of the terminal apparatus 1.

  As one configuration example, in the processing program as described above, when the operation data acquisition unit provided in the terminal device acquires data related to an operation received by the operation unit as data in a format independent of the terminal , A step of switching the operation accepted by the operation unit to a state where the operation data acquisition unit cannot be transmitted (in the above embodiment, a step of switching to the state T22 shown in FIG. 4), and an instruction from the external device In the above embodiment, the step of switching the operation accepted by the operation unit to a state in which the operation data can be transmitted to the operation data acquisition unit according to the instruction of the process T6 shown in FIG. 2 (in the above embodiment, FIG. A step of switching to the state T23 shown in FIG. It is a non.

  As one configuration example, an operation unit (operation unit 41 in the above embodiment) that receives an operation performed by a user (in the above embodiment, the user 101), and data related to the operation received by the operation unit are stored in the terminal Operation data acquisition unit (in the above embodiment, operation data acquisition unit 32) that acquires the acquired data as data in a format independent of the data and transmits the acquired data to an external device (in the above embodiment, the management device 2) A terminal device (in the above embodiment, the terminal device 1).

  As one configuration example, a terminal device (in the above embodiments, the terminal device 1) and a management device (in the above embodiments, the management device 2) are included, and the terminal device is a user (in the above embodiments, a user). 101), the operation unit (in the above embodiment, the operation unit 41) that receives the operation performed by (101) and the data related to the operation received by the operation unit are acquired and acquired as data in a format independent of the terminal. An operation data acquisition unit (in the above embodiment, the operation data acquisition unit 32) that transmits data to the management device, and the management device records the operation data received from the terminal device ( In the above embodiment, a control signal for starting the operation data acquisition unit is transmitted to the recording unit 72) and the terminal device, and to the terminal device, (In the above embodiments, the instruction unit 71) instructing unit for transmitting a control signal to terminate the activation of the serial operation data acquisition unit includes a, a, a processing system, characterized in that.

  As an example of the configuration, an instruction unit (in the above embodiment, the instruction unit 71) provided in the management device (in the above embodiment, the management device 2) is a terminal device (in the above embodiment, the terminal device 1). In contrast, an operation data acquisition unit provided in the terminal device (in the above embodiment, the operation data acquisition unit 32) is transmitted to send a control signal, and the operation unit provided in the terminal device (above In the embodiment, the operation unit 41) receives an operation performed by a user (in the above embodiment, the user 101), and the operation data acquisition unit provided in the terminal device is received by the operation unit. Data relating to the operation is acquired as data in a format independent of the terminal, the acquired data is transmitted to the management device, and the recording unit provided in the management device (in the above embodiments, Recording unit 72) records the operation data received from the terminal device, and the instruction unit (in the above embodiment, the instruction unit 71) provided in the management device performs the operation on the terminal device. A processing method is characterized in that a control signal for ending activation of a data acquisition unit is transmitted.

[Summary of the above embodiments]
As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

  For example, a program for realizing the function of each device (for example, the terminal device 1 or the management device 2) according to each embodiment described above is recorded on a computer-readable recording medium and recorded on the recording medium. The program is loaded into the computer system and executed.

The “computer system” mentioned here may include an operating system (OS) and hardware such as peripheral devices.
The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a writable nonvolatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disk), A storage device such as a hard disk built in a computer system.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (DRAM)) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Dynamic Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 1 ... Terminal device, 2 ... Management apparatus, 3 ... Cable, 11 ... User's hand, 31 ... Operation display part, 32 ... Operation data acquisition part, 33, 64 ... Information processing part, 34, 65 ... Communication part, 35, 66: Storage unit, 41, 61 ... Operation unit, 42, 62 ... Display unit, 63 ... Management unit, 71 ... Instruction unit, 72 ... Recording unit, 73 ... Editing unit, 74 ... Playback unit, 101 ... User, 102 ... Management program 103 ... Information collection application 104 ... Operation target function 201 ... Screen

Claims (5)

A step of receiving an operation performed by a user by an operation unit provided in the terminal device;
An operation data acquisition unit provided in the terminal device acquires data relating to an operation received by the operation unit as data in a format independent of the terminal;
The operation data acquisition unit provided in the terminal device transmits the acquired data to an external device;
Processing program for causing a computer to execute. When the operation data acquisition unit provided in the terminal device acquires data related to the operation received by the operation unit as data in a format independent of the terminal, the operation received by the operation unit is the operation data acquisition A step of switching to a state incapable of being transmitted to the part,
In response to an instruction from the external device, the step of switching to a state in which an operation accepted by the operation unit can be transmitted to the operation data acquisition unit;
The processing program according to claim 1, for causing the computer to execute. An operation unit that receives an operation performed by the user;
An operation data acquisition unit that acquires data related to an operation received by the operation unit as data in a format independent of the terminal, and transmits the acquired data to an external device;
A terminal device comprising: A terminal device and a management device;
The terminal device acquires an operation unit that receives an operation performed by a user and data related to the operation received by the operation unit as data in a format independent of the terminal, and transmits the acquired data to the management device An operation data acquisition unit
The management device transmits a control signal for starting the operation data acquisition unit to the recording device for recording the operation data received from the terminal device and the terminal device, and also transmits the control signal to the terminal device. On the other hand, an instruction unit that transmits a control signal for ending activation of the operation data acquisition unit,
A processing system characterized by that. The instruction unit provided in the management device transmits a control signal for starting the operation data acquisition unit provided in the terminal device to the terminal device,
The operation unit provided in the terminal device accepts an operation performed by a user,
The operation data acquisition unit provided in the terminal device acquires data related to the operation received by the operation unit as data in a format independent of the terminal, and transmits the acquired data to the management device.
The recording unit provided in the management device records the operation data received from the terminal device,
The instruction unit provided in the management device transmits a control signal for ending activation of the operation data acquisition unit to the terminal device.
A processing method characterized by the above.

Images