JP5640035B2 - Operation log collection method, operation log collection device, operation log collection program - Google Patents

Description

  The present invention relates to a log operation log collection method, an operation log collection device, and an operation log collection program suitable for use in a terminal device that operates using a touch panel display.

  In smartphones and tablet terminals, various operations are performed when a user touches a finger on a touch panel display. In particular, in recent touch panel displays, contact points of a plurality of fingers can be detected, and various operations can be performed by moving the finger contact positions and fingers. Typical operations include scrolling by sliding a finger on the touch panel display, pinching in and pinching out by touching two fingers on the touch panel display, and closing or spreading the fingers.

  If a user browses content using a smartphone or tablet device, if the user is interested in what content, how long and what content is viewed, the user's interests and trends Can be used to recommend content and advertisements that match the user's interests. In addition, analysis results of user interests and trends can be used to create content. In the case of manual contents, it is possible for the worker to grasp whether the work has been performed while checking the manual in order.

  When the user is viewing the electronic file displayed on the display terminal, the estimated location on the display screen that the user is gazing at is determined based on the position of the scroll bar and the coordinates of the mouse. And a method for estimating the user's gaze point from the residence time for each display range and the mouse reaction has been proposed (Patent Document 1).

JP 2008-198171 A

  However, in the case of Patent Document 1, the gaze point is estimated from the residence time and the mouse reaction. In the method of Patent Document 1, it is difficult to analyze the operation on the touch panel display.

  Therefore, it is conceivable to store the position operated on the view screen of the touch panel display as an operation log. However, as described above, the operation of the touch panel display is an operation of touching a predetermined position on the screen, such as scrolling, pinching in, and pinching out, and it is necessary to know which place is touched and how. The meaning of the operation cannot be determined. Further, the position of the displayed content is changed by a scrolling operation, and the size thereof is changed by a pinch-in and pinch-out operation. Therefore, even if the touch coordinates on the view screen of the touch panel display are left in the operation log, it is not possible to analyze what operation corresponds to what position on the content. difficult.

  In view of the above-described problems, the present invention provides an operation log collection method, an operation log collection apparatus, and an operation log collection that can easily analyze what operation is performed on which part of content when performing an operation on a touch panel display. The purpose is to provide a program.

In view of the above-described problems, the operation log collection method according to the present invention includes a first process of extracting a partial area of content and displaying it on a touch panel display in response to a user operation, and the first process. The second process of detecting the position of the user operation on the touch panel display of the displayed content image, and the position of the user operation on the touch panel display detected in the second process are displayed on the content. into a position based on the position of the user operation, to determine the operation type, a third process to be stored in the converted position and the operation type and operation log storage unit in association with respect to the content , organic and fourth step of displaying overlapping an identification display indicating the operation type in which the operation log storage unit is stored in correspondence to the position to the position stored, the And wherein the Rukoto.

The operation log collecting apparatus according to the present invention extracts a partial area of content and displays the content on the touch panel display according to a user operation, and a position of the user operation on the touch panel display of the content image. An operation log calculation unit that detects and converts the detected position of the user operation on the touch panel display into a position on the content, and determines an operation type based on the position of the user operation, and the converted position And an operation log storage unit that stores the operation type in association with the operation type, and an operation type that is stored in association with the position at the position stored in the operation log storage unit for the content And a display function unit that displays an identification display indicating the above.

The operation log collection program according to the present invention includes a first step of extracting a partial area of content and displaying it on a touch panel display in response to a user operation, and an image of the content displayed in the first step. a second step of detecting a position of a user operation on the touch panel display, the position of the user operation on the touch panel display detected by the second step, is converted into a position on the content, the A third step of determining an operation type based on a position of a user operation, associating the converted position and the operation type and storing them in the operation log storage unit; and for the content, the operation log storage unit Is a fourth step in which an identification display indicating the operation type stored in association with the position is superimposed on the stored position. Characterized in that it comprises a flop, a.

  According to the present invention, touch coordinates on the view screen of the touch panel display operated by the user are converted into touch coordinates on the content and stored as an operation log. The touch coordinates on the content stored as the operation log are visually displayed. For this reason, it is possible to intuitively grasp the user's operation on the touch panel display.

It is a functional block diagram which shows an example of the terminal device which concerns on the 1st Embodiment of this invention. It is explanatory drawing of the content memory | storage part in the terminal device which concerns on the 1st Embodiment of this invention. It is explanatory drawing of the operation log memory | storage part in the terminal device which concerns on the 1st Embodiment of this invention. It is a figure which shows the relationship between the display area of a content, and the view area on a touchscreen display. It is explanatory drawing of the coordinate transformation in the case of touch operation. It is explanatory drawing of the coordinate transformation in the case of scroll operation. It is explanatory drawing of the coordinate transformation in the case of multi-touch operation. It is explanatory drawing of the coordinate transformation in the case of multi-touch slide operation. It is explanatory drawing of the finger line log of pinch operation. It is explanatory drawing of the finger line log on a content image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The terminal device according to the present embodiment can store the content displayed on the touch panel display and the operation performed by the user on the content as an operation log. Here, the operation of the touch panel display means that the user touches the touch panel display to scroll, or touches the touch panel display with two fingers and opens or closes the finger (pinch-in / pinch-out). ), An operation for enlarging or reducing the content.

  In the present embodiment, touch coordinates on the view screen of the touch panel display are converted into touch coordinates on the content and stored. That is, the operation performed on the touch panel display by the user is an operation on the view screen of the touch panel display, and this operation is based on the physical position (coordinates) touched by the user on the view screen of the touch panel display. . However, the content displayed on the view screen of the touch panel display can be scrolled, enlarged, or reduced, and the position where the operation on the content is performed is the physical position on the touch panel display. Change. For this reason, even if the touch coordinates on the view screen of the touch panel display operated by the user are left as a log, it is difficult to analyze where the user is interested in the content and what operation is performed.

  Therefore, in this embodiment, touch coordinates on the touch panel display operated by the user are converted into touch coordinates on the content and stored as an operation log. In the present embodiment, the touch coordinates on the content saved as the operation log are visually displayed so that the analysis can be easily performed.

  FIG. 1 is a functional block diagram showing an example of a terminal device according to the first embodiment of the present invention. As shown in FIG. 1, the terminal device 1 according to the present embodiment includes a touch panel function unit 11, a display function unit 12, a viewer unit 13, an operation log calculation unit 14, a content storage unit 15, and an operation log storage. Section 16 and an operation log display section 17.

  The touch panel function unit 11 is a part that is operated by a user touching a finger, and detects coordinates on a view screen of a touch panel display touched by the user. Input operations to the touch panel function unit 11 include “touch”, “slide”, “multi-touch”, “multi-touch slide”, and the like. The touch panel function unit 11 acquires touch coordinates on the view screen of the touch panel display operated by the user.

  The display function unit 12 is a part that extracts and displays a partial region of the content sent from the viewer unit 13. The touch panel function unit 11 described above is stacked on the display function unit 12.

  The viewer unit 13 reads the content from the content storage unit 15, enlarges, reduces, or slides (scrolls) the content according to the input from the touch panel function unit 11, and sends the display image to the display function unit 12. Further, the viewer unit 13 sends display parameters such as a display margin, a display scale, and scroll coordinates, and touch coordinates to the operation log calculation unit 14.

  The operation log calculation unit 14 is based on display parameters acquired from the viewer unit 13 such as display margin, display scale, and scroll coordinates, and touch coordinates detected by the user touching the touch panel function unit 11. The touch coordinates on the view screen of the touch panel display operated by is converted to the touch coordinates on the content and stored in the operation log storage unit 16. Further, the operation log calculation unit 14 determines the operation type, and outputs it to the operation log storage unit 16 in association with the content ID, the time, and the operation type. Note that the calculation at the time of conversion from touch coordinates on the view screen of the touch panel display to touch coordinates on the content will be described later.

  As shown in FIG. 2, the content storage unit 15 stores content data in association with the content ID.

  The operation log storage unit 16 stores the operation log obtained by the operation log calculation unit 14. As shown in FIG. 3, the operation log stores a content ID, a time, an operation type, and coordinates. Here, the operation types are “touch”, “scroll”, “multi-touch”, and “multi-touch slide (pinch in, pinch out)”. Here, the coordinates are coordinates converted into touch coordinates on the content. In the case of “touch”, this coordinate is one coordinate, but in the case of “scroll”, the coordinate is two coordinates of a scroll start point and an end point. In the case of “multi-touch”, there are two or more touched points. In the case of “multi-touch slide (pinch-in, pinch-out)”, there are two touched points as the start point and two touched points as the end point.

  The operation log display unit 17 displays the operation log on the content stored in the operation log storage unit 16 on the content image stored in the content storage unit 15. Here, a finger line indicating the touch position on the content is drawn based on the coordinates of the user's operation position converted into the touch coordinates on the content stored in the operation log storage unit 16.

  As described above, in the present embodiment, the operation log calculation unit 14 converts the touch coordinates on the view screen of the touch panel display operated by the user into the touch coordinates on the content and stores them in the operation log storage unit 16. ing. For this reason, it is possible to analyze a browsing tendency such as where the user is interested in the content, how long, and what operation has been performed.

  Next, a process for converting the touch coordinates on the view screen of the touch panel display to the touch coordinates on the content in the operation log calculation unit 14 will be described.

  FIG. 4 shows the relationship between the content display area and the view area on the touch panel. In FIG. 4, a view area 101 indicates an area of a display portion of the touch panel display. The view area 101 is determined by the physical size of the touch panel display. The view area 101 is a touch area that is operated by a user touching a finger. The touch panel function unit 11 detects physical coordinates with respect to the view origin at the upper left corner of the view area 101.

  A display area 102 indicates a content area including a portion (margin) displayed on the outer edge of the content when the content is displayed on the touch panel display. The display area 102 is managed by the viewer unit 13, and the size of the display area 102 can be enlarged or reduced. The position on the display area 102 is two-dimensional from the display area origin at the upper left corner of the display area 102. Determined by coordinates.

  In the example of FIG. 4, the size of the display area 102 is larger than the view area 101, and a part of the content is cut out and displayed in the view area 101. The entire display area 102 can be scrolled vertically and horizontally.

  The content area 104 is a content area. The background area 105 is a margin area. The background area is a margin area excluding the content area from the display area. The display margin is a margin that can be given vertically and horizontally when content is displayed in the display area 102. Here, the display margin is indicated by (xm, ym). The size of the display margin changes due to scrolling, enlargement, and reduction operations.

  The scroll coordinates (Xs, Ys) indicate the position of the upper left corner of the view area 101 from the upper left corner of the display area 102. The scroll coordinates (Xs, Ys) are given from the viewer unit 13 and change when the screen is scrolled, enlarged or reduced.

  The display scale S indicates the ratio of enlargement / reduction when displaying content. When the display scale S is increased, it is enlarged, and when it is reduced, it is reduced.

  Touch coordinates (Xv, Yv) on the view screen of the touch panel display are coordinates detected when the user touches the view area 101 with a finger. Touch coordinates (Xv, Yv) on the view screen of the touch panel display are two-dimensional coordinates with the upper left corner of the view area 101 as the origin.

  As will be described later, the operation log calculation unit 14 obtains the touch coordinates (Xct, Yct) on the content using these values. The touch coordinates (Xct, Yct) on the content are two-dimensional coordinates with the upper left corner of the content area as the origin.

  Next, for each operation, a process of converting touch coordinates on the view screen of the touch panel display into touch coordinates on the content and storing them as operation logs will be described. Here, “touch”, “scroll”, “multi-touch”, and “multi-touch slide” will be described as operations.

  “Touch” is an operation in which the user touches a finger at a predetermined position on the touch panel. “Scrolling” is an operation in which the user touches a predetermined position on the touch panel and moves the finger up and down or left and right as it is. “Multi-touch” is an operation in which the user touches two or more points on the touch panel simultaneously with a finger. “Multi-touch slide” is an operation in which the user touches the touch panel with two fingers to open or close the fingers. For example, an operation of touching two or more fingers, such as five, and opening or closing the fingers may be included in the “multi-touch slide”.

  First, conversion in the case of “touch” will be described. In the case of “touch”, there is no change in the position or size of the content before and after the touch. FIG. 5 shows conversion in the case of “touch”.

  As shown in FIG. 5, in “touch”, the scroll coordinates are (Xs, Ys), the content margin is (Xm, Ym), the display scale is S, and the touch coordinates on the view screen of the touch panel display are (Xv). , Yv), the touch coordinates (Xct, Yct) on the content are

Can be calculated as In the case of “touch”, the touch coordinates (Xct, Yct) on the content thus obtained are stored in the operation log storage unit 16 as an operation log.

  Next, the case of scrolling will be described. As shown in FIGS. 6A and 6B, in the case of “scrolling”, the screen position with respect to the content changes when it is slid in the arrow direction from the touch point. Here, when the scroll coordinates scrollA before scrolling are (Xsa, Ysa) and the scroll coordinates scrollB after scrolling are (Xsb, Ysb), the touch coordinates (Xcs, Ycs) on the content after scrolling are:

Can be calculated as In scrolling, on the content, assuming that the point of interest has moved from the touch coordinates (Xct, Yct) to the touch coordinates (Xcs, Ycs), the touch coordinates (Xct, Yct) of the start point on the content and the touch coordinates of the end point on the content (Xcs, Ycs) is recorded in the operation log storage unit 16 as an operation log.

  Next, the case of multi-touch will be described. As shown in FIG. 7, in the case of multi-touch, the scroll coordinates are (Xs, Ys), the content margin is (Xm, Ym), the display scale is S, and a plurality (here, If the two touch coordinates are (Xv1, Yv1) and (Xv2, Yv2), the two touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on the content are

Can be calculated as In the case of multi-touch, a plurality of touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on the plurality of (here, two) contents are recorded in the operation log storage unit 16 as operation logs.

  Next, the case of multi-touch slide (pinch-in / pinch-out) will be described. As shown in FIGS. 8A and 8B, in the case of multi-touch slide, the size of the display area 102 with respect to the view area 101 changes. Here, the two start points on the content of the pinch operation are (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2), and the two end points on the content of the pinch operation are (Xcms1, Ycms1) and (Xcms2, Ycms2) To do.

  The two starting points (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) can be calculated in the same manner as the multi-touch described above. That is, the scroll coordinates are (Xs, Ys), the content margin is (Xm, Ym), the display scale is S, and the start points of the two touch coordinates on the view screen of the touch panel display are (Xv1, Yv1) and (Xv2). , Yv2), the two touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on the content are

Can be calculated as

  The two end points (Xcms1, Ycms1) and (Xcms2, Ycms2) are the scroll coordinates before scrolling (Xsa, Ysa), the scroll coordinates after scrolling (Xsb, Ysb), and the margin before scrolling (Xma, Yma). When the margin after scrolling is (Xmb, Ymb), the display scale before the pinch operation is Sa, and the display scale after the pinch operation is Sb,

Can be calculated as In the case of “multi-touch slide”, two touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on the content before the pinch operation, and two touch coordinates (Xcms1, Ycms1) on the content after the pinch operation. ) And (Xcms2, Ycms2) are recorded in the operation log storage unit 16 as operation logs.

  As described above, in this embodiment, operations such as “touch”, “scroll”, “multi-touch”, “multi-touch slide” are calculated as coordinates on the content, and as shown in FIG. The content is stored in the operation log storage unit 16 in correspondence with the content ID and time.

  Next, operation log display processing in the operation log display unit 17 will be described. As described above, in this embodiment, in the “touch” operation, the touch coordinates (Xct, Yct) on the content are stored in the operation log storage unit 16. In the “scroll” operation, the touch coordinates (Xct, Yct) on the content at the scroll start point and the touch coordinates (Xcs, Ycs) on the content at the scroll end point are stored in the operation log storage unit 16. In “multi-touch”, touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on two contents are stored in the operation log storage unit 16. In the “multi-touch slide”, touch coordinates (Xcmt1, Ycmt1) and (Xcmt2, Ycmt2) on the two contents at the start point of the pinch operation, and touch coordinates (Xcms1, Ycms1) on the two contents at the end point of the pinch operation ) And (Xcms2, Ycms2) are stored in the operation log storage unit 16. The touch coordinates on these contents are stored in the operation log storage unit 16 in association with the time, the content ID, and the operation type. Thus, the touch coordinates of the start point and the end point are stored for each finger. Here, since the case where multi-touch slide is performed with two fingers is shown, two touch coordinates are stored for each of the start point and the end point. However, when multi-touch slide is performed with three or more fingers, , Three or more touch coordinates are stored for each of the start point and the end point.

  From the contents of the operation log storage unit 16, a number is assigned in order of each operation time, corresponding to the position on the content, and displaying the operation type on the content from the content storage unit 15 in an identifiable manner, It is possible to visually recognize what operation has been performed on what content portion. In an operation where there are a plurality of coordinates, the operation can be understood at a glance by connecting the coordinates with a line.

  FIG. 9 is a finger line log at the time of pinch-in and pinch-out. As shown in FIG. 9, in the pinch-in and the pinch-out, the operation direction is indicated by an arrow.

  FIG. 10 shows an example of the finger line log created in this way. In FIG. 10, numbers are assigned in order of time of operation logs, and finger lines are displayed on the content image. Here, for example, reduction (pinch-in) is a blue line (lines with numbers 5 and 6 in the figure), and enlargement (pinch-out) is a red line (lines with numbers 1 and 2 in the figure). ), Touch is a green circle (circle with number 7 in the figure), scroll is a black line (lines with numbers 3 and 4 in the figure), and the operation is identified by its color I can do it.

  As described above, in the present embodiment, the touch coordinates on the view screen of the touch panel display operated by the user are converted into the touch coordinates on the content and stored as an operation log. The touch coordinates on the content stored as the operation log are visually displayed. For this reason, it is possible to intuitively grasp the user's operation on the touch panel display.

Note that a program for realizing all or part of the functions of the terminal device 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. You may perform the process of. Here, the “computer system” includes an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

1: Terminal device 11: Touch panel function unit 12: Display function unit 13: Viewer unit 14: Operation log calculation unit 15: Content storage unit 16: Operation log storage unit 17: Operation log display unit

Claims (4)

A first process of extracting a partial area of the content and displaying it on the touch panel display in response to a user operation;
A second step of detecting a position of a user operation on the touch panel display of the image of the content displayed in the first step;
The position of the user operation on the touch panel display detected in the second process is converted into a position on the content , an operation type is determined based on the position of the user operation, and the converted position and the A third process of associating the operation type with each other and storing the operation type in the operation log storage unit ;
A fourth step of displaying an identification display indicating an operation type stored in association with the position on the content stored in the operation log storage unit, overlaid on the content;
An operation log collecting method characterized by comprising: In the third step, the converted position, the operation type, and the order of user operations are associated with each other and stored in the operation log storage unit,
In the fourth process, the operation type is displayed in an identifiable manner according to the position stored in the operation log storage unit and the order of user operations.
The operation log collection method according to claim 1. A viewer unit that extracts a partial area of the content and displays it on the touch panel display in response to a user operation;
An operation log calculator that detects a position of a user operation on the touch panel display of the image of the content, and converts the detected position of the user operation on the touch panel display into a position on the content;
An operation log storage unit that determines an operation type based on the position of the user operation and stores the converted position and the operation type in association with each other as an operation log ;
A display function unit that displays, on the content, an identification display indicating an operation type stored in association with the position at a position stored in the operation log storage unit;
An operation log collection device characterized by comprising: A first step of extracting a partial area of the content and displaying it on the touch panel display in response to a user operation;
A second step of detecting a position of a user operation on the touch panel display of the image of the content displayed in the first step;
The position of the user operation on the touch panel display detected in the second step is converted into a position on the content , an operation type is determined based on the position of the user operation, and the converted position and the A third step of associating the operation type with each other and storing the operation type in the operation log storage unit ;
A fourth step of displaying, on the content, an identification display indicating an operation type stored in association with the position at a position stored in the operation log storage unit;
An operation log collection program that can be executed by a computer.