JP5422723B1 - User interface conversion apparatus, method and program - Google Patents

Abstract

A user interface corresponding to the user's preference is always displayed.
Operation history information and screen shot information are stored in a storage unit 101 for each application, and all combinations of operation types and input form types are stored based on the stored operation history information and screen shot information. The familiarity score is calculated and stored in the storage unit 101. Then, when a new application is activated, the GUI of the application is converted so as to correspond to the button operation position and design of the application having a high familiarity score among the applications used in the past.
[Selection] Figure 2

Description

  The present invention relates to a user interface conversion device, method, and program for converting a screen user interface displayed on a user terminal according to user preferences.

  It has long been said that the quality of a GUI (Graphic User Interface) used in a user terminal affects services and applications. In particular, the effect becomes significant under circumstances where it is difficult to respond calmly, such as during a disaster. In fact, in the Great East Japan Earthquake, there were many people who did not know how to use the disaster message board service and gave up using it. However, since an easy-to-use GUI varies from individual to individual, it is very difficult to design and implement a GUI that is easy to use for all.

  Therefore, as a technology for converting the user interface according to the user, the user's ability, for example, language ability, vision, hearing, computer skill, information terminal performance, for example, processing ability, display size, sound quality, remaining power, etc. In addition, there has been proposed a method for changing and presenting the user interface (see, for example, Non-Patent Document 1).

Yusaku Maeda et al. “User interface design for universal multimedia access”, IEICE technical report, KBSE, Intelligent Software Engineering, vol. 108, no. 326, PP. 71-76, 2008.

  However, the method described in the non-patent document is for the user to select a favorite one from templates prepared in advance. For this reason, there is a problem that usability cannot be improved unless a user interface suitable for the user exists in the prepared template.

  The present invention has been made paying attention to the above circumstances, and an object of the present invention is to provide a user interface conversion apparatus, method, and program capable of always displaying a user interface according to the preference for each user. It is in.

In order to achieve the above object, one aspect of the present invention is a user interface conversion apparatus that presents a user interface defined by an application to a user for each application, and the user's familiarity with the application for each application. When the application is executed, the user interface of the application is converted so as to be closer to the user interface of the application having a high degree of familiarity calculated above among the applications used in the past. The user interface is presented to the user.
More specifically, when calculating the degree of familiarity, each time the application is executed, the operation type and form type are detected and stored as an operation history, and based on the stored operation history. For each combination of operation type and form type, the appearance frequency is calculated for each application, and from this calculated appearance frequency, the probability of the appearance frequency of each application is obtained for all combinations of operation type and form type, This probability is stored as the degree of familiarity. When the application is executed, the operation type and the form type of the application to be executed are extracted, and the user interface of the application to be executed is set to the extracted operation type among the applications used in the past. The conversion is made so that it is closer to the user interface of an application with a high degree of familiarity by combining form types .

One aspect of the present invention is characterized by comprising the following aspects.
In the first aspect, when the user interface is converted, the position and form of the operation button of the application are changed to the position and form of the operation button of the application having a high degree of familiarity among the applications used in the past. It is to convert to.

In the second mode, the form design is converted based on the screen shot stored when executing the application with a high degree of familiarity.

According to one aspect of the present invention, when an application is executed, the user interface of the application is converted and presented so as to be closer to the user interface of an application that is used in the past and has a high degree of familiarity. The For this reason, when a user uses a new application, it is possible to display a GUI according to his / her preference for each user without preparing a large number of GUI templates. But you will know how to use it.
In addition, the degree of familiarity is calculated as the appearance frequency for each combination of application operation type and form type. For this reason, it becomes possible to determine the degree of user familiarity with the user interface of the application more accurately than in the case of simply calculating the operation type or the form type.

According to the first aspect, when the user interface is converted, the position and form of the operation button of the application are converted into the position and form of the operation button of the application having a high degree of familiarity among the applications used in the past. The For this reason, it is possible to present the converted user interface in a form that is easy to understand even for users who are unfamiliar with the operation of the application.

According to the second aspect, at the time of the conversion, the form design is converted based on the screen shot stored at the time of executing the application having a high degree of familiarity. Therefore, the form design can be in a form corresponding to the screen shot.

  That is, according to one aspect of the present invention, it is possible to provide a user interface conversion apparatus, method, and program that can always display a user interface according to the preference for each user.

The figure which shows the conversion display example of GUI used for the principle explanation of this invention. The block diagram which shows the function structure of the terminal provided with the user interface conversion function concerning one Embodiment of this invention. The figure which shows an example of GUI information and operation information which are memorize | stored in the terminal shown in FIG. The figure which shows an example of the operation history information memorize | stored in the terminal shown in FIG. The figure which shows an example of the screen shot information memorize | stored in the terminal shown in FIG. The figure which shows an example of the familiarity score information calculated and memorize | stored in the terminal shown in FIG. The flowchart which shows the process sequence and processing content of GUI conversion processing and data storage processing by the terminal shown in FIG. The flowchart which shows the process sequence and process content of the familiarity score calculation process by the terminal shown in FIG.

[principle]
First, the principle of the present invention will be described.
FIG. 1 shows an example in which when a screen user interface corresponding to a certain Web message board service is displayed on a user terminal, it is converted into a user interface corresponding to an application that the user is familiar with and displayed for each user. Yes. For example, if user A is familiar with an application of a microblog service, the screen user interface corresponding to the Web message board service is converted into a display form similar to the familiar microblog service and displayed. Let For the user B who is used to mail, the screen user interface corresponding to the Web message board service is converted into a display form similar to the mailer used to display it.

  In this way, by converting the GUI of a certain service into the GUI display form of the service or application that the user is usually familiar with for each user, the user can himself / herself without preparing many templates. It is possible to use the service by a display form similar to a service or application that is familiar to users.

  In order to realize the above-described GUI conversion, the present invention takes the following measures. That is, first, an application or service that is familiar to the user is determined. For this purpose, an operation history of the application used by the user (for example, the number of times the button is pressed) and a screen shot are stored. Then, from the operation history, it is extracted which operation is frequently performed in what form in which application and what operation is arranged at which position. For example, in the “application A”, information is extracted that the “post” operation of the comment is often performed by pressing the “button” form at the position of “x coordinate is 50 and y coordinate is 10”.

  Next, the same combination of “operation” and “form” is converted into “position” or “design” with a high familiarity score. For example, if the application B used for the first time has an operation “post” and a form “button” and the familiar score of the combination of “post” and “button” of the application A is high, the post button of the application B is used as the application A's post button. Convert to position and design. The GUI conversion is performed by executing this process for all combinations of “operation” and “form” of the application that has started up. In the “design” conversion, the form design is extracted from the position coordinates of the operation history, the width and height of the form, and the screen shot recorded at the same time as the operation history.

  The familiarity score is calculated for all combinations of operation type and form type. The operation type refers to an act of operation such as “bookmark” and “page scroll” in addition to the above “post”. The form type refers to an input form for performing an operation such as “scroll bar” in addition to the “button” described above. The familiarity score can be calculated using, for example, the probability of the number of operations of the application with respect to the appearance frequency of the combination of the same operation type and form type for all applications. For example, it is assumed that an operation history that is a combination of “post” and “button” exists 100 times for all applications, and the number of times for application A is 75 times. In this case, the familiarity score of “post”, “button”, and “application A” is 0.75.

  With the above method, it is possible to display a user interface according to the preference for each user without preparing a large number of templates, so that the user can easily understand how to use even first-time services and applications.

[One Embodiment]
Embodiments according to the present invention will be described below with reference to the drawings.
(Constitution)
FIG. 2 is a block diagram showing a functional configuration of a terminal having a user interface conversion function according to an embodiment of the present invention.
The terminal 100 includes a mobile terminal (node) such as a mobile phone, a smart phone, a tablet-type terminal, and a laptop personal computer. As functions necessary for realizing this embodiment, a storage unit 101 and application execution Unit 102, application monitoring unit 103, operation history information generation unit 104, screenshot information generation unit 105, familiarity score information calculation unit 106, GUI conversion processing unit 107, and timer 108. Among these, each function excluding the storage unit 101 is realized by causing a CPU (Central Processing Unit) (not shown) to execute a pro stored in a program memory (not shown).

  The storage unit 101 is provided with an operation history information storage area 1011, a screen shot information storage area 1012, and a familiarity score information storage area 1013. The operation history information storage area 1011 is used for storing information representing an operation history when the user uses the application. FIG. 4 shows an example of stored operation history information. Each operation history information includes an application name, a storage time, an operation purpose type, an operation form type, an x-coordinate and a y-coordinate on the screen. And information indicating the width and height of the form design.

  The screen shot information storage area 1012 is used for storing screen shot information. A screen shot is a captured image of the screen. FIG. 5 shows an example of stored screen shot information. Each screen shot information is represented by information indicating an application name, a storage time, and a storage location of the captured image file.

  The familiar score information storage area 1013 is calculated by the familiar score information calculation unit 106 described later for all combinations of operation types and form types stored in the operation history information storage area 1011 and the screen shot information storage area 1012. Used to store familiar information. FIG. 6 shows an example of stored familiarity score information, which is stored in association with the operation type, form type, application name, x-coordinate and y-coordinate on the screen.

  The application execution unit 102 passes the operation information and the screen shot to the application monitoring unit 103 when an input event such as a button being pressed or a screen scrolling occurs. Also, when the application is started, the GUI information is passed to the application monitoring unit 103, and when converted GUI information is received from a GUI conversion processing unit 107 described later, the screen data is displayed according to the converted GUI information.

  FIG. 3 shows an example of the GUI information and operation information. The GUI information and operation information are represented by information indicating the application name, operation type, form type, x-coordinate and y-coordinate on the screen, and the width and height of the form design. The data format of the GUI information and the operation information is basically the same, but the GUI information is a set of data of all the same applications, and the operation information is a unique combination of the application, operation type, and form type. For example, in FIG. 3, when the application B is activated, a set of data on the second row and data on the third row becomes the GUI information. When the posting button of application B is pressed, the data on the third line is the operation information. Note that the operation type and the form type are defined in the application execution unit 102 in advance. For example, if the application is a Web application, the input type of the form tag defined in HTML is the form type, and the name given to the form is the operation type.

  When receiving the information from the application execution unit 102, the application monitoring unit 103 sends the operation information to the operation history information generation unit 104 if the received information is operation information, and the screen shot information generation unit 105 if the received information is a screen shot. If so, the data is transferred to the GUI conversion processing unit 107. When the converted GUI information is received from the GUI conversion processing unit 107, the converted GUI information is transferred to the application execution unit 102.

  When the operation history information generation unit 104 receives operation information from the application monitoring unit 103, the operation history information generation unit 104 adds a time stamp to the operation information and generates operation history information. The generated operation history information is stored in the operation history information storage area 1011 of the storage unit 101.

  When the screen shot information generation unit 105 receives a screen shot from the application monitoring unit 103, the screen shot information generation unit 105 generates time-stamp information and a screen shot image storage destination file path for the screen shot, and generates screen shot information. Then, a process for storing the generated screen shot information in the screen shot information storage area 1012 in the storage unit 101 is performed.

  When the familiarity score information calculation unit 106 receives the start trigger, the familiarity score information calculation unit 106 reads the operation history information and the screen shot information from the operation history information storage area 1011 and the screen shot information storage area 1012 of the storage unit 101, respectively. Based on the above, calculate the familiarity score. The familiarity score is calculated as the probability of the number of operations of the application with respect to the appearance frequency of the combination of the same operation type and form type for all applications, for example. The calculated familiar score information is stored in the familiar score information storage unit 1013 of the storage unit 101.

When the GUI conversion processing unit 107 receives the GUI information from the application monitoring unit 103, the GUI conversion processing unit 107 reads the familiar score information from the storage unit 101, converts the GUI information based on the read familiar score information, and after the conversion It has a function of outputting GUI information to the application monitoring unit 103.
The timer 108 has a function of transmitting a start trigger to the familiarity score calculation unit 106 at a preset cycle.

(Operation)
Next, the operation of the user interface conversion process by the terminal 100 configured as described above will be described. FIG. 7 is a flowchart showing the processing procedure and processing contents.
(1) Monitoring of occurrence of input event When an input event such as a button press or a screen scroll operation occurs during the execution of an application, the operation information or screen shot is output from the application execution unit 102 to the application monitoring unit 103. Is done. In addition, GUI information is output to the application monitoring unit 103 when the application is activated.

  The application monitoring unit 103 monitors the output of information by the application execution unit 102 in step S1. When the information is received, it is determined whether the received information is GUI information, operation information, or a screen shot. If it is GUI information, the GUI information is transferred to the GUI conversion processing unit 107, if it is operation information, the operation information is transferred to the operation history information generating unit 104, and if it is a screen shot, the screen shot information generating unit 105 is transferred. Transfer the screenshot to.

(2) Generation of Operation History Information The operation history information generation unit 104 monitors reception of operation information in step S7, and when operation information is received, proceeds to step S8. In step S8, operation history information is generated, and the generated operation history information is stored in the operation history information storage area 1011 of the storage unit 101. The operation history information is generated by adding a time stamp to the received operation information and further shaping the data structure into, for example, the form of a data set shown in FIG. When the generation and storage processing of the operation history information for this one operation is completed, the process returns to the standby state. Thereafter, similarly, every time operation information is received, the operation history information is generated and stored repeatedly.

(3) Generation of Screen Shot Information The screen shot information generation unit 105 monitors the reception of the screen shot in step S9, and proceeds to step S10 when the screen shot is received. In step S <b> 10, processing for generating screen shot information and storing it in the screen shot information storage area 1012 of the storage unit 101 is executed. For example, as shown in FIG. 5, the screen shot information is generated by giving a time stamp and a screen path for saving the screen shot image to the received screen shot. Then, when the generation and storage processing of the screen shot information for this one screen shot is completed, the process returns to the standby state. Thereafter, similarly, every time a screen shot is received, generation and storage processing of screen shot information is repeatedly executed.

(4) Calculation of habituation score When a start trigger is generated from the timer 108, the habituation score calculation unit 106 calculates a habituation score for all applications as follows. FIG. 8 is a flowchart showing the processing procedure and processing contents.
That is, the familiarity score calculation unit 106 first reads all the operation history information and the screen shot information from the operation history information storage area 1011 and the screen shot information storage area 1012 of the storage unit 101 in step S901, respectively. Keep in memory. At the same time, in order to update all of the familiar score information, all the familiar score information stored in the familiar score information storage unit 1013 of the storage unit 101 is deleted.

  Next, in step S902, it is determined whether or not there is an application name for which the familiarity score has not yet been calculated among the pieces of information held in the work memory. As a result of the determination, if there is an application name for which the familiarity score has not been calculated, the process proceeds to step S903.

  In step S903, one of the application names for which the familiarity score is not calculated is selected, and all the operation history information and screen shot information associated with the selected application name are selected. The selected information is held in the work memory. In step S904, a familiarity score is calculated for each combination of the operation type and the form type based on the operation history information selected in step S903.

  The familiarity score is a probability of the number of appearances of the combination of the same operation type and form type in the application selected in step S903 with respect to all the occurrences of the combination of the same operation type and form type included in the operation history information. calculate.

  For example, the selected application is application A, the combination of operation type and form type is “post” and “button”, and the operation history of the combination of “post” and “button” exists 100 times for all applications. Suppose that the number of times in application A is 75 times. In this case, the familiarity score of “post”, “button”, and “application A” is 0.75. Such processing is performed for all combinations of the operation type and the form type, and the habituation score calculated thereby is written in association with the corresponding application A in the habituation score information storage unit 1013 of the storage unit 101.

  Thereafter, similarly, the processes in steps S903 and S904 are executed for all the applications. When the processes are completed, the familiarity score calculation process is terminated, and a standby state is entered until the next start trigger is generated from the timer 108.

(5) Conversion of GUI The GUI conversion processing unit 107 monitors reception of GUI information in step S1, and moves to step S2 when GUI information is received. In step S2, first, the user's familiarity score for each application executed in the past is read from the familiarity score information storage unit 1013 of the storage unit 101. The read familiarity score is held in the work memory.

  In step S3, the GUI conversion processing unit 107 analyzes the operation type and the form type included in the received GUI information. Here, the operation type and the form type indicate posting, page movement, buttons, and scroll bars as shown in FIGS. What operation type or form type is included in the received GUI information is extracted, and the result is stored in the work memory.

  Subsequently, in step S4, converted GUI information is generated based on the familiarity score information acquired in step S2 and the operation type or form type extracted in step S3.

  For example, it is assumed that “post” and “button” exist as combinations of the operation type and the form type of the GUI information just received, and the familiar score information is as shown in FIG. In this case, there are two combinations of “post” and “button” in the familiar score information, but since the large familiar score is that of “application A”, the received GUI information “ The “post”, “button”, and user interface are rewritten to the x and y coordinate values of application A. For design conversion, the operation type and form type selected first, and the name of the application with the highest familiarity score are used as search keys, and the latest storage time is selected from the operation history information stored in the storage unit 101. Get one piece of data.

  Next, using the application name and storage time of the acquired operation history information as search keys, one piece of data having the closest storage time is acquired from the screen shot information stored in the storage unit 101. Then, a screen shot image is read from the acquired screen shot information, and the image is cut out from the read screen shot image using the x, y coordinates, width, and height of the operation history information acquired previously as parameters. Since the cut-out image has a converted design, this image data is added to the converted GUI information.

  The above processing is performed for all combinations of the operation type and form type of the received GUI information, and the information after the conversion is completed is stored in the work memory as the converted GUI information.

  Subsequently, in step S5, the converted GUI information is sent to the application execution unit 102 serving as the application information transmission source, and the process proceeds to step S6. In step S6, the application execution unit 102, which is the application information transmission source, draws a user interface based on the received converted GUI information, the GUI conversion process ends, and a standby state is entered until an event occurs again.

(effect)
As described above in detail, in one embodiment, operation history information and screenshot information are stored in the storage unit 101 for each application, and an operation type and an input form are stored based on the stored operation history information and screenshot information. The familiarity score is calculated and stored in the storage unit 101 for all combinations of types. When a new application is activated, the GUI of the application is converted so as to correspond to the button operation position and design of the application having a high familiarity score among the applications used in the past.

  Therefore, when a user uses a new application, it is possible to display a GUI according to his / her preference for each user without preparing a large number of GUI templates. You will soon know how to use it.

  In addition, since the familiarity score for the application GUI is calculated for the combination of the operation type and the input form type of the application, the user's score for the application GUI can be compared with the case where the operation type or the input form type is simply calculated. It becomes possible to determine the degree of familiarity more accurately.

[Other Embodiments]
The present invention is not limited to the above embodiment. For example, in the embodiment, the case where the terminal 100 is provided with the user interface conversion function has been described as an example, but the same function may be provided in a cloud computer or a Web server. In this case, the cloud computer or the server acquires application operation history information and screen shot information from the terminal via the network. Then, a familiarity score for each application is calculated based on the acquired information, and the GUI information is converted with reference to the calculated familiarity score and downloaded to the terminal. In this way, the processing load on the terminal can be reduced.

  In the embodiment, the familiarity score is recalculated every time a start trigger is generated from the timer 108. However, every time execution of one application is completed, the familiarity score information storage area 1013 of the storage unit 101 is stored. All stored habituation scores may be recalculated. In this way, it is possible to prevent a problem that the familiarity score is recalculated in a state where the application is not executed, and to reduce the processing load and battery consumption of the terminal 100.

In addition, the device type and configuration, the user interface conversion processing procedure and processing contents, and the like can be variously modified without departing from the scope of the present invention.
In short, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.

  DESCRIPTION OF SYMBOLS 100 ... Terminal, 101 ... Storage part, 1011 ... Operation history information storage area, 1012 ... Screenshot information storage area, 1013 ... Familiarity score information storage area, 102 ... Application execution part, 103 ... Application monitoring part, 104 ... Operation history information Generation unit 105 ... Screenshot information generation unit 106 ... Familiarity score information calculation unit 107 ... GUI conversion processing unit 108 ... Timer.

Claims (6)

A user interface conversion device that presents to a user a user interface defined by the application for each application,
Calculating means for calculating the degree of user familiarity with the application for each application;
Conversion means for converting a user interface of the application so as to approach the user interface of the application having a high degree of familiarity among the applications used in the past when the application is executed;
Presenting means for presenting the converted user interface to a user ,
The calculating means includes
Means for detecting the operation type and form type each time the application is executed, and storing it as an operation history;
Based on the stored operation history, for each combination of operation type and form type, means for calculating the appearance frequency in the combination for each application;
Means for obtaining a probability of appearance frequency of each application for each combination of operation type and form type from the calculated appearance frequency, and storing the probability as a degree of familiarity;
With
The converting means includes
When an application is executed, the operation type and form type of the application to be executed are extracted, and the user interface of the application to be executed is extracted from the previously used applications. A user interface conversion device that performs conversion so as to be closer to a user interface of an application having a high degree of familiarity with .   The conversion means converts the position and form of the operation button of the application into the position and form of the operation button of the application having a high degree of familiarity among the applications used in the past. The user interface conversion device according to 1. 3. The user interface conversion apparatus according to claim 2 , wherein the conversion unit converts the design of the form based on a screen shot stored when an application with a high degree of familiarity is executed. A user interface conversion method executed in a device for presenting a user interface defined by the application to the user for each application,
Calculating the degree of user familiarity with the application for each application;
When an application is executed, a process of converting the user interface of the application so as to approach the user interface of the application having a high degree of familiarity among the applications used in the past;
Presenting the converted user interface to a user ,
The calculation process includes:
Each time the application is executed, the process of detecting the operation type and form type and storing it as an operation history;
Based on the stored operation history, for each combination of operation type and form type, a process of calculating the appearance frequency in the combination for each application;
From the calculated appearance frequency, for each combination of operation type and form type, a probability of the appearance frequency of each application is obtained, and this probability is stored as a degree of familiarity;
With
The converting process includes:
When an application is executed, the operation type and form type of the application to be executed are extracted, and the user interface of the application to be executed is extracted from among the applications used in the past. A user interface conversion method, characterized in that conversion is performed so that the user interface of an application having a high degree of familiarity with is approximated . Process for the conversion according to the position and form of the application operation buttons, and converting the position and form of the operation buttons of the calculated accustomed degree higher application of application used in the past Item 5. A user interface conversion method according to Item 4 . The program which makes the computer of the said user interface conversion apparatus perform the process corresponding to the means with which the user interface conversion apparatus in any one of Claim 1 thru | or 3 is provided.

Images