JP2017146740A - Operation support system, operation support program, and operation support method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation support system for appropriately performing an operation guide even to a user unfamiliar with operations by indicating an operation procedure that dynamically corresponds to operations of the user.SOLUTION: The operation support system includes: a scenario acquisition part 112c for acquiring scenario information indicating an operation procedure about functions belonging to a smart phone, corresponding to a screen displayed on display means belonging to the smart phone; a state acquisition part 112b for acquiring screen information displayed currently on the display means, and status information about an operation state of an information processing device as state information; an operation analysis part 112a for analyzing an operation state by a user with reference to the scenario information on the basis of the screen information and the status information included in the state information; and an instruction output part 112e for outputting an operation instruction on the basis of an analysis result by the operation analysis part 112a and the scenario information.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an operation support system, an operation support program, and an operation support method that support an operation by presenting an operation procedure of an information processing apparatus such as a smartphone to a user.

  In recent years, information processing terminals such as smartphones have become widespread, and the number of users who are unfamiliar with computer operations has also increased. According to statistics, 80% of smartphone users feel uneasy when using smartphones. In order to alleviate such anxiety, a method such as visiting a store such as a shop or the like is mainly used each time. It is said that 60% of users who have anxiety give up without being able to resolve the anxiety because it is difficult for the users to communicate.

  In order to deal with such a problem, conventionally, as a technique for guiding the operation of the information processing apparatus to the user, the information processing apparatus stores information for displaying the operation guide in advance and is received via the network. Display an operation guide using the above-mentioned information based on a remote instruction or a user operation on the information processing apparatus (see, for example, Patent Document 1).

JP 2012-023491 A

  However, in the technique disclosed in Patent Document 1 described above, information for displaying the operation guidance is created and stored in advance, so the operation guidance is often different from the actual screen, There was a problem of lack of practicality. More specifically, in recent smartphones, a user can freely customize or install an arbitrary application. Therefore, the screen display changes variously depending on the setting contents arbitrarily customized by the user and the installed application. Therefore, the operation guidance prepared in advance and the actual screen display may be inconsistent, and the user is confused and often cannot follow the operation guidance. In particular, since there are many display items on the setting screen, etc., there are many cases where screen scrolling is required, and for users who are unfamiliar with the operation, the items indicated in the operation guide are often not found. If an incorrect operation is performed in the middle of the operation guidance, there is a case where it is impossible to return to the screen instructed by the operation guidance and to follow the operation guidance.

  Therefore, the present invention solves the above-described problems. An information processing apparatus such as a smartphone is provided with an operation procedure that dynamically corresponds to a user's operation in accordance with a variety of actual screens. It is an object of the present invention to provide an operation support system, an operation support program, and an operation support method that can appropriately guide operation even for a user who is unfamiliar with the operation.

In order to solve the above problem, an operation support system according to the present invention is a system that supports an operation by presenting an operation procedure of an information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
An instruction output unit that outputs an operation instruction based on the analysis result by the operation analysis unit and the scenario information is provided.
An operation support program according to the present invention is a program for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
Based on the analysis result by the operation analysis unit and the scenario information, it functions as an instruction output unit that outputs an operation instruction.
Furthermore, the operation support method according to the present invention is a method for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition step in which a scenario acquisition unit acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A state acquisition step in which the state acquisition unit acquires the screen information currently displayed on the display means and the status information on the operation state of the information processing apparatus as state information;
The operation analysis unit refers to the scenario information based on screen information and status information included in the state information, and analyzes an operation state by a user;
An instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information; and
It is characterized by including.

  According to these aspects of the invention, it is possible to analyze the screen currently displayed on the display means such as the touch panel and the operation state in real time, and dynamically indicate the operation procedure according to the screen by referring to the scenario information. As a result, according to the present invention, the screen display changes depending on the setting contents arbitrarily customized by the user and the installed application, screen scrolling is required, or an incorrect operation is performed during the operation guidance. Even in this case, appropriate operation support can be provided.

In the above-described invention, the scenario information is described in association with screen specifying information including a graphic or a character string for specifying a screen related to a function related to operation support, and an operation instruction necessary for the function related to the operation support,
The operation analysis unit refers to the scenario information, extracts screen specifying information from the image information, determines an operation state by a user based on the extracted screen specifying information,
The instruction output unit can display an operation instruction described in the scenario information superimposed on the screen information based on the operation state determined by the operation analysis unit.

  In this case, it is possible to identify the currently displayed screen using screen identification information including graphics or character strings such as icons and setting item names as marks, and coordinates on the screen such as icons serving as marks Since the position can also be specified, the operation procedure according to the screen can be dynamically shown more accurately.

In the above invention, the scenario information is described in association with a screen related to a function related to operation support and operation target specifying information including a figure or a character string specifying an operation target necessary for the function related to the operation support. ,
The operation analysis unit refers to the scenario information, extracts the operation target specifying information from the image information, determines an operation state by the user based on the extracted operation target specifying information,
The instruction output unit, based on the operation state determined by the operation analysis unit, the operation instruction described in the scenario information in the screen information according to the extracted operation target specifying information and its position on the screen. It can be displayed superimposed.

In this case, it is possible to specify operation target specifying information including a graphic or a character string such as an icon to be operated next or a setting item name, and a touch panel such as an icon to be operated. Since the coordinate position on the screen can also be specified, the operation procedure according to the screen can be dynamically shown more accurately.
In the above invention, the operation instruction output by the instruction output unit and the scenario information referred to when the operation instruction is displayed are recorded as an instruction history in the state information history acquired by the state acquisition unit. A history storage unit to
A history analysis unit that aggregates and analyzes the instruction history stored in the history storage unit and holds the analysis result in association with the referenced scenario information so as to be output can be further provided.

  In this case, the user's reaction based on the scenario information can be analyzed by recording and analyzing the operation instructions indicated by referring to the scenario information and the user's operation corresponding to the instruction as state information. The result can be output in association with the scenario information. Thereby, the appropriateness of scenario information can be evaluated, it can be reflected in correction of scenario information, etc., and more suitable operation support can be provided.

  As described above, according to these inventions, an information processing device such as a smartphone is provided with an operation procedure that dynamically corresponds to the operation of the currently displayed user in accordance with an actual screen that varies in various ways. Even if the user is unfamiliar with the operation, the operation guidance can be performed appropriately.

It is a block diagram which shows the internal structure of the information processing apparatus which concerns on 1st Embodiment. It is a block diagram which shows the module constructed | assembled on CPU of the information processing apparatus which concerns on 1st Embodiment. It is a flowchart figure which shows operation | movement of the operation assistance system which concerns on 1st Embodiment. It is explanatory drawing which shows the screen structure of information processing apparatus regarding the operation | movement of the operation assistance system which concerns on 1st Embodiment. It is explanatory drawing which shows the screen structure of information processing apparatus regarding the operation | movement of the operation assistance system which concerns on 1st Embodiment. It is explanatory drawing which shows the screen structure of information processing apparatus regarding the operation | movement of the operation assistance system which concerns on 1st Embodiment. It is a conceptual diagram which shows typically the whole structure of the operation assistance system which concerns on 2nd Embodiment. It is a block diagram which shows the internal structure of the assistance server which concerns on 2nd Embodiment. It is a block diagram which shows the internal structure of the information processing apparatus which concerns on 2nd Embodiment. It is a sequence diagram which shows operation | movement of the operation assistance system which concerns on 2nd Embodiment. It is explanatory drawing which shows the screen structure of an information processing apparatus and a display apparatus regarding the operation | movement of the operation assistance system which concerns on 1st Embodiment. It is a conceptual diagram which shows typically the whole structure of the operation assistance system which concerns on the example of a change.

  Hereinafter, embodiments of an operation support system, an operation support method, and an operation support program according to the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
A first embodiment of the present invention will be described. In the present embodiment, a case where an operation support program is installed in an information processing apparatus and the information processing apparatus operates stand-alone will be described. In each of the following embodiments, the “module” is a functional unit that is configured by hardware such as an apparatus or a device, software having the function, or a combination thereof, and achieves a predetermined operation. Show.

(Configuration of information terminal device)
FIG. 1 is a block diagram illustrating an internal configuration of the smartphone 1 according to the present embodiment, and FIG. 2 is a block diagram illustrating modules built on the CPU of the smartphone 1.

  The smartphone 1 is an information processing apparatus used by a user who receives an operation support service. The smartphone 1 includes a communication function and a CPU, and various functions can be implemented by installing various application software. In the present embodiment, an operation support application is installed in the smartphone 1 to function as an operation support system. This information processing terminal can be realized by, for example, a general-purpose computer such as a personal computer or a dedicated device with specialized functions, in addition to a smartphone, such as a tablet PC, a mobile computer, a PDA (Personal Digital Assistance), a mobile phone. Includes telephone.

(Configuration of smartphone 1)
Specifically, the smartphone 1 includes a CPU 11 that performs arithmetic processing, a memory 12 that temporarily stores information, a communication interface 13 that performs communication with the outside via a communication line or the Internet, and a storage 14 that is a storage unit. , An output interface 15 and an input interface 16. In the present embodiment, these devices 11 to 16 are connected via the CPU bus 10 and can exchange data with each other.

  The CPU 11 is a device that performs various arithmetic processes necessary for controlling each unit, and virtually constructs various modules on the CPU 11 by executing various programs. In the present embodiment, the CPU 9 causes the general-purpose smartphone 1 to function as an operation support system by executing an operation support program.

  The communication interface 13 is a module that communicates with the wireless base station 31 by wireless communication. The communication interface 13 is an LTE line, a 3G / 4G line, a wireless communication network such as WiFi (registered trademark), Bluetooth (registered trademark), or other publicly known networks. A device / communication system can be used.

  The storage 14 is a device that accumulates data in a recording medium and reads out the accumulated data in response to a request from each device. For example, the storage 14 includes a hard disk drive (HDD), a solid state drive (SSD), a memory card, and the like. Can be configured.

  The CPU bus is connected to position information acquisition means for measuring the current position of the own device from GPS, the identifier and coordinates of the radio base station 31, and clock means for measuring the current time. The measurement results obtained by the position information acquisition means and the clock means are linked to each other and accumulated in the memory 12 and the storage 14 as the status information of the own device, and in response to a request from the OS or application executed on the CPU 11. Delivered accordingly.

  The output interface 15 is a module that transmits a video signal and an audio signal to output video and audio from an output device such as a display and a speaker, and outputs video and audio related to application software executed by the application execution unit 112. be able to. The input interface 16 is a module that receives an operation signal from an operation device such as the touch panel 16a or the physical operation button 16b. The received operation signal is transmitted to the CPU 11 and can perform an operation on the OS and each application. . In the present embodiment, the touch panel 16a connected to the input interface 16 and the output interface 15 is configured such that the touch panel as the input device is superimposed on the display as the display device, and the display data of the operation screen is displayed. Is displayed on the display, and a GUI for acquiring a user operation signal is provided by performing a touch operation or a gesture operation according to the display.

  On the CPU 11, an OS (Operating System) provided for the smartphone by the OS execution unit 111 is activated and executed, and basic functions of the smartphone 1 are controlled by the OS. Also, on the OS, various applications can be executed by the application execution unit 112, and various functions are executed by executing the applications by the application execution unit 112.

  More specifically, the OS execution unit 111 is a module for executing an OS program on the CPU. When the OS program is executed by the OS execution unit 111, the basic functions of the smartphone 1 are managed and controlled. Thus, each module constructed by the application execution unit 112 and each device / function of the smartphone 1 can be linked. The application execution unit 112 is a module for executing an application program on the CPU, and various function modules are virtually constructed on the CPU by executing the application program on the application execution unit 112.

(Configuration of operation support program)
In the present embodiment, when the operation support application is executed by the application execution unit 112, the operation analysis unit 112a, the state acquisition unit 112b, the scenario acquisition unit 112c, the support selection unit 112d, and the instruction output unit 112e Provided.

  The scenario acquisition unit 112c is a module that acquires scenario information. This scenario information can be acquired as a data file by downloading the operation support program at the same time as part of the application, accessing the support server to download it, or updating it periodically. For example, it is stored in the memory 12 or the storage 14 which is a local device, and is read out appropriately when the application is executed.

This scenario information is table data describing operation procedures related to the functions of the smartphone 1 in correspondence with the screen displayed on the touch panel 16a of the smartphone 1, and information as shown in the table below is described for each screen. ing.

  In the above table, “screen information” includes a screen name representing the screen and an activity class name representing the screen currently being operated. The “screen specific landmark image” and “screen specific landmark text” are screen identification information including a graphic or a character string for identifying the screen as shown in FIG. 3D, for example. Is in the screen, the screen is determined as the current screen. The “touch mark image” is operation specifying information which is a graphic such as an icon for specifying an operation target necessary for the function related to the operation support. In addition to the graphic such as an icon as shown in FIG. Character strings such as item names on the menu screen are also included. The “guide end mark image” is screen specifying information including a graphic or a character string that specifies a screen on which an operation related to operation support has been completed.

In this scenario information, an operation instruction necessary for a function related to operation support is associated with the above-described screen specifying information and operation specifying information. This operation instruction includes “scroll direction”, “margin of frame line”, “message”, and “next screen information” in the above table. For example, as an operation instruction when making a call,
1. 1. Press the phone icon 2. Enter the number input screen. Enter the number 4. An operation procedure such as pressing the send button is included, and the screen identification information and operation identification information necessary for the guidance of this operation procedure are in accordance with the order of guidance of the above operation procedure,
1. Home icon 2. Phone app icon Button to open number input screen 4. Number buttons (0-9)
5. A call button is described in the scenario information.

  The support selection unit 112d displays a list of scenarios related to operation support on the menu screen or start screen of the operation support application as shown in FIG. 4A, and selects an operation for which the user is requesting support. The scenario acquisition unit 112c acquires the scenario selected by the user through a touch operation or the like.

  The state acquisition unit 112b is a module that acquires screen information regarding the screen and status information regarding the operation state of the smartphone 1 as state information. Here, the “screen information regarding the screen” is a still image or a moving image obtained by capturing the GUI displayed on the display as it is. This "status information" includes the name of the currently displayed screen, the class name of the Activity that identifies the active screen that is the target of the operation, the application being executed, a call, and the camera Event information generated from the smartphone 1, such as a shutter release, various setting information, layout information such as applications, and the like are included. Each information acquired by the state acquisition unit 112b is input to the operation analysis unit 112a and the instruction output unit 112e.

  The operation analysis unit 112a is a module that analyzes the operation state of the user by referring to the scenario information based on the screen information and the status information included in the state information acquired by the state acquisition unit 112b. The operation analysis unit 112a refers to the scenario information read by the scenario acquisition unit 112c, searches the image information, extracts an image or character string that matches the screen specific landmark image or the screen specific landmark text, and is extracted. Based on the screen-specific landmark image or screen-specific landmark text, the current user's operation state is determined, and the image information is searched to extract images and character strings that match the touch landmark image and character strings. Based on the operation target specifying information, the operation state of the current user is determined, and the position and timing of the operation instruction to be displayed next are determined.

  The instruction output unit 112e is a module that outputs an operation instruction based on the analysis result and the scenario information by the operation analysis unit 112a. For example, the operation output unit 112a extracts the operation target specifying information extracted based on the operation state determined by the operation analysis unit 112a. Depending on the position on the screen, operation instructions described in the scenario information such as guidance marks and messages are superimposed on the screen information and displayed. Further, as an operation instruction output by the instruction output unit 112e, a guidance message, a support message, or an alert may be output by voice or sound.

(Operation support procedure)
By operating the operation support system described above, the operation support method of the present invention can be implemented. FIG. 3 is a flowchart showing the operation of the operation support system according to the present embodiment. Here, a case will be described as an example where the user is requesting assistance for an operation of making a call.

  First, an operation support application is activated on the smartphone 1 (S101). By this activation, the support selection unit 112d displays a list of scenarios related to operation support on the menu screen W1 of the operation support application as illustrated in FIG. The support selection unit 112d prompts the user to select which operation the user is requesting support in the scenario list.

  When the user selects the scenario list by a touch operation or the like (S102), the support selection unit 112d causes the scenario acquisition unit 112c to acquire the scenario information selected by the user (S103). Here, since the user is requesting support for the operation of making a call, it is assumed that the icon M1 “make a call (2)” indicated by an arrow in FIG. 4A is selected.

  At this time, since the present embodiment operates in a stand-alone format, the scenario information is already stored in the memory 12 and the storage 14 which are local devices, and is appropriately read when executing the application. Here, when acquiring the scenario information, the scenario acquisition unit 112c can access the support server and download the necessary scenario information when an appropriate scenario does not exist in the local device. Further, when acquiring this scenario information, it may be confirmed whether or not the latest scenario information has been released and updated to the latest version at all times.

  As preprocessing for analysis of user operation, information necessary for searching for and detecting a specific screen and operation target is extracted from the selected scenario information. Specifically, “screen information”, “screen specific landmark image”, “screen specific landmark text”, “touch landmark image”, and “guide end landmark image” described in the scenario information are extracted. Here, since the user is requesting support for the operation of making a call, the icon M4 shown in FIG. 4 (d) is used as a screen specific landmark image in order to detect the home screen W2 shown in FIG. 4 (b). And the icon M2 shown in FIG. 4C, which is the object to be operated next by the user, is extracted.

  Next, the screen information displayed on the display and the status information regarding the operation state of the smartphone 1 are acquired as state information (S104). Specifically, as the screen information, the GUI displayed on the display is acquired as it is as a still image or a moving image, and as the status information, the currently displayed screen name and the operation target become active. In addition to the class name of the activity that identifies the screen that is running, the application that is being executed, event information generated from the smartphone 1, such as making a call or releasing the camera shutter, various setting information, layout information of the application, etc. To be acquired. Each information acquired by the state acquisition unit 112b is input to the operation analysis unit 112a and the instruction output unit 112e.

  Then, based on the screen information and status information included in the status information acquired by the status acquisition unit 112b, the scenario information is referred to, and the operation status by the user is analyzed. Specifically, the operation analysis unit 112a refers to the scenario information read by the scenario acquisition unit 112c and searches the image information to match the screen specific landmark image, the screen specific landmark text, or the operation target identification information. An image or a character string is searched (S105). Here, as shown in FIG. 4, an operation for opening the home screen W <b> 2 is guided in order to perform a touch operation on the icon M <b> 2 for placing a call placed on the home screen W <b> 2. Since the icon M4 for transitioning to the sub screen is arranged on the home screen W2, the icon M4 is searched for as a screen specific landmark image of the home screen W2, and after the home screen W2 is detected, The icon M2 is searched.

  This state acquisition is repeated until a screen specific landmark image is found (“N” in S105). If the home screen W2 is not found after a certain period of time, the current screen is switched to a scenario for popping up a message or opening the home screen as operation support for transition to the home screen. The scenario for making a telephone call may be resumed after providing guidance for transition from to the home screen W2.

  When the screen specific landmark image (icon M4) and the operation target specific information (M2) are detected (“Y” in S105), it is determined that the current user operation is in accordance with the scenario. Then, together with the message to that effect, an instruction to be operated next is output (S106). Here, an instruction for prompting the user to touch the icon M2 is displayed to shift to the telephone dialing screen W3 for making a call. Here, a frame M3 for emphasizing the icon M2 in a rectangular shape is displayed. In addition, when it is necessary to display different parts even on the same screen, such as scrolling and swiping, the user designates the scrolling direction and the swipe direction. In addition to the display of these operation instructions, or instead of the display of the operation instructions, a message or alert may be output by voice or sound.

  When the user performs an appropriate operation according to the guidance by the frame M3 (“Y” in S107), it is determined whether a series of operations is completed and the purpose is achieved (S108). Here, when the icon M2 is touched, a transition is made to the telephone dialing screen W3 shown in FIG. 5A, and similar search for specific information is also executed on the telephone dialing screen W3. More specifically, in the telephone dialing screen W3, a dial number or a call icon M5 is searched for as a screen specific mark image or operation target specifying information, and when these icons M5 are detected (S105). ("Y"), it is determined that the operation by the current user is in accordance with the scenario, and an instruction to be operated next is output together with a message to that effect (S106).

  Here, frames M3 that are enclosed and emphasized in a rectangular shape are sequentially displayed so that the numbers that are push buttons are sequentially touched for dialing. Thereafter, as shown in FIG. 5B, the frame M3 is emphasized so as to touch the icon M5 for outgoing. In addition to the display of these operation instructions, or instead of the display of the operation instructions, it is also possible to output the operation instructions by voice or sound as messages or alerts.

  The determination in step S108 as to whether or not the purpose has been achieved is, for example, whether or not an instruction for the next screen is described in the scenario information, or the “guide end mark image” is on the screen. Whether or not the operation is completed or whether there is a next operation is determined based on whether or not. On the other hand, if the user does not perform an appropriate operation even after a lapse of a certain time in step S107, a message prompting the operation or a sound may be output. If the user performs an incorrect operation in step S107, the process returns to step S104 to acquire the state again, and pops up a message or returns to the appropriate screen as operation support for redoing the operation. The scenario for making a call may be resumed after switching to the scenario for the purpose, performing guidance for transition from the current screen to the appropriate screen, and so on.

  In step S108, when the operation is completed (“Y” in S108), as shown in FIG. 6, the screen returns to the menu screen W1 of the operation support application and informs that the operation is completed, such as pop-up M6. The completion display for this is executed (S109). On the other hand, when the operation is not completed (“N” in S108), the process returns to step S103, and after referring to the scenario, the above steps S104 to S108 are repeated.

(Action / Effect)
According to the present embodiment, the screen and the operation state currently displayed on the touch panel can be analyzed in real time, and the operation procedure according to the screen can be dynamically indicated by referring to the scenario information. Appropriate operation support even if the screen display changes, screen scrolling is necessary, or if an incorrect operation is performed during operation Can be provided.

  In particular, in this embodiment, the entire system is installed in the smartphone 1 as a single application and operates in a stand-alone manner, so communication with a support server on the network is unnecessary, and the system is used even offline. Therefore, the time lag required for communication with the support server does not occur, and the system can be used without stress.

(Example of change)
The above description of the embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made according to the design and the like as long as they do not depart from the technical idea of the present invention.

  For example, in the above embodiment, it is assumed that the operation is performed in a stand-alone format, but the history of the state information acquired by the state acquisition unit 112b, the operation instruction displayed by the instruction output unit 112e, and the operation instruction are displayed. The scenario information D1 referred to at the time of recording may be associated and recorded as an instruction history, and may be periodically uploaded to a support server on the communication network. In the support server, the uploaded instruction history may be aggregated and analyzed in association with the scenario information referred to when the operation instruction is displayed.

  According to such a modified example, the user's reaction based on the scenario information is analyzed by recording and analyzing the operation instruction shown with reference to the scenario information and the user's operation corresponding to the operation instruction as state information. The analysis result can be output in association with the scenario information. Thereby, the appropriateness of scenario information can be evaluated, it can be reflected in correction of scenario information, etc., and more suitable operation support can be provided.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. This embodiment demonstrates the case where the smart phone 1 and the support server which a service provider operates are made to cooperate through the internet. In the present embodiment, the same components as those in the above-described embodiment are denoted by the same reference numerals, and the functions and the like are the same unless otherwise specified, and the description thereof is omitted.

(Overall configuration of operation support system)
FIG. 7 is a conceptual diagram schematically showing the overall configuration of the operation support system according to the second embodiment, and FIGS. 8 and 9 are blocks showing the internal configuration of the support server 2 and the smartphone 1 according to this embodiment. FIG.

  As shown in FIG. 7, the operation support system according to the present embodiment includes a support server 2 that is a server group that provides an operation support service, a smartphone 1a that is used by a user, and a personal computer that is a display unit. The computer 1b is arranged to be accessible to each other.

  The Internet 3 includes a wireless communication network, and various communication lines (telephone lines, ISDN lines, ADSL lines, public lines such as optical lines, dedicated lines, 4G, LTE, 3G, etc.) using the communication protocol TCP / IP. Network 3) is a distributed communication network constructed by connecting each other, and this Internet 3 includes an intranet (intra-company network) such as 10BASE-T and 100BASE-TX, and a predetermined area (company, home, etc.) LAN / WiFi such as an internal network is also included.

  The wireless base station 31 is connected to the Internet 3 through a relay device, establishes wireless communication connection with various communication methods such as 4G, LTE, 3G, and wireless LAN between the smartphone 1a and the PC 1b, and calls by the smartphone 1a and the PC 1b. And a device that provides data communication. The wireless base station 31 is provided with node devices such as a modem, a terminal adapter, and a gateway device for connecting to the Internet 3, and performs communication path selection and mutual conversion of data (signals). The relay process between 31 and the Internet 3 is also performed.

  The support server 2 is a server device having a user management function, an operation support function, a content distribution function, and the like, and can be realized by a server group in which a plurality of functions are distributed. This support server 2 includes a Web server function, and is composed of material data such as HTML (HyperText Markup Language), video, audio, and images in a document system such as WWW (World Wide Web) and is necessary for operation support. Content such as simple web pages can be distributed.

  The smartphone 1a is an information processing terminal that includes a communication function and a CPU as in the above-described embodiment, and various functions can be implemented by installing various application software. A plurality of applications cannot be executed and displayed in a superimposed manner, and the mark cannot be displayed on the home screen, the setting screen, or other screens as in the first embodiment. . For this reason, in the present embodiment, in response to such OS specifications, a status transmission unit that transmits terminal status information to the support server 2 through the Internet 3 to the smartphone 1a as shown in FIG. 9A. 112f is provided, status acquisition and operation analysis with reference to the scenario are performed on the support server 2 side, and operation support instruction output is performed on the display of the personal computer 1b.

(Configuration of smartphone 1a)
More specifically, the OS execution unit 111 is a module for executing an OS program on the CPU. When the OS program is executed by the OS execution unit 111, the basic functions of the smartphone 1 are managed and controlled. Thus, each module constructed by the application execution unit 112 and each device / function of the smartphone 1 can be linked. The application execution unit 112 is a module for executing an application program on the CPU, and various function modules are virtually constructed on the CPU by executing the application program on the application execution unit 112.

  In the present embodiment, an operation support application is executed by the application execution unit 112, thereby providing a state transmission unit 112f and a support selection unit 112d.

  The support selection unit 112d displays a list of scenarios related to operation support on the menu screen or start screen of the operation support application as shown in FIG. 4A, and selects an operation for which the user is requesting support. The scenario acquisition unit 23 on the server side acquires the scenario selected by the user by a touch operation or the like.

  The state transmission unit 112f is a module that transmits screen information about the screen and status information about the operation state of the smartphone 1 to the support server 2 as state information D2. Here, the “screen information” is a still image or a moving image obtained by capturing the GUI displayed on the display as it is. This "status information" includes the name of the currently displayed screen, the class name of the Activity that identifies the active screen that is the target of the operation, the application being executed, a call, and the camera Event information generated from the smartphone 1, such as a shutter release, various setting information, layout information such as applications, and the like are included.

(Configuration of support server 2)
Next, the configuration of the support server 2 will be described. FIG. 8 is a block diagram illustrating functional modules of the support server 2 according to the embodiment. As shown in the figure, the support server 2 is a server device installed on the Internet 3, and includes a communication interface 21 that accesses the Internet 3 and transmits and receives data to and from the smartphone 1a and the personal computer 1b. I have.

  Also in the support server 2, an operation support program is executed on the CPU, and various functions are executed by executing applications on the CPU 11. The support server 2 includes an operation analysis unit 22, a scenario acquisition unit 23, an operation instruction generation unit 24, and a state acquisition unit 25 as modules relating to operation support.

  The scenario acquisition unit 23 is a module that acquires scenario information D1, and is a module that appropriately reads the scenario information D1 stored in the storage unit 26 such as a hard disk when executing an application.

  The state acquisition unit 25 is a module that acquires screen information regarding the screen of the smartphone 1 and status information regarding the operation state as state information through the state transmission unit 112f on the smartphone 1 side. In the present embodiment, the state information D2 acquired by the state acquisition unit 25 includes an identifier that identifies a scenario selected by the user by a touch operation or the like in the support selection unit 112d on the smartphone 1 side.

  The operation analysis unit 22 is a module that analyzes the operation state by the user by referring to the scenario information D1 based on the screen information and status information included in the state information D2 acquired by the state acquisition unit 25. The operation analysis unit 22 refers to the scenario information read by the scenario acquisition unit 23, searches the image information, extracts images and character strings that match the screen specific landmark image or the screen specific landmark text, and is extracted. Based on the screen-specific landmark image or screen-specific landmark text, the current user's operation state is determined, and the image information is searched to extract images and character strings that match the touch landmark image and character strings. Based on the operation target specifying information, the operation state of the current user is determined, and the position and timing of the operation instruction to be displayed next are determined. The analysis result is output to the operation instruction generation unit 24.

  The operation instruction generation unit 24 is a module that generates display data and audio data as operation instructions to be output to the personal computer 1b based on the analysis result by the operation analysis unit 22 and the scenario information D1, for example, the operation analysis unit 22 Display data in which operation instructions described in scenario information, such as guidance marks and messages, are superimposed on screen information according to the extracted operation target identification information and its position on the screen based on the determined operation state Is transmitted to the personal computer 1b through the communication interface 21.

  In the present embodiment, the support server 2 includes a history analysis unit 27 as a mechanism for improving operation support system performance and service quality. Specifically, in this embodiment, the operation instruction data D3 generated by the operation instruction generation unit 24 and the operation instruction data D3 are added to the state history data D4 that is the history of the state information D2 acquired by the state acquisition unit 25. The scenario information D1 referred to at the time of output is correlated and recorded in the storage unit 26 as instruction history data D5, and by analyzing the instruction history data D5, the scenario information D1 can be improved. ing.

  The history analysis unit 27 aggregates and analyzes the instruction history data D5 stored in the storage unit 26, and stores the analysis results in the storage unit 28 as analysis result data D6 in association with the referenced scenario information D1. The analysis result data D6 can be output together with the associated scenario information D1, the operation instruction data D3, and the state history data D4, and how the user behaves according to each scenario information and what operation is performed. It is possible to analyze the user's reaction based on the scenario information, such as whether the result has been obtained. Thereby, the appropriateness of scenario information can be evaluated, it can be reflected in correction of scenario information, etc., and more suitable operation support can be provided.

  The analysis by the history analysis unit 27 includes, for example, whether or not the user has completed the target operation based on each scenario information. For example, the operation instruction (instruction output information) output at that stage is specified. Also, as another analysis method, even if the user can complete the target operation, the time required to complete the operation, the repeated operation, the place where rework occurred, etc. are extracted Then, the operation instruction (instruction output information) output at that stage may be specified.

(Configuration of personal computer 1b)
Next, the personal computer 1b will be described. FIG. 9B is a block diagram showing the internal configuration of the display device according to this embodiment. In the present embodiment, the personal computer 1b is a device that has a communication function through the Internet 3 and has a function of displaying and outputting data received through the Internet 3. For example, in addition to a personal computer, a tablet PC, and a smartphone, It may be a television receiver having an internet connection function.

  Also in the personal computer 1b, an image display program is executed on the CPU, and various functions are provided by executing the application on the CPU 11. The personal computer 1b includes an instruction output unit 112e as a module related to operation support.

  The instruction output unit 112e is a module that outputs the operation instruction data D3 generated by the operation instruction generation unit 24 on the support server 2 side. As shown in FIG. 11, the display 151 of the personal computer 1b has a smartphone 1a side. The same image as the screen displayed on the screen is displayed in real time, and an operation instruction described in the scenario information such as a guidance frame M3 and a message is displayed on the icon M2 which is the operation target specifying information. Is displayed superimposed on. Further, as an operation instruction output by the instruction output unit 112e, a guidance message, a support message, or an alert may be output by voice or sound. The instruction output unit 112e may be provided as an application dedicated to this system, or may be a general-purpose program such as browser software.

(Operation support procedure)
By operating the operation support system described above, the operation support method according to the present embodiment can be implemented. FIG. 10 is a sequence diagram showing the operation of the operation support system according to the present embodiment. Here, a case will be described as an example where the user is requesting assistance for an operation of making a call.

  First, an operation support application is activated on the smartphone 1a (S201). By this activation, the support selection unit 112d displays a list of scenarios related to operation support on the menu screen W1 of the operation support application as illustrated in FIG. The support selection unit 112d prompts the user to select which operation the user is requesting support in the scenario list.

  When the user selects the scenario list by a touch operation or the like (S202), the support selection unit 112d transmits an identifier for identifying the scenario information selected by the user to the support server 2 side for acquisition (S202 and S301). Here, since the user is requesting support for the operation of making a call, it is assumed that the icon M1 “make a call (2)” indicated by an arrow in FIG. 4A is selected.

  In this embodiment, since the operation analysis is performed on the support server 2 side, the scenario information D1 is stored in the storage unit 26, and the scenario information selected according to the selection operation in the support selection unit 112d is stored in the scenario acquisition unit. 23, as appropriate (S302). The read scenario information D1 is sent to the operation analysis unit 22, the operation instruction generation unit 24, and the state acquisition unit 25.

  As preprocessing for analysis of user operation, information necessary for searching for and detecting a specific screen and operation target is extracted from the selected scenario information. Specifically, “screen information”, “screen specific landmark image”, “screen specific landmark text”, “touch landmark image”, and “guide end landmark image” described in the scenario information are extracted. Here, since the user is requesting support for the operation of making a call, the icon M4 shown in FIG. 4 (d) is used as a screen specific landmark image in order to detect the home screen W2 shown in FIG. 4 (b). And the icon M2 shown in FIG. 4C, which is the object to be operated next by the user, is extracted.

  Next, the screen information displayed on the display and the status information regarding the operation state of the smartphone 1 are transmitted to the support server 2 as state information (S203, S303). Specifically, on the smartphone 1 side, the GUI displayed on the display as screen information is transmitted to the support server 2 as a still image or video captured as it is, and as the status information, the currently displayed screen name, In addition to the class name of Activity that identifies the screen that is active as an operation target, the application being executed, event information generated from the smartphone 1, such as making a call or releasing the camera shutter, various setting information The layout information of the application etc. is transmitted. The state information transmitted here is recorded in time series and is stored in the storage unit 26 as state history data D4.

  Then, on the support server 2 side, based on the screen information and the status information included in the state information acquired by the state acquisition unit 25, the scenario information is referred to and the operation state by the user is analyzed. Specifically, the state acquisition unit 25 refers to the scenario information read by the scenario acquisition unit 23 and searches the image information to match the screen specific landmark image, the screen specific landmark text, or the operation target identification information. An image or character string is searched (S304). Here, as shown in FIG. 4, an operation for opening the home screen W <b> 2 is guided in order to perform a touch operation on the icon M <b> 2 for placing a call placed on the home screen W <b> 2. Since the icon M4 for transitioning to the sub screen is arranged on the home screen W2, the icon M4 is searched for as a screen specific landmark image of the home screen W2, and after the home screen W2 is detected, The icon M2 is searched.

  This state acquisition is repeated until a screen specific landmark image is found (“N” in S304). If the home screen W2 is not found after a certain period of time, a message is displayed on the personal computer 1b side or a scenario for opening the home screen is switched as operation support for transition to the home screen. Then, after providing guidance for transition from the current screen to the home screen W2, a scenario for making a call may be resumed.

When the screen specific landmark image (icon M4) and the operation target specific information (M2) are detected (“Y” in S303), it is determined that the current user operation is in accordance with the scenario. Then, along with the message to that effect, an instruction to be operated next is transmitted to the personal computer 1b and output on the personal computer 1b side (S305 and S403). Here, an instruction for prompting the user to touch the icon M2 to display the telephone dial screen W3 for making a call is displayed, and a frame M3 for emphasizing the target icon M2 in a rectangular shape is displayed. In addition, when it is necessary to display different parts even on the same screen, such as scrolling and swiping, the user designates the scrolling direction and the swipe direction. In addition to the display of these operation instructions, or instead of the display of the operation instructions, a message or alert may be output by voice or sound.
The instruction output information transmitted in step S305 is recorded in time series and is stored in the storage unit 26 as instruction history data D5.

  When the user performs an appropriate operation on the smartphone 1 according to the guidance by the frame M3 (“Y” in S306), it is determined whether the series of operations is completed and the purpose is achieved (S307). ). As the determination in step S307, for example, the operation is performed based on whether or not an instruction for the next screen is described in the scenario information, or whether or not the “guide end mark image” is on the screen. Determine if it is complete or if there is a next operation. On the other hand, if the user does not perform an appropriate operation even after a predetermined time has elapsed in step S306, a message prompting the operation may be output by display or voice. If the user performs an incorrect operation in step S306, the process returns to step S303 to acquire the state again, and pops up a message or returns to the appropriate screen as operation support for redoing the operation. The scenario for making a call may be resumed after switching to the scenario for the purpose, performing guidance for transition from the current screen to the appropriate screen, and so on.

  In step S307, when the operation is completed (“Y” in S307), the screen returns to the operation support application menu screen W1, and a completion display for notifying the completion of the operation, such as pop-up M6, is displayed on the personal computer. 1b and displayed on the personal computer 1b side (S308 and D404). When the operation support of this system is terminated by the smartphone 1, the termination operation is performed by the application on the smartphone 1 side (S207 and completion display S208), and this termination operation is transmitted to the support server 2 side as status information. Thus, in step S307, it is determined that the operation is completed. On the other hand, when the operation is not completed (“N” in S307), the process returns to step S302, and after referring to the scenario, the above steps S303 to S307 are repeated.

  In the present embodiment, the operation support system for each user is executed and the history is accumulated, and the operation support history for each user is periodically aggregated and analyzed. Improves performance and service quality. Specifically, the operation instruction data D3 generated in step S305 and the scenario information D1 referred in step S302 are associated with the state history data D4 that is the history of the state information D2 acquired in step S303. The instruction history data D5 is recorded in the storage unit 26, and the instruction history data D5 is analyzed. As a result, the user's reaction based on the scenario information, such as how the user behaves according to each scenario information and what operation results were obtained, can be analyzed, and the appropriateness of the scenario information can be evaluated. It can be reflected in the correction of scenario information.

(Action / Effect)
According to the present embodiment, the screen and the operation state currently displayed on the touch panel can be analyzed in real time, and the operation procedure according to the screen can be dynamically indicated by referring to the scenario information. Appropriate operation support even if the screen display changes, screen scrolling is necessary, or if an incorrect operation is performed during operation Can be provided.

  In particular, in the present embodiment, the state transmission unit 112f is provided on the smartphone 1 side, and operation analysis and instruction output are performed on the support server 2 side. Therefore, the arithmetic processing performed on the smartphone 1 side can be reduced. Furthermore, in this embodiment, since the personal computer 1b is provided separately from the smartphone 1 and the instruction output is performed by the personal computer 1b, for example, it is not possible to execute and display a plurality of applications superimposed on the OS specification. Even in this case, the personal computer 1b can display the mark on the home screen, the setting screen, and other screens, and can provide an operation support service regardless of the OS specifications.

(Example of change)
The description of the above-described embodiment is an example of the present invention. The present invention is not limited to the above-described embodiment, and is within the scope of the technical idea according to the present invention, depending on the design and the like. Various modifications are possible.

  For example, in the said 2nd Embodiment, although the state transmission part 112f by the side of the smart phone 1 was made into the module constructed | assembled on CPU of the smart phone 1, for example, it is good also as a dedicated relay apparatus 32 as shown in FIG. . The relay device 32 acquires screen information as a video signal from the video output terminal of the smartphone 1 through a USB cable 32a, a video / audio cable, etc., digitizes the video signal, and sends it to the support server 2 side by streaming delivery. Provide a function to transmit.

  For example, when the USB cable 32a is connected to the connection terminal, the relay device 32 automatically accesses the support server 2 and uses the video signal acquired through the USB cable 32a as status information. 2 may be provided with a function of starting transmission. Further, the relay device 32 may be provided with a video output / input device such as a touch panel, and the functions of the above-described support selection unit 112d and instruction output unit 112e may be provided.

  According to this modified example, since the operation support system can be executed simply by connecting the smartphone 1 to the dedicated relay device 32, even a user unfamiliar with the operation of the information processing device can easily support the operation. Can receive service.

D1 ... Scenario information D2 ... Status information D3 ... Operation instruction data D4 ... State history data D5 ... Instruction history data D6 ... Analysis result data M1, M2, M4, M5 ... Icon M3 ... Frame M6 ... Popup W1 ... Menu screen W2 ... Home Screen W3 ... Phone dial screen 1, 1a ... Smartphone 1b ... Personal computer (display device)
2 ... Support server 3 ... Internet 10 ... CPU bus 11 ... CPU
DESCRIPTION OF SYMBOLS 12 ... Memory 13 ... Communication interface 14 ... Storage 15 ... Output interface 16 ... Input interface 16a ... Touch panel 16b ... Operation button 21 ... Communication interface 22 ... Operation analysis part (server side)
23 ... Scenario acquisition unit (server side)
24. Operation instruction generation unit (server side)
25 ... Status acquisition part (server side)
26, 28 ... Accumulation unit 27 ... History analysis unit 31 ... Wireless base station 32 ... Relay device 32a ... USB cable 111 ... OS execution unit 112 ... Application execution unit 112a ... Operation analysis unit (smart phone side)
112b ... Status acquisition unit (smart phone side)
112c ... Scenario acquisition unit (smart phone side)
112d ... Support selection unit (smart phone side)
112e ... Instruction output unit (smart phone side)
112f ... Status transmitter (smart phone side)
151. Display

In order to solve the above problem, an operation support system according to the present invention is a system that supports an operation by presenting an operation procedure of an information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
An instruction output unit that outputs an operation instruction based on the analysis result by the operation analysis unit and the scenario information ,
In the scenario information, screen specifying information including a graphic or character string specifying a screen related to a function related to operation support is described in association with an operation instruction necessary for the function related to the operation support,
The operation analysis unit refers to the scenario information, extracts screen specifying information from the screen information, determines an operation state by a user based on the extracted screen specifying information,
The instruction output unit, the operation analysis unit based on the operation state determined, characterized Rukoto to output the described operation instruction to the scenario information.
The operation support system according to the present invention is an operation support system that supports an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with screen information displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on the screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by the user;
An instruction output unit that outputs an operation instruction based on the analysis result by the operation analysis unit and the scenario information;
A history storage unit that records, as an instruction history, the operation instruction output by the instruction output unit and the scenario information referred to when the operation instruction is displayed in the state information history acquired by the state acquisition unit When,
A history analysis unit that aggregates and analyzes the instruction history accumulated in the history storage unit and holds the analysis result in association with the referenced scenario information so as to be output;
It is characterized by providing.
An operation support program according to the present invention is a program for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
Based on the analysis result by the operation analysis unit and the scenario information, function as an instruction output unit that outputs an operation instruction ,
In the scenario information, screen specifying information including a graphic or character string specifying a screen related to a function related to operation support is described in association with an operation instruction necessary for the function related to the operation support,
The operation analysis unit refers to the scenario information, extracts screen specifying information from the screen information, determines an operation state by a user based on the extracted screen specifying information,
The instruction output unit, the operation analysis unit based on the operation state determined, characterized Rukoto to output the described operation instruction to the scenario information.
An operation support program according to the present invention is an operation support program for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with screen information displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
An instruction output unit that outputs an operation instruction based on the analysis result by the operation analysis unit and the scenario information;
A history storage unit that records, as an instruction history, the operation instruction output by the instruction output unit and the scenario information referred to when the operation instruction is displayed in the state information history acquired by the state acquisition unit When,
As a history analysis unit that collects and analyzes the instruction history accumulated in the history storage unit and holds the analysis result in association with the referenced scenario information so that it can be output
It is made to function.
Furthermore, the operation support method according to the present invention is a method for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition step in which a scenario acquisition unit acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A state acquisition step in which the state acquisition unit acquires the screen information currently displayed on the display means and the status information on the operation state of the information processing apparatus as state information;
The operation analysis unit refers to the scenario information based on screen information and status information included in the state information, and analyzes an operation state by a user;
An instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information; and
Only including,
In the scenario information, screen specifying information including a graphic or character string specifying a screen related to a function related to operation support is described in association with an operation instruction necessary for the function related to the operation support,
The operation analysis unit refers to the scenario information, extracts screen specifying information from the screen information, determines an operation state by a user based on the extracted screen specifying information,
The instruction output unit outputs an operation instruction described in the scenario information based on the operation state determined by the operation analysis unit .
An operation support method according to the present invention is an operation support method for supporting an operation by presenting an operation procedure of the information processing apparatus,
A scenario acquisition step in which a scenario acquisition unit acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A state acquisition step in which the state acquisition unit acquires the screen information currently displayed on the display means and the status information on the operation state of the information processing apparatus as state information;
An operation analysis step for analyzing an operation state by a user with reference to the scenario information based on screen information and status information included in the state information,
An instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information; and
The history storage unit records the operation instruction output by the instruction output unit and the scenario information referred to when the operation instruction is displayed in the history of the state information acquired by the state acquisition unit as an instruction history. And steps to
A history analysis unit that aggregates and analyzes the instruction history stored in the history storage unit, and stores the analysis result in association with the referenced scenario information so that it can be output;
It is characterized by including.

Further, according to the above invention, the operation target specifying information including a figure or a character string such as an icon to be operated next and a setting item name can be specified, and an icon to be operated, Since the coordinate position on the screen such as the touch panel can be specified, the operation procedure according to the screen can be dynamically shown more accurately.

  Furthermore, according to the above invention, the user's reaction based on the scenario information can be analyzed by recording and analyzing the operation instruction shown with reference to the scenario information and the user's operation corresponding to the operation instruction as state information. The analysis result can be output in association with the scenario information. Thereby, the appropriateness of scenario information can be evaluated, it can be reflected in correction of scenario information, etc., and more suitable operation support can be provided.

  In the above-described invention, the scenario information is described in association with screen specifying information including a graphic or a character string for specifying a screen related to a function related to operation support, and an operation instruction necessary for the function related to the operation support,
  The operation analysis unit refers to the scenario information, extracts screen specifying information from the image information, determines an operation state by a user based on the extracted screen specifying information,
  The instruction output unit causes the operation instruction described in the scenario information to be superimposed on the screen information and displayed based on the operation state determined by the operation analysis unit.
be able to.

  In this case, it is possible to identify the currently displayed screen using screen identification information including graphics or character strings such as icons and setting item names as marks, and coordinates on the screen such as icons serving as marks Since the position can also be specified, the operation procedure according to the screen can be dynamically shown more accurately.

Claims (6)

An operation support system for supporting an operation by presenting an operation procedure of an information processing apparatus,
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with screen information displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on the screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by the user;
An operation support system comprising: an instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information. In the scenario information, screen specifying information including a graphic or character string specifying a screen related to a function related to operation support is described in association with an operation instruction necessary for the function related to the operation support,
The operation analysis unit refers to the scenario information, extracts screen specifying information from the screen information, determines an operation state by a user based on the extracted screen specifying information,
The operation support system according to claim 1, wherein the instruction output unit outputs an operation instruction described in the scenario information based on the operation state determined by the operation analysis unit. The scenario information is described in association with a screen related to a function related to operation support and operation target specifying information including a figure or a character string that specifies a target of an operation necessary for the function related to the operation support.
The operation analysis unit refers to the scenario information, extracts the operation target specifying information from the screen information, determines an operation state by the user based on the extracted operation target specifying information,
The instruction output unit outputs an operation instruction described in the scenario information according to the extracted operation target specifying information and a position on the screen based on the operation state determined by the operation analysis unit, and the screen information The operation support system according to claim 2, wherein the operation support system is displayed while being superimposed on the screen. A history storage unit that records, as an instruction history, the operation instruction output by the instruction output unit and the scenario information referred to when the operation instruction is displayed in the state information history acquired by the state acquisition unit When,
The history analysis unit further comprises a history analysis unit that aggregates and analyzes the instruction history stored in the history storage unit and holds the analysis result in association with the referenced scenario information so as to be output. The operation support system described. An operation support program for supporting an operation by presenting an operation procedure of the information processing apparatus, wherein the information processing apparatus is
A scenario acquisition unit that acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with screen information displayed on a display unit included in the information processing apparatus;
A status acquisition unit that acquires screen information currently displayed on the display means and status information related to the operating status of the information processing apparatus, as status information;
Based on screen information and status information included in the state information, an operation analysis unit that refers to the scenario information and analyzes an operation state by a user;
An operation support program that functions as an instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information. An operation support method for supporting an operation by presenting an operation procedure of an information processing apparatus,
A scenario acquisition step in which a scenario acquisition unit acquires scenario information indicating an operation procedure related to a function of the information processing apparatus in correspondence with a screen displayed on a display unit included in the information processing apparatus;
A state acquisition step in which the state acquisition unit acquires the screen information currently displayed on the display means and the status information on the operation state of the information processing apparatus as state information;
An operation analysis step for analyzing an operation state by a user with reference to the scenario information based on screen information and status information included in the state information,
An instruction output unit that outputs an operation instruction based on an analysis result by the operation analysis unit and the scenario information; and
An operation support method comprising: