CN115516405A - Operation support device, operation support method, and program - Google Patents
Operation support device, operation support method, and program Download PDFInfo
- Publication number
- CN115516405A CN115516405A CN202180028339.0A CN202180028339A CN115516405A CN 115516405 A CN115516405 A CN 115516405A CN 202180028339 A CN202180028339 A CN 202180028339A CN 115516405 A CN115516405 A CN 115516405A
- Authority
- CN
- China
- Prior art keywords
- screen
- unit
- control command
- information
- intention
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
- G06F3/005—Input arrangements through a video camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
Abstract
The model (31) is configured to be trained by machine learning with the information for outputting intention information indicating an intention of the question based on the question and the screen recognition information. An inference unit (21) acquires intention information from the model based on the question and screen identification information for identifying the description screen displayed when the input unit (11) has acquired the question. An inference unit (21) obtains a response corresponding to the acquired intention information, a control command corresponding to the intention information and screen identification information, and conversion destination screen identification information for identifying a description screen of a conversion destination corresponding to the intention information and the screen identification information. The application program outputs the response obtained by the inference unit (21), causes the control command execution unit (41) to execute the control command obtained by the inference unit (21), and displays a description screen corresponding to the conversion destination screen identification information obtained by the inference unit (21) on the display unit (43).
Description
Technical Field
The invention relates to an operation support device, an operation support method, and a program.
Background
In every industry, there is a need to provide a method that simply and appropriately responds to questions from the user regarding how to operate the device, etc.
CITATION LIST
Patent literature
Patent document 1: japanese unexamined patent application publication No.2005-107583
Disclosure of Invention
Technical problem
However, in each industry, products are becoming smaller and thinner, and the processes of manufacturing or testing such products are also becoming more complicated, and therefore, devices used in each of these processes are also becoming more complicated. In such a case, there is a problem in that the use of the device becomes more and more difficult and operation errors of the operator are easily caused. This problem affects companies and the like in each industry, and the direction of the effect is to increase the loss.
In the case where there is the above-described complexity, there is a problem that, in the case where a problem is posed by a user or the like who operates the target device, the intention of the problem may differ depending on the situation. In addition, there is a problem in that simply returning a response to a question from the user does not guarantee that the user has reliably understood all correct operation methods. In other words, the operation support apparatus must provide a response in accordance with the intention of the question, which depends on the situation. Further, it is desirable that the operation support apparatus be able to appropriately control the target apparatus as necessary, in addition to simply returning a response to the question.
The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an operation support device, an operation support method, and a program which are capable of not only responding to a surface layer of question content but also performing at least part of control of a target device on behalf of a user in some cases, depending on the circumstances.
Technical scheme
[1] In order to solve the above problem, according to an aspect of the present invention, there is provided an operation support apparatus including an input unit, a display unit, a control command execution unit, an inference unit, an application execution unit, a response storage unit, and a control command generation unit. The input unit is used for acquiring the question. The display unit displays a description screen for the target device. The control command execution unit executes a control command for controlling the target device. The inference unit acquires intention information from the model based on the model, the question, and screen identification information for identifying a description screen displayed when the input unit has acquired the question. The inference unit obtains a response corresponding to the acquired intention information, a control command corresponding to the intention information and screen identification information, and transformation destination screen identification information for identifying a description screen for a transformation destination corresponding to the intention information and the screen identification information. The model is configured to be trained by machine learning using the information to output intention information indicating an intention of the question based on the question and the screen recognition information. The application execution unit executes the application program for outputting the response obtained by the inference unit, causing the control command execution unit to execute the control command obtained by the inference unit, and displaying the description screen corresponding to the conversion destination screen identification information obtained by the inference unit on the display unit. The response storage unit stores a correspondence between the intention information and the response. The control command generating unit stores information on a control command and transformation destination screen identification information corresponding to the intention information and the screen identification information, and responds to an inquiry of the reasoning unit based on the intention information and the screen identification information with the control command and the transformation destination screen identification information corresponding to the inquiry. Further, the inference unit obtains a response corresponding to the intention information by referring to the response storage unit based on the acquired intention information. Further, the inference unit obtains the control command and the conversion destination screen identification information by querying the control command generation unit based on the acquired intention information and the screen identification information.
[2] In addition, the operation support apparatus according to an aspect of the present invention further includes a machine learning processing unit that provides training data containing questions, screen recognition information, and intention information to train the model by machine learning.
[3] Further, in the operation support apparatus according to an aspect of the present invention, the display unit displays the response output by the application execution unit.
[4] Further, in the operation support apparatus according to another aspect of the present invention, the display unit has an area for displaying an image captured by a camera included in the target apparatus.
[5] According to an aspect of the present invention, there is provided an operation support method including: the method includes acquiring a question by an input unit, displaying a description screen for a target device by a display unit, executing a control command for controlling the target device by a control command execution unit, providing a model trained by machine learning with information for outputting intention information indicating an intention of the question based on the question and screen identification information, acquiring the intention information from the model based on the question and screen identification information for identifying the description screen displayed when the input unit has acquired the question by an inference unit, and acquiring a response corresponding to the acquired intention information, control information corresponding to the intention information and the screen identification information, and transformation destination screen identification information for identifying the description screen for a transformation destination corresponding to the intention information and the screen identification information, and executing an application program by an application execution unit for outputting the inference response acquired by the inference unit, causing the control command execution unit to execute the control command acquired by the control command execution unit, and displaying the description screen corresponding to the transformation destination screen identification information acquired by the display unit on the display unit.
[6] According to an aspect of the present invention, there is provided a program for causing a computer to function as the operation support apparatus according to any one of [1] to [4] above.
Advantageous effects
According to the present invention, the operation support apparatus can respond to a question and control external hardware based on the intention according to the situation.
Drawings
Fig. 1 is a block diagram showing a schematic functional configuration of an operation support apparatus according to an embodiment of the present invention.
Fig. 2 is a schematic diagram (1/2) showing an example of a flow of information (signal) exchange in the operation support apparatus of the embodiment.
Fig. 3 is a schematic diagram (2/2) showing an example of a flow of information (signal) exchange in the operation support apparatus of the embodiment.
Fig. 4 is a schematic diagram showing a relationship between data processed by the operation support apparatus according to the embodiment, which shows a relationship between the question content and the screen ID and the interface.
Fig. 5 is a schematic diagram showing a relationship between data processed by the operation support apparatus according to the embodiment, which shows a relationship between an intention and response content.
Fig. 6 is a schematic diagram showing a relationship between data processed by the operation support apparatus according to the embodiment, which shows a relationship between an intention, a screen ID, a control command, and a conversion destination screen ID.
Fig. 7 is a table listing types of control commands to be used by the operation support apparatus according to the embodiment.
Fig. 8 is a schematic diagram showing a configuration example of a screen displayed by the display unit of the operation support apparatus according to the embodiment.
Fig. 9 is a diagram showing a configuration example of a management table for managing information on a description screen displayed by the operation support apparatus according to the embodiment.
Fig. 10 is a diagram illustrating an example of training data for training a learning model by machine learning according to an embodiment.
Fig. 11 is a diagram showing a case where the operation support apparatus according to the embodiment handles processing based on a specific problem example.
Detailed Description
Next, an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, the operation support apparatus estimates intention information indicating an intention of the question input by the user based on the question, and performs processing based on the intention information. Hereinafter, the intention information is also referred to as "intention". The intention is estimated based on the text (surface layer) of the problem and the situation at that time.
In addition, in the present embodiment, the "question" is a sentence or the like input by the user using the input unit 11. However, the "question" here is not limited to a sentence with a question sentence as a grammatical form. For example, as an expression input by the user, in the present embodiment, an arbitrary language expression (such as "i do not know o —," o does not work "," — "o does not appear", "i want to start o —," "i want to stop o —," i want to change the setting of o —) related to the operation of the apparatus is regarded as "problem".
The "screen ID" described in the embodiments is information for uniquely identifying the type of the description screen displayed on the operation support apparatus. The "screen ID" is also referred to as "screen identification information". In addition, the description screen of the next destination to be converted is referred to as a "conversion destination screen". The "conversion destination screen Identification (ID)" is information for uniquely identifying the type of the conversion destination screen. The "conversion destination screen ID" is also referred to as "conversion destination screen identification information". In addition, the description screen is a screen for describing a device (target device 3) as a target to be operated. The description screen may include text, images (still images), and video (moving images). Further, when the operation support apparatus displays the description screen, the description may also be output as a predetermined voice. Further, the control command is a command that the operation support apparatus can execute to control the external apparatus or the like. The "control command ID" is information for uniquely identifying the control command. The "control command ID" is also referred to as "control command identification information".
Fig. 1 is a block diagram showing a schematic functional configuration of an operation support apparatus according to the present embodiment. As shown, the operation support apparatus 1 may include an input unit 11, an application execution unit 12, an inference unit 21, a response storage unit 22, a control command generation unit 23, a learning model 31, a machine learning processing unit 32, a control command execution unit 41, a hardware control unit 42, and a display unit 43. Each of the functional units may be implemented by, for example, a computer and a program. Further, each functional unit has a storage mechanism as necessary. The storage mechanism is, for example, a memory allocated by a variable on a program or execution of the program. Also, as for the storage mechanism, a nonvolatile storage mechanism such as a magnetic hard disk device or a Solid State Drive (SSD) may be used as necessary. In addition, at least some of the functions of each functional unit may be implemented by dedicated electronic circuitry rather than programs.
Specifically, the operation support apparatus 1 is constructed using, for example, a Personal Computer (PC) or various portable terminal apparatuses (e.g., a tablet-type terminal apparatus, a smartphone, or the like). The user who wants to operate the target apparatus 3 operates the operation support apparatus 1.
The target device 3 is a device to be operated or the like. The target device 3 is, for example, a device used at a manufacturing site or a service implementation site. The target device 3 is, for example, a manufacturing device, a processing device, a testing/inspecting device, a transfer device, or the like (however, the target device 3 is not limited thereto). The target device 3 may include a camera therein. For example, when the target device 3 is an inspection device for inspecting a semiconductor product, the camera photographs the semiconductor product to be inspected. The image captured by the camera is originally an image to be processed in the target apparatus 3, but a signal of the image may be transmitted to the operation support apparatus 1. In this case, the image captured by the camera may be displayed on the display screen of the operation support apparatus 1. The operation support apparatus 1 and the target apparatus 3 are connected by a wired or wireless communication mechanism. The operation support apparatus 1 and the target apparatus 3 may be interconnected through a communication line such as the internet.
The function of each of the units constituting the operation support apparatus 1 will be described later.
The input unit 11 has a function of acquiring an input from a user. The input unit 11 acquires a character input signal from a keyboard, for example. In addition, the input unit 11 acquires a character input signal through the touch panel. Alternatively, the input unit 11 may have a voice input function of taking the voice of the user by a microphone or the like and performing voice recognition processing.
The application execution unit 12 is a mechanism for executing an application program. Specifically, the application execution unit 12 includes a processing unit (CPU or the like) for executing an application program and a memory for storing data or the like. The application execution unit 12 executes a dedicated application program for controlling the target device 3.
By executing the application program, the application execution unit 12 acquires the text of the question from the input unit 11. In addition, the application execution unit 12 passes the text of the question to the inference unit 21. In addition, the application execution unit 12 holds a screen ID for identifying the description screen being displayed. When the above-described problem is input, the application execution unit 12 passes the screen ID to the inference unit 21. The application execution unit 12 receives and outputs a response obtained by the inference unit 21 based on the question and the screen ID. In addition, the application execution unit 12 causes the control command execution unit 41 to execute the control command obtained by the inference unit 21. In addition, the application execution unit 12 causes the display unit 43 to display the description screen corresponding to the conversion destination screen ID obtained by the inference unit 21. In this way, in response to the input question, the application program can output a response, execute a control command, or display a specific description screen (description of the target apparatus 3).
The inference unit 21 performs inference processing based on the input question. Specifically, the inference unit 21 receives the question text and the screen ID (screen identification information) from the application execution unit 12. The screen ID is identification information for identifying a description screen displayed when the input unit 11 has acquired a question. The inference unit 21 transmits the question and the screen ID to the learning model 31, and receives an intention (intention information) from the learning model 31 as a response thereto. In addition, the inference unit 21 refers to the response storage unit 22 based on the intention to obtain a response corresponding to the intention. Further, the inference unit 21 refers to the control command generation unit 23 based on the intention and the screen ID to obtain a corresponding conversion destination screen ID (conversion destination screen identification information). The conversion destination screen ID is identification information of a description screen for identifying a next destination to be converted. The inference unit 21 returns the response, the control command, and the conversion destination screen ID obtained as described above to the application execution unit 12.
The response storage unit 22 stores a response associated with the intention (intention information). In other words, the answer related to the intention can be read by referring to the answer storage unit 22. The response is presented, for example, as text data.
The control command generating unit 23 stores information for a control command and a conversion destination screen ID (conversion destination screen identification information) in association with a combination of an intention (intention information) and a screen ID (screen identification information). The control command generating unit 23 responds to the inquiry from the inference unit 21 with a control command corresponding to the intention and the screen ID and the conversion destination screen ID based on the stored information.
The learning model 31 has information for outputting an intention (intention information) indicating an intention of a question based on the question and a screen ID (screen identification information). In the learning model 31, machine learning is performed with the relationship between the question and the screen ID as input information and the intention as output information. The learning model 31 is constructed using, for example, a neural network.
The machine learning processing unit 32 has a function of providing training data to the learning model 31 so that the learning model 31 performs machine learning processing. The training data is data of a set of screen ID (screen identification information), question, and intention (intention information). In addition, when the machine learning of the learning model 31 is completed, the operation support apparatus 1 may be configured to omit the machine learning processing unit 32.
The control command execution unit 41 executes a command for controlling an external device or the like. The control command execution unit 41 executes, for example, a command for controlling the target device 3. The control of the external device is performed by outputting a signal from the input/output port, for example. In addition, the control command execution unit 41 may execute a command for displaying characters or images on the display unit 43. Specifically, the control command execution unit 41 may display a question text, a response text, a description screen, or the like on the display unit 43.
The hardware control unit 42 controls external hardware (the target apparatus 3 and the like) according to the control command executed by the control command execution unit 41. The hardware control unit 42 specifically outputs a control signal from an I/O port or the like.
The display unit 43 displays an image, text, or the like. Specifically, the display unit 43 displays a description screen for describing the target apparatus 3, displays text of a question input by the user, displays response text output by the operation support apparatus 1, or displays an image based on an image signal from the outside. The image signal from the outside is, for example, a signal of an image (video) captured by a camera included in the target apparatus 3. A configuration example of the screen displayed by the display unit 43 will be described later with reference to the drawings.
Fig. 2 and 3 are schematic diagrams showing an example of a flow of information (signal) exchange in the operation support apparatus 1. Hereinafter, an example of the operation sequence of the operation support apparatus 1 will be described according to the schematic diagram.
First, in step S1 of fig. 2, the input unit 11 acquires a question from the user and passes it to the application execution unit 12. The question is obtained using a mechanism such as key operation on a keyboard or voice recognition. The application execution unit 12 acquires a question in the form of text data. The application execution unit 12 displays a predetermined description screen on the display unit 43 even at the time point of acquiring the question. The application execution unit 12 holds the screen ID of the description screen currently being displayed.
Next, in step S2, the application execution unit 12 passes the screen ID of the currently described screen and the text of the question acquired in step S1 to the inference unit 21. Thereby, the inference unit 21 starts inference processing.
Next, in step S3, the inference unit 21 transmits the screen ID and question acquired in step S2 to the learning model 31. The learning model 31 has completed machine learning. The learning model 31 obtains an intention suitable for the screen ID and the question from the result of the machine learning.
Next, in step S4, the learning model 31 sends the intention obtained in step S3 to the inference unit 21. The inference unit 21 receives the intention.
Next, in step S5, the inference unit 21 performs access with reference to the response storage unit 22 based on the intention received in step S4. The response storage unit 22 specifies a response corresponding to the intention based on the stored information.
Next, in step S6, the response storage unit 22 sends a response corresponding to the intention to the inference unit 21. In other words, the inference unit 21 reads information of the response corresponding to the intention from the response storage unit 22. In addition, what the inference unit 21 obtains is text data such as a response.
Next, in step S7, the inference unit 21 transmits the screen ID received in step S2 and the intention received in step S4 to the control command generation unit 23. The control command generating unit 23 receives information on the above items. The control command generating unit 23 stores information on a control command corresponding to an input combination of the screen ID and the intention and the conversion destination screen ID in advance. In other words, the control command generating unit 23 generates information about the control command and the conversion destination screen ID corresponding to the combination of the received screen ID and the intention.
Next, in step S8, the control command generating unit 23 sends the control command and the conversion destination screen ID generated in step S7 to the inference unit 21. In other words, the inference unit 21 acquires the control command and the conversion destination screen ID based on the screen ID and the intention.
Next, in step S9, the inference unit 21 delivers the response obtained in step S6 and the control command and the conversion destination screen ID obtained in step S8 to the application program running on the application running unit 12. The application execution unit 12 receives the response, the control command, and the conversion destination screen ID.
Next, referring to fig. 3, the application program running in the application running unit 12 in step S10 sends the text of the response received in step S9 to the control command execution unit 41. In addition, the application program sends the control command received in step S9 to the control command execution unit 41. Further, the application program sends the screen identified by the conversion destination screen ID received in step S9 to the control command execution unit 41.
Next, in step S11, the control command execution unit 41 sends the control command received in step S10 to the hardware control unit 42. Thereby, the hardware control unit 42 can execute the control command. The control command is, for example, a command for controlling the target device 3. In other words, the control command execution unit 41 may control the behavior of the target apparatus 3, for example, by executing the control command.
Next, in step S12, the control command execution unit 41 transfers the text of the response received in step S10 or the screen identified by the conversion destination screen ID (signal indicating the screen) to the display unit 43. Thereby, the display unit 43 can display a screen of a conversion destination (screen for describing the target apparatus 3). In addition, the display unit 43 may display the response output by the inference unit 21.
Fig. 4, 5, and 6 show the relationship between data processed by the operation support apparatus 1.
Fig. 4 is a schematic diagram showing the relationship between the screen ID and the intention and the question content in the processing in the operation support apparatus 1. Here, the question content is a text of a question acquired from the user by the input unit 11. The screen ID is identification information for identifying a screen displayed when the input unit 11 inquires about a question from the user. In other words, the screen ID is information indicating a situation where a question has been posed. The intention is intention information indicating the intention of the question content in this case. As described with reference to fig. 2, the learning model 31 estimates and outputs an intention based on the input question content and screen ID. Since the learning model 31 has completed machine learning, the learning model 31 has intention information corresponding to the input question content and the screen ID. The learning model 31 does not necessarily have decisive question contents and relationship information between the screen ID and the intention. However, the learning model 31 has at least information about possible intentions corresponding to the question contents and the screen ID. The learning model 31 may estimate the intention based on such information.
Fig. 5 is a schematic diagram showing a relationship between the intention and the response content in the processing in the operation support apparatus 1. Here, as described above, the intention is information output from the learning model 31. The response content is text of a response to be output. As already described, the response storage unit 22 stores the intention associated with the response content. In other words, by searching the response storage unit 22 while using the intention as a key, the response content corresponding to the intention can be acquired. In other words, the response storage unit 22 stores data corresponding to the conversion table from the intention to the response content. The inference unit 21 can perform conversion from intention to answer content by referring to the answer storage unit 22.
Fig. 6 is a schematic diagram showing the relationship between the intention and the screen ID in the processing in the operation support apparatus 1 and the control command and the conversion destination screen ID. As described above, the control command generating unit 23 stores information about the correspondence between the intention and the screen ID and the control command and the conversion destination screen ID. In other words, the control command generating unit 23 stores data corresponding to a conversion table for converting a combination of the intention and the screen ID into the control command and the conversion destination screen ID. When receiving the intention and the screen ID from the inference unit 21, the control command generation unit 23 can determine the control command to be output and the conversion destination screen ID by referring to the data corresponding to the conversion table.
As described above with reference to fig. 4, 5, and 6, the operation support apparatus 1 has information and processing logic for obtaining the response content (text), the control command, and the conversion destination screen ID based on the question content (text) and the screen ID.
Fig. 7 is a table listing types of control commands to be used by the operation support apparatus 1. As shown, the control target of the control command includes the target device 3. The operation support apparatus 1 can use several kinds of control commands for each control target apparatus. Here, for convenience, a control command ID for identifying a control command is provided. For example, the control command ID of the control command for the target device 3 as the control target starts with "C-se:Sub>A-" and includes se:Sub>A serial number. For example, se:Sub>A control command with se:Sub>A control command ID of C-A-01 is se:Sub>A command for "power on". The control command with the control command ID C-A-02 is se:Sub>A command for "power off". The control command with the control command ID C-se:Sub>A-03 is se:Sub>A command for "initializing the motor system". The control command with the control command ID of C-se:Sub>A-04 is se:Sub>A command for "transfer start". The control command having the control command ID of C-se:Sub>A-05 is se:Sub>A command for powering on the camerse:Sub>A of the target apparatus 3. The control command having the control command ID of C-se:Sub>A-06 is se:Sub>A command for powering off the camerse:Sub>A of the target device 3. There may be further control commands.
The control command symbol string shown in fig. 7 is a code actually executed by the control command execution unit 41. The control command symbol string may be represented as, for example, a character string, a hexadecimal string, or a bit string.
The operation support apparatus 1 may execute the control command shown in fig. 7 or other control commands. Specifically, as described above, the control command generating unit 23 appropriately selects and generates a control command according to a specific situation. The control command execution unit 41 may execute the generated control command. In addition, although a single command is enumerated in fig. 7, the control command generating unit 23 may generate a control command for sequentially executing a plurality of commands or a control command for executing a plurality of commands in parallel. In addition, the control command execution unit 41 also executes such commands.
Fig. 8 is a schematic diagram showing a configuration example of a screen displayed by the display unit 43 of the operation support apparatus 1. The screen is displayed using, for example, a color liquid crystal display or the like. As shown, the screen 1001 includes a description screen display area 1002, a captured image display area 1003, and a question and answer display area 1004. The illustrated arrangement of the description screen display area 1002, the captured image display area 1003, and the question and answer display area 1004 is exemplary, and other arrangements may be made. Each of the description screen display area 1002, the captured image display area 1003, and the question and answer display area 1004 may be scrollable or non-scrollable.
The description screen display area 1002 is a screen display area for displaying a description of the operation method of the target device 3 and the like. In the illustrated example, the description screen display area 1002 displays a screen ID (G00101) describing the screen and a title "operation method" of the screen.
The captured image display area 1003 is an area for displaying an image (a still image or a moving image) captured by the camera of the target apparatus 3. In other words, the captured image display area 1003 may display an image output by the target apparatus 3.
The question and answer display area 1004 is an area for displaying the text of the question acquired by the operation support apparatus 1 and the text of the answer output by the operation support apparatus 1 in response to the question. In the example shown, the question is displayed in the language balloon on the right side in the area. In addition, the response is displayed in a language balloon in the area. The questions and responses are displayed in chronological order. In addition, the display method of the question and the response is exemplary. Other display methods may be used to display the question or the response.
Fig. 9 is a schematic diagram showing a management table of a description screen managed by an application program operated by the application execution unit 12. As shown in the figure, the management table is relational data containing a screen ID, a screen title, and each item in the screen definition. The application program can access a management table stored in the operation support apparatus 1 or external. The screen ID is identification information for uniquely identifying the type of the description screen. The screen title is a character string representing the title of the screen. As shown in fig. 8, for example, a screen ID or a title may be displayed in the description screen display area 1002. The screen definition is definition information about the configuration of the screen. The application program can configure a screen by referring to the screen definition and transmit an image signal for presenting the screen to the display unit 43. The information for screen definition may include images, videos (moving images), texts, and the like. In the illustrated example, the management table may hold screen information having titles such as "operation method", "error message", "initialization method", and "charging warning display", which correspond to respective unique screen IDs.
Fig. 10 is a diagram showing an example of training data for training the learning model 31 by machine learning. As shown in the figure, the training data is data of a set of screen ID (screen identification information), question, and intention (intention information). The first row in the illustrated example of data is the following data: the screen ID is "G02231", and the problem is "no image is seen from the camera", and the intention is "no image of the camera is seen". The machine learning processing unit 32 supplies a large amount of data having the structure shown in fig. 10 to the learning model 31 as training data, so that the learning model 31 is trained by machine learning. Through machine learning, when a question is given, the learning model 31 can estimate the screen ID and the corresponding intention.
Fig. 11 is a schematic diagram showing an operation state of the operation support apparatus 1 based on a specific problem example. In the illustrated example, the input unit 11 passes the question "the camera screen is black" to the application execution unit 12. The application execution unit 12 passes the text of the question and the screen ID (G02232) of the currently displayed description screen to the inference unit 21. The inference unit 21 passes the text and screen ID of the question to the learning model 31. The learning model 31 estimates the intention by the action of the model based on the text and the screen ID of the question received from the inference unit 21. In this example, the estimated intent is "no view of the camera's screen". In addition, the intent may be in the form of text as described above, or may be in the form of any symbol. The learning model 31 returns an intention to the inference unit 21. Based on the intention, the inference unit 21 acquires a response "you can check the power of the camera on the screen" from the response storage unit 22. In addition, the inference unit 21 acquires the conversion destination screen ID (G02501) and a control command for performing control of "camera power on" from the control command generation unit 23 based on the intention and the screen ID (G02232). Therefore, the application program running in the application running unit 12 can execute the following processing: the output answer "you can check the power on the screen of the camera", display the description screen identified by the conversion destination screen ID (G02501), and execute the control command corresponding to "camera power on".
The processing shown in fig. 11 is an example, and the operation support apparatus 1 can output various responses based on the information stored in the response storage unit 22. In addition, the operation support apparatus 1 can execute various control commands or display various description screens based on the information stored in the control command generating unit 23.
For example, in response to the intention "i want to initialize the motor system", the operation support apparatus 1 may display a description screen on initializing the motor system to the target apparatus 3 or execute a control command on initializing the motor system for the target apparatus 3. In addition, the operation support apparatus 1 can perform various operations such as starting or stopping the operation of the target apparatus 3 or changing the setting of the target apparatus 3.
Further, at least some of the functions of the operation support apparatus and the like in the above-described embodiments may be implemented by a computer. In this case, the embodiment can be realized by recording a program for implementing functions on a computer-readable recording medium and reading the program recorded on the recording medium into a computer system and executing the program. Further, a "computer system" referred to herein includes an OS and hardware such as peripheral devices. Further, the "computer-readable recording medium" refers to portable media such as floppy disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, and USB-memories, and storage devices such as hard disks embedded in computer systems. In addition, the "computer-readable recording medium" may also include those that temporarily and dynamically hold the program (for example, a communication line when the program is transmitted through a network such as the internet or a communication line such as a telephone line), and those that hold the program for a certain period of time (for example, a volatile memory within a computer system that is a server or a client in the above cases). Further, the program may be designed to implement some of the above-described functions, or may be designed to implement the above-described functions in conjunction with a program that has been recorded in a computer system.
Although the embodiments have been described above, the present invention can also be implemented in the following modified examples.
[ modified example ]
In the above embodiment, the case where the answer text is displayed on the screen has been described; alternatively, the response voice may be output using a recorded message or synthesized voice.
The embodiments of the present invention have been described above with reference to the drawings, but the specific configuration is not limited to the embodiments and also includes designs within a scope not departing from the scope of the present invention.
Industrial applications
The present invention can be used for the purpose of, for example, supporting the operation of devices and the like in all industries. However, the scope of use of the present invention is not limited to those uses exemplified herein.
Description of the symbols
1. Operation support device
3. Target device
11. Input unit
12. Application execution unit
21. Inference unit
22. Response storage unit
23. Control command generating unit
31. Learning model
32. Machine learning processing unit
41. Control command execution unit
42. Hardware control unit
43. Display unit
1001. Screen
1002. Describing screen display areas
1003. Capturing an image display area
1004. Question and answer display area
Claims (6)
1. An operation support apparatus comprising:
an input unit for acquiring a question;
a display unit that displays a description screen for a target device;
a control command execution unit that executes a control command for controlling the target device;
a model configured to be trained by machine learning with information for outputting intention information indicating an intention of the question based on the question and screen recognition information;
an inference unit that acquires the intention information from the model based on the question and screen identification information, and obtains a response corresponding to the acquired intention information, a control command corresponding to the intention information and the screen identification information, and transformation destination screen identification information for identifying a description screen displayed when the input unit has acquired the question, the transformation destination screen identification information for identifying a description screen of a transformation destination corresponding to the intention information and the screen identification information;
an application execution unit that executes an application program for outputting the response obtained by the inference unit, causing the control command execution unit to execute the control command obtained by the inference unit, and displaying a description screen corresponding to the conversion destination screen identification information obtained by the inference unit on the display unit;
a response storage unit that stores a correspondence relationship between the intention information and the response; and
a control command generating unit that stores information on the control command and the transformation destination screen identification information corresponding to the intention information and the screen identification information, and responds to the query based on the intention information and the screen identification information from the inference unit with the control command and the transformation destination screen identification information corresponding to the query,
wherein the inference unit obtains the response corresponding to the intention information by referring to the response storage unit based on the acquired intention information, and
the inference unit obtains the control command and the conversion destination screen identification information by querying the control command generation unit based on the acquired intention information and the screen identification information.
2. The operation support apparatus according to claim 1, further comprising a machine learning processing unit that provides training data containing the question, the screen identification information, and the intention information to train the model by machine learning.
3. The operation support apparatus according to claim 1 or 2, wherein the display unit displays the response output by the application execution unit.
4. The operation support device according to any one of claims 1 to 3, wherein the display unit has an area for displaying an image captured by a camera included in the target device.
5. An operation support method comprising:
obtaining a question by an input unit;
displaying, by a display unit, a description screen for a target device;
executing, by a control command execution unit, a control command for controlling the target device;
providing a model configured to be trained by machine learning with information for outputting intention information indicating an intention of the question based on the question and screen recognition information;
acquiring, by an inference unit, the intention information from the model based on the question and screen identification information, and obtaining a response corresponding to the acquired intention information, a control command corresponding to the intention information and the screen identification information, and transformation destination screen identification information for identifying a description screen displayed when the input unit has acquired the question, the transformation destination screen identification information for identifying a description screen of a transformation destination corresponding to the intention information and the screen identification information;
executing, by an application execution unit, an application program for outputting the response obtained by the inference unit, causing the control command execution unit to execute the control command obtained by the inference unit, and displaying on the display unit a description screen corresponding to the conversion destination screen identification information obtained by the inference unit;
storing, by a response storage unit, a correspondence between the intention information and the response; and
storing, by a control command generating unit, information on the control command and the transformation destination screen identification information corresponding to the intention information and the screen identification information, and responding to the query from the inference unit based on the intention information and the screen identification information with the control command and the transformation destination screen identification information corresponding to the query,
wherein the inference unit obtains the response corresponding to the intention information by referring to the response storage unit based on the acquired intention information, and
the inference unit obtains the control command and the conversion destination screen identification information by querying the control command generation unit based on the acquired intention information and the screen identification information.
6. A program that causes a computer to function as the operation support apparatus according to any one of claims 1 to 4.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2020083098A JP6886738B1 (en) | 2020-05-11 | 2020-05-11 | Operation support equipment, operation support methods, and programs |
| JP2020-083098 | 2020-05-11 | ||
| PCT/JP2021/017665 WO2021230190A1 (en) | 2020-05-11 | 2021-05-10 | Operation support device, operation support method, and program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115516405A true CN115516405A (en) | 2022-12-23 |
Family
ID=76310195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202180028339.0A Pending CN115516405A (en) | 2020-05-11 | 2021-05-10 | Operation support device, operation support method, and program |
Country Status (4)
| Country | Link |
|---|---|
| JP (1) | JP6886738B1 (en) |
| KR (1) | KR20220113534A (en) |
| CN (1) | CN115516405A (en) |
| WO (1) | WO2021230190A1 (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000123028A (en) * | 1998-10-13 | 2000-04-28 | Mitsubishi Electric Corp | Procedure base help disk system method and device for retrieving example |
| JP2004021325A (en) * | 2002-06-12 | 2004-01-22 | Toshiba Corp | Communication controller and communication control method |
| JP2005107583A (en) | 2003-09-26 | 2005-04-21 | Toshiba Corp | Device operation guidance system |
| JP4124142B2 (en) * | 2004-03-02 | 2008-07-23 | 日本電気株式会社 | Answer creation support server, answer creation support method, and answer creation support program |
| JP4172802B2 (en) * | 2005-12-28 | 2008-10-29 | インターナショナル・ビジネス・マシーンズ・コーポレーション | System that supports answering inquiries received from users |
| US10909329B2 (en) * | 2015-05-21 | 2021-02-02 | Baidu Usa Llc | Multilingual image question answering |
| JP7059929B2 (en) * | 2016-05-30 | 2022-04-26 | ソニーグループ株式会社 | Information processing equipment |
| JP7256357B2 (en) * | 2018-10-31 | 2023-04-12 | キヤノンマーケティングジャパン株式会社 | Information processing device, control method, program |
-
2020
- 2020-05-11 JP JP2020083098A patent/JP6886738B1/en active Active
-
2021
- 2021-05-10 KR KR1020227025337A patent/KR20220113534A/en not_active Application Discontinuation
- 2021-05-10 CN CN202180028339.0A patent/CN115516405A/en active Pending
- 2021-05-10 WO PCT/JP2021/017665 patent/WO2021230190A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021230190A1 (en) | 2021-11-18 |
| JP2021179655A (en) | 2021-11-18 |
| JP6886738B1 (en) | 2021-06-16 |
| KR20220113534A (en) | 2022-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11164091B1 (en) | Natural language troubleshooting engine | |
| US9495145B1 (en) | Customized electronic product configuration | |
| US20110029103A1 (en) | Multiple language development environment using shared resources | |
| US20190294912A1 (en) | Image processing device, image processing method, and image processing program | |
| US10795886B1 (en) | Dynamic query routing system | |
| US10936815B2 (en) | Removable spell checker device | |
| JP2023001377A (en) | Information processing apparatus, method, and program | |
| Soad et al. | Quality evaluation of mobile learning applications | |
| JP2019125145A (en) | Device, method, and program for processing information | |
| CN109240931A (en) | Problem feedback information treating method and apparatus | |
| US20180247552A1 (en) | Study-support system, and associated devices and methods | |
| CN110276183B (en) | Reverse Turing verification method and device, storage medium and electronic equipment | |
| CN115516405A (en) | Operation support device, operation support method, and program | |
| CN115858049A (en) | RPA flow componentization arrangement method, device, equipment and medium | |
| JP2023028401A (en) | Information processing device, information processing method, and information processing program | |
| CN102999448A (en) | Method and device for detecting external equipment | |
| US20170277501A1 (en) | Information display device, information display method and storage medium | |
| US20220254159A1 (en) | Information processing device, learning device, and method for controlling information processing device | |
| KR20150074608A (en) | System for providing personally custom made study service | |
| KR102658967B1 (en) | Method, electronic device, and system for providing solution based on artificial intelligence | |
| JP2019133418A (en) | Search device, search method, program, and database | |
| JP7010987B2 (en) | Learning support systems, learning support methods and programs | |
| US20240111961A1 (en) | System and method for incorporating risk related topics into chatbot functionality | |
| JP6925155B2 (en) | Generation program, generation method, and generation device | |
| KR100611052B1 (en) | Internet studying service system for providing studying data using pop-up window and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |