JP2022138750A - Question-answering system, information processing apparatus, information processing method, and program - Google Patents

Abstract

To solve the problem that, in prior arts, when a question-answering system cannot answer to a prescribed question, ambiguity remains even in an additionally inputted question because an additional request given for answering to the prescribed question is indistinct.SOLUTION: A transmission/reception unit 31 of an information processing apparatus 3 transmits an answer to a question, which is generated by an answer generation unit 37, to an input device 2 (step S14) and receives at least one of image information and speech information newly inputted for additional request information included in the response to the question, from the input device 2 (step S12).SELECTED DRAWING: Figure 5

Description

The present invention relates to a question answering system, an information processing device, an information processing method and a program.

As one of applications using natural language processing, there is a question answering system that automatically generates answers to user questions. The most common question answering setting is the one-question-one-answer setting in which the user's question is answered once. On the other hand, there is known a technique of generating an answer while interacting with the user when the user's question is ambiguous.

Also, in recent years, there is a field called VQA (Visual Question Answering) in which an image is given and a question is answered with reference to the image. In this field, the automatic response device 100 applies each algorithm of speech recognition and image recognition when the reception unit 10 receives an inquiry that is a combination of voice and image from the customer terminal 200 for the Nth inquiry. There is a known technique for converting voice data into text corresponding to the content of a customer's conversation indicated by voice data (see Patent Document 1, for example).

However, in the conventional technology, when an answer to a given question cannot be answered, the additional request to be given for answering is unclear, so ambiguity remains even in the additionally input question. I had a problem.

In order to solve the above-described problems, the invention according to claim 1 provides an input device for inputting a predetermined question, and an information processing device for responding to the predetermined question transmitted by the input device. A system, wherein the information processing device includes receiving means for receiving question information representing the predetermined question transmitted by the input device, and answer information representing an answer to the predetermined question, or generating the answer information. transmitting means for transmitting, to the input device, additional request information representing an additional request associated with the predetermined question in order to perform the additional request transmitted by the information processing device; display control means for displaying request information on a display means; and transmission means for transmitting at least one of image information and audio information input in response to the additional request information to the information processing apparatus. To provide a question answering system characterized by

As described above, according to the present invention, it is possible to eliminate the ambiguity of the additionally input question when the answer to the predetermined question cannot be answered.

1 is an overall configuration diagram showing an example of a question answering system; FIG. It is a figure which shows an example of the hardware constitutions of an input device and an information processing apparatus. It is a figure showing an example of functional composition of a question answering system. 4 is a conceptual diagram showing an example of a maintenance information management table; FIG. 4 is a conceptual diagram showing an example of a component configuration management table; FIG. FIG. 4 is a conceptual diagram showing an example of an additional information management table; FIG. 10 is a sequence diagram showing an example of question answering processing; It is an example of a screen display of a question reception screen which receives input of a question. It is a flow chart which shows an example of generation processing of reply information and additional information. It is a screen display example of a response notification screen displaying the content of the response notification. It is a screen display example of an additional information reception screen for receiving input of additional information.

DETAILED DESCRIPTION OF THE INVENTION Embodiments for carrying out the invention will be described below with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and if there are overlapping parts, the description thereof will be omitted.

[Embodiment]
This embodiment will be described with reference to FIGS. 1 to 9. FIG.

[Overall configuration of question answering system 1]
FIG. 1 is an overall configuration diagram showing an example of a question answering system according to this embodiment. The question answering system 1 shown in FIG. 1 is an example in which an answer generated by an information processing device 3 is responded to an input device 2 that has given a question to the information processing device 3 . The question answering system 1 is, for example, a system in which an information processing device 3 generates an answer, which is one of the tasks of natural language processing, in response to a question generated based on text data such as characters input by an input device 2. is. As shown in FIG. 1, a question answering system 1 communicates with an input device 2 used by a user who inputs a question, and communicates with the input device 2, processes the contents of the question sent by the input device 2, and inputs the question. It has an information processing device 3 that responds an answer to the device 2 . Also, the input device 2 and the information processing device 3 are connected to each other via a communication network 100 . Here, the communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), or the like. The communication network 100 includes not only wired communication but also wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). A network of communications may be included.

<Input device>
The input device 2 is realized by an information processing device (computer system) on which a general OS is installed. The input device 2 transmits the input text information to the information processing device 3 via the communication network 100 . Further, the input device 2 receives the text information transmitted by the information processing device 3, converts the text information into voice (sound) information, and outputs the voice information to the outside through a speaker. Further, the input device 2 converts the received text information into screen information to be displayed on the display means, and causes the display means to display the screen information. Also, the input device 2 inputs speech (sound) information such as human speech (natural language) or machine-generated speech obtained through a microphone, converts the speech (sound) information into text information, It is also possible to transmit to the information processing device 3 via the communication network 100 . Note that the input device 2 may be, for example, a communication terminal having a communication function such as a smart phone, a tablet terminal, a PDA (Personal Digital Assistant), or a wearable PC (sunglass type, wristwatch type, etc.). Furthermore, the input device 2 may be a general PC (Personal Computer). That is, the input device 2 may be a terminal capable of operating software such as browser software.

<Information processing device>
The information processing device 3 is realized by an information processing device (computer system) having a general OS installed and having a server function. The information processing device 3 communicates with the input device 2 via the communication network 100, and performs machine learning (hereinafter, the term “deep learning,” which is an example of machine learning) is used to process data related to questions transmitted by the input device 2. Processing is performed to generate answer (response) information related to the question and additional request information for accepting additional questions, and transmit them to the input device 2 . Note that the information processing device 3 may be a general PC as long as it has a configuration capable of generating the knowledge source 300 described above.

●Terms●
The “question answering” (hereinafter simply referred to as “question answering”) used in the present embodiment means a user (questioner) who reads and understands an input question using deep learning and pinpoints desired information. ).

[Hardware configuration]
<Hardware Configuration of Input Device and Information Processing Device>
Next, the hardware configuration of each device according to the embodiment will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the input device and the information processing device according to this embodiment. The input device 2 includes, as shown in FIG. RW (Digital Versatile Disk-Rewritable) drive 214, media I/F 216, microphone 218, speaker 219, sound input/output I/F 217, camera 220, imaging element I/F 221, and hardware resources including bus line 210 .

Among these, the CPU 201 controls the operation of the entire input device 2 . The ROM 202 stores programs used to drive the CPU 201 . A RAM 203 is used as a work area for the CPU 201 . The HD 204 stores various data such as programs. The HDD controller 205 controls reading or writing of various data to/from the HD 204 under the control of the CPU 201 . A display 206 displays various information such as cursors, menus, windows, characters, numeric keys, execution keys, and images. The display 206 is an example of display means. The external device connection I/F 208 is an interface for connecting various external devices. The external device in this case is, for example, a USB memory or a USB device. A bus line 210 is an address bus, a data bus, or the like for electrically connecting components such as the CPU 201 .

A network I/F 209 is an interface for data communication using the communication network 100 . The keyboard 211 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 212 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, or the like. Input means may be not only the keyboard 211 and the pointing device 212 but also a touch panel, a voice input device, or the like. A DVD-RW drive 214 controls reading or writing (storage) of various data to a DVD-RW 213 as an example of a removable recording medium. It should be noted that not only DVD-RW but also DVD-R or Blu-ray (registered trademark) Disc (Blu-ray Disc) may be used. A media I/F 216 controls reading or writing of data to a recording medium 215 such as a flash memory. The microphone 218 is a kind of voice input means for inputting voice and ambient sounds (sound signals). The speaker 219 is a kind of audio output means for outputting an output sound signal obtained by converting an input sound signal. A sound input/output I/F 217 is a circuit for processing input/output of sound signals between the microphone 218 and the speaker 219 under the control of the CPU 201 . The camera 220 is a type of built-in image capturing means that captures an image of a subject under the control of the CPU 201 and obtains image data. The imaging device I/F 221 is a circuit that controls driving of the camera 220 .

The information processing device 3 is constructed by a computer, and as shown in FIG. It has hardware resources including an I/F 309 keyboard 311 , pointing device 312 , media I/F 316 and bus line 310 .

Of these, the CPU 301-pointing device 312 has the same configuration as each hardware resource from the CPU 201-pointing device 212 of the input device 2, so detailed description thereof will be omitted. A media I/F 316 controls reading or writing (storage) of data to a recording medium 315 such as a flash memory. Here, when the information processing device 3 is a general PC, the DVD-RW drive 214, the microphone 218, the speaker 219, the sound input/output I/F 217, the camera 220, and the imaging device I/F 221 provided in the input device 2 Corresponding hardware resources may be provided.

The computer shown in FIG. 2 is an example, and may be a HUD (Head Up Display) device, an industrial machine, a network appliance, a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), or the like. good too.

Furthermore, each of the above programs may be recorded in a computer-readable recording medium in an installable format or executable format file, or may be downloaded via a network and distributed. Examples of recording media include CD-R (Compact Disc Recordable), DVD, Blu-ray (registered trademark) Disc, SD card, and USB memory. Also, the recording medium can be provided domestically or internationally as a program product. For example, the information processing device 3 implements the information processing method according to the present invention by executing the program according to the present invention.

[Functional configuration of question answering system]
Next, the functional configuration of each device according to the embodiment will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 is a diagram showing an example of the functional configuration of the question answering system according to this embodiment.

<Functional configuration of input device>
As shown in FIG. 3 , the input device 2 has a transmission/reception section 21 , an operation reception section 22 , an input/output section 23 , a display control section 24 , a judgment section 25 , a conversion generation section 27 and a memory/read section 29 . 2 operates according to instructions from the CPU 201 according to the program for the input device 2 developed on the RAM 203 from at least one of the ROM 202 and the HD 204. It is a function or means realized by The input device 2 also has a storage unit 2000 constructed by at least one of the ROM 202 and the HD 204 shown in FIG. The storage unit 2000 also stores an input processing program executed by the input device 2 . Further, the storage unit 2000 stores and manages a communication application for communicating with the information processing device 3 .

<<Each functional configuration of the input device>>
Next, each functional configuration of the input device 2 will be described in detail. The transmission/reception unit 21 of the input device 2 shown in FIG. 3 is mainly realized by the processing of the external device connection I/F 208 and the network I/F 209 shown in FIG. to transmit and receive various data (or information) to and from the information processing device 3 . In this embodiment, the transmitting/receiving unit 21 functions as an example of transmitting means and receiving means.

The operation reception unit 22 is mainly implemented by the CPU 201 processing signals generated by various operations received by the keyboard 211 and pointing device 212 shown in FIG. Accept input for selection. In addition to the keyboard 211 and the pointing device 212, an operation button (having a UI (User Interface) that can be pressed or tapped) or the like may be used as other input means. In this embodiment, the operation receiving unit 22 functions as an example of operation receiving means.

The input/output unit 23 mainly receives sound signals related to sounds (sounds) input/output by the microphone 218, the speaker 219, and the sound input/output I/F 217 shown in FIG. This is achieved by the CPU 201 processing an image signal related to an image (moving image, still image) input/output by the F221. The input/output unit 23 inputs a voice uttered by a user using the input device 2 or a sound produced by a machine. Further, the input/output unit 23 is mainly realized by the processing of the CPU 201 for the speaker 219 and the sound input/output I/F 217 shown in FIG. and outputs audio through the speaker 219 . In the present embodiment, the input/output unit 23 functions as an example of input/output means. Furthermore, in the present embodiment, the operation reception unit 22 and the input/output unit 23 described above function as an example of input means.

The display control unit 24 is mainly realized by processing of the display 206 shown in FIG. In the present embodiment, the display control section 24 functions as an example of display control means. In addition, in the present embodiment, various information displayed by the display control unit 24 may be displayed not only on the display 206 of the input device but also on a display unit or the like of another device that can communicate with the input device 2 .

The determination unit 25 is mainly realized by processing of the CPU 201 shown in the drawing, and performs various determinations in the input device 2 . In this embodiment, the determination unit 25 functions as an example of determination means.

The conversion generation unit 27 is mainly realized by the processing of the CPU 201 shown in the figure, converts sound information in natural language etc. input by the input/output unit 23 into text (character) information, and generates a question sentence. do. In this embodiment, the conversion generation unit 27 functions as an example of conversion generation means.

The memory reading unit 29 is mainly implemented by the processing of the CPU 201 on at least one of the ROM 202 and the HD 204 shown in FIG. Various data (or information) are read out from the . In the present embodiment, the memory reading unit 29 functions as an example of memory reading means.

<Functional Configuration of Information Processing Device>
As shown in FIG. 3 , the information processing device 3 has a transmission/reception section 31 , an acquisition section 32 , an analysis calculation section 33 , a determination section 35 , an answer generation section 37 and a memory/read section 39 . Each of these functional units operates according to a command from the CPU 301 according to the program for the information processing apparatus 3 developed in the RAM 303 from at least one of the ROM 302 and the HD 304 by one of the hardware resources shown in FIG. It is a function or means realized by The information processing device 3 also has a storage unit 3000 configured by at least one of the ROM 302 and the HD 304 shown in FIG. Further, an information processing program executed by the information processing device 3 is stored in the storage unit 3000 . Furthermore, a communication application for communicating with the input device 2 is stored and managed in the storage unit 3000 .

●Maintenance information management table●
FIG. 4A is a conceptual diagram showing an example of a maintenance information management table. A maintenance information management DB 3001 configured by a maintenance information management table as shown in FIG. 4A is constructed in the storage unit 3000 . In the maintenance information management table, each item of an event that has occurred and a coping method are associated with each event identification information, and stored and managed. Of these, the event identification information represents identification information for identifying an event that has occurred, and is given and managed as MID001, MID002, etc., for example.

The event that occurred is a pre-extracted event that occurred in the managed device to be maintained. Here, the managed device is a general term for devices and machines that require regular maintenance, such as MFP (Multifunction Peripheral/Product/Printer) and machine tools. Events that have occurred include, for example, "part A is damaged," "part C makes an abnormal noise," "part C is deformed," and "part D makes an abnormal noise." These events are obtained in advance from a call center or the like that manages the managed device by a maintenance person who maintains the managed device, and the maintenance information is managed based on the obtained information. This is the information entered into the table.

The countermeasure method is an extraction of what kind of countermeasure was taken in the managed device to be maintained. For example, "unit E was replaced", "mounting adjustment was made", "part C was replaced", "part D was replaced", etc. are given and managed.

●Component information management table●
FIG. 4B is a conceptual diagram showing an example of a component information management table. In the storage unit 3000, a parts information management DB 3002 configured by a parts information management table as shown in FIG. 4B is constructed. In this parts information management table, the first layer and the second layer are associated with each piece of configuration identification information and stored and managed. Among these, the configuration identification information represents identification information for identifying the target of the configuration for each hierarchy, such as the parts and units that make up the device to be managed. .

The first layer is the highest level when the parts, units, etc. that make up the device to be managed are represented by photographs, images in documents such as service manuals, and CAD data. given and managed. The second hierarchy is located in the hierarchy one level lower than the unit E of the first hierarchy, and is given and managed by, for example, "part A" and "part D". It should be noted that the components of the product configuration management table may not be up to the second level, but may be in the first level or lower, or may be in the third level or higher.

●Additional information management table●
FIG. 4C is a conceptual diagram showing an example of an additional information management table. In the storage unit 3000, an additional information management DB 3003 configured by an additional information management table as shown in FIG. 4C is constructed. In this additional information management table, event image information and event sound information are stored and managed in association with each event identification information.

Of these, the incident image information is information representing an image (at least one of a still image and a moving image) of the location where the event occurred or its surroundings, and is ".jpg", ".gif" transmitted by the input device 2 , ``.png'', ``.tif'', ``.bmp'', ``.mp4'', ``.avi'', ``.mov'', etc. .

Occurrence event audio information is information representing the location where the event occurred or the surrounding audio (sound), and is represented in a file format such as ".wav", ".mp3", ".dct" transmitted by the input device 2. Various audio (sound) data are used.

Note that the above-described maintenance information management table (maintenance information management DB 3001), parts information management table (parts information management DB 3002), and additional information management table (additional information management DB 3003) are stored in storage unit 3000, not as table data. The data may be data managed in respective predetermined areas.

<<Each functional configuration of the information processing device>>
Next, each functional configuration of the information processing device 3 will be described in detail. The transmission/reception unit 31 of the information processing device 3 shown in FIG. Various data (or information) are transmitted/received to/from the input device 2 via. The transmitting/receiving unit 31 also receives a question from the user in a natural sentence and inputs it to the information processing device 3 . Interfaces include chatbots and text input boxes on web browsers, but any form is acceptable. Further, the transmitting/receiving section 31 transmits to the input device 2 the image information that caused the event that occurred in addition to the answer to the question input by the user. The question from the user may include a question by voice output by a machine in addition to natural sentences such as text information. In this embodiment, the transmitting/receiving unit 31 functions as an example of transmitting means and receiving means.

The acquisition unit 32 is mainly implemented by the processing of the CPU 301 shown in FIG. 2, and acquires additional information for additional request information and question information relating to a predetermined question transmitted by the input device 2 . In the present embodiment, the acquisition unit 32 functions as an example of acquisition means.

The analysis calculation unit 33 is mainly realized by the processing of the CPU 301 shown in FIG. 2, analyzes the input question text, and extracts structured information. For example, the analysis calculation unit 33 analyzes specific expressions such as part names and unit names of the device to be managed and relationships (place-action) using a predicate term analyzer, deep learning, etc., and extracts structured information. It is possible. The analysis calculation unit 33 also functions as a query generation unit that generates queries. Here, a query is an instruction for performing various processes such as extracting and manipulating data from a predetermined database (DB). Query may also be translated as "query". In the present embodiment, the analysis calculation unit 33 functions as an example of analysis calculation means.

The determination unit 35 is mainly implemented by the processing of the CPU 301 shown in FIG. 2 and performs various determinations in the information processing device 3 . Further, the judgment unit 35 has a function of requesting multimodal information from the input device 2 when there are multiple answer candidates for a predetermined question, that is, judging that the contents of the answer candidates are ambiguous. In this embodiment, the determination unit 35 functions as an example of determination means.

The answer generator 37 is mainly realized by the processing of the CPU 301 shown in FIG. 2, and generates answer information corresponding to the content of the input question or additional request information necessary to narrow down the answers to the question. do. In this embodiment, the answer generator 37 functions as an example of generating means.

The memory reading unit 39 is mainly realized by the processing of the CPU 301 on at least one of the ROM 302 and the HD 304 shown in FIG. Various data (or information) are read out from the . In the present embodiment, the memory reading unit 39 functions as an example of memory reading means.

○Multimodal information○
Multimodal information used in this embodiment will be described. One form of multimodal used in this embodiment is a state in which information extracted from a natural language data group, image information, and audio information are derived by deep learning and associated with each other. That is, when considering information extracted from a natural language data group as text information, information other than the information extracted from the natural language data group, for example, at least one of image information and voice information is used. Become.

In addition, in this embodiment, a natural language data group including text information is treated as an example of natural language information, and a data group including part information, image information and audio information is treated as an example of non-linguistic information other than natural language information. The natural language data group is, for example, data relating to natural language owned by a manufacturer, which is a provider of a device to be managed maintained and managed by the question answering system 1 . A natural language data group as an example of natural language information includes design specifications, manuals, maintenance reports, and parts information. On the other hand, a data group as an example of non-linguistic information is a data group other than linguistic information, and includes a data group other than a natural language data group. Among data groups other than natural language data groups, for example, CAD data, 3D CAD data (product mechanism and configuration information), bill of materials, etc. are structured data, and CAD images, images in documents, etc. are unstructured data. be.

In the present embodiment, a user who uses the input device 2 or a person in charge of maintenance of the managed device is hereinafter referred to as a "user or maintenance person".

[Processing or operation of the embodiment]
Next, operations and processes in the question answering system according to this embodiment will be described with reference to FIGS. 5 to 9. FIG.

<Question input processing>
FIG. 5 is a sequence diagram showing an example of question answering processing. First, a user or a maintenance person uses the input device 2 to activate a communication application for communicating with the information processing device. After that, as shown in FIG. 5, the input/output unit 23 of the input device 2 cooperates with the operation reception unit 22 to ask a predetermined question for the managed device input by the user or maintenance personnel. Accept and input (step S11). At this time, the user or the maintenance staff utters, for example, "I want you to tell me how to deal with the abnormal noise △△△□□□" in order to find a way to deal with the event that occurred in the managed device. to input a question to the input device 2 . As a result, the input/output unit 23 of the input device 2 receives a specific question about the managed device from the user or the person in charge of maintenance. If the information input by the input/output unit 23 is sound information in a natural language or the like, the conversion generation unit 27 converts the sound information into text (character) information to generate a question sentence. Further, the operation reception unit 22 of the input device 2 may input a question to the input device 2 by text input by the user or maintenance staff. In this case, the operation reception unit 22 of the input device 2 can directly receive a specific question for the managed device by text input by the user or the person in charge of maintenance.

●Screen display example●
FIG. 6 is a screen display example of a question acceptance screen for accepting input of a question. As shown in FIG. 6, the display control unit 24 of the input device 2 causes the display 206 to display a question acceptance screen 2101 including the following contents when preparations for questions by the user or maintenance personnel are complete.

On the question reception screen 2101, a title of "question reception screen", a predetermined message requesting the user to input a question, a question content input column 2111, and a "send" button 2112 are displayed. The question content entry field 2111 is an entry field into which text information can be entered, for example, and displays the text information entered by the user or the person in charge of maintenance. After the text information is input, the text information input in the question content input field 2111 is transmitted to the information processing device 3 by operating (pressing, tapping, etc.) the "Send" button by the user or maintenance staff. and you can transition to another screen. As described above, the question acceptance screen 2101 may accept questions by voice (sound). In that case, the question reception screen 2101 is provided with a microphone button used as a microphone by the user or the maintenance staff, a stop button operated after the question is uttered, etc., and a UI (User Interface) is provided to receive each operation. may be made.

<Generation processing of response information and additional request information>
Returning to FIG. 5, the transmitter/receiver 21 of the input device 2 that has received the question input transmits a question request to the information processing device 3 (step S12). Thereby, the transmitting/receiving unit 31 of the information processing device 3 receives the question request transmitted by the input device 2 . At this time, the question request includes text information or the like representing the content of the question accepted in step S11.

The information processing device 3 that has received the question request performs a process of generating answer information and additional request information (step S13). Specifically, the answer generation unit 37 refers to various databases (DB) described later, and generates answer information corresponding to the content of the input question or additional request information necessary to narrow down the answers to the question. do.

<Details of generating response information and additional request information>
FIG. 7 is a flow chart showing an example of processing for generating answer information and additional information. As shown in FIG. 7, first, the acquisition unit 32 of the information processing device 3 acquires question information (text information representing question content, etc.) included in the received question request (step S13-1). Specifically, the question information is text information (for example, information such as text mail) including the content of "I want you to tell me how to deal with the abnormal noise △△△□□□".

Subsequently, the analysis calculation unit 33 converts the acquired question information into a vector and generates a query for the question (step S13-2).

After the conversion into vectors and the generation of the query for the question, the answer generator 37 generates a group of answer candidates for the predetermined question input by the user or maintenance staff (step S13-3). In generating the answer candidate groups, the analysis calculation unit 33 cooperates with the memory reading unit 39 based on the content of the acquired question information, and searches the maintenance information management DB 3001 using "abnormal noise" included in the occurred event as a search key. (see FIG. 4A), a combination of two pieces of information corresponding to "Part C" and "Corrective Action": "Mounting Adjustment" and "Part D" and "Corrective Action": "Replacement" can be obtained. read out.

Subsequently, the determination unit 35 determines whether or not the answer in the answer candidate group generated by the answer generation unit 37 is uniquely determined for the predetermined question (step S13-4). If the determination unit 35 determines that the answer generated in the answer candidate group is uniquely determined (step S13-4; YES), the answer generation unit 37 generates an answer to the question based on the content of the generated answer candidate group. Information (answer sentence) is generated and this flow is exited (step S13-5). In other words, in step S13-5, the response generation unit 37 associates the pre-registered predetermined event, the component information related to the component for which the response to the predetermined event was performed, and the additional information, and generates the response information (response text). ).

On the other hand, if the determination unit 35 determines that the answer generated in the answer candidate group is not uniquely determined (step S13-4; NO), the determination unit 35 determines whether or not the multimodal information has been acquired ( step S13-6). If it is determined that the multimodal information has been acquired (step S13-6; YES), the analysis calculation unit 33 converts the acquired multimodal information into a vector, and then performs narrowing down based on the degree of similarity. -5 (step S13-7). Specifically, the analysis calculation unit 33 calculates the degree of similarity between the event that occurred and the additional information, and narrows down.

Here, as an example of narrowing down by similarity, there is a method of calculating the similarity of a text-image pair. For example, it can be calculated by cosine similarity of vectors obtained from ViLBERT, UINTER, and the like. Both ViLBERT and UINTER are methods of inputting text and images at the same time, extracting the features of both, and expressing them as vectors. These methods are supposed to calculate the degree of similarity between text and image combinations by calculating the cosine similarity between vectors. Note that these methods are known methods that have already been published in papers and the like.

In the processing of step S13-6, if it is determined that the multimodal information has not been obtained (step S13-6; NO), the reply generation unit 37 generates additional request information and exits this flow (step S13-8). ). In the example of step S13-8, the answer generation unit 37 has generated two pieces of information as answer sources and has not acquired multimodal information. Therefore, the answer generation unit 37 executes the process of step S13-8 described above and exits this flow. The content of the additional request information generated at this time is, for example, "Are there any detailed images or sounds in the vicinity where the abnormal noise is occurring?" The content is to encourage the provision of at least one of image information and audio information that will serve as a reference for answering a predetermined question.

<Display processing of response information and additional request information>
Returning to FIG. 7, the transmitting/receiving unit 31 supplies the input device 2 with answer information indicating an answer to a predetermined question, or additional request information indicating an additional request associated with a predetermined question in order to generate answer information. to send. Specifically, the transmitter/receiver 31 transmits the question and answer generated by the answer generator 37 to the input device 2 (step S14). Thereby, the transmitting/receiving section 21 of the input device 2 receives the question and answer transmitted by the information processing device 3 . At this time, the question answer includes an answer sentence as answer information to the question, or additional request information for requesting input of additional content necessary to narrow down the answer to the question. That is, the information processing device 3 transmits addition request information to the input device 2 when an answer to a predetermined question cannot be generated. A screen displayed on the input device 2 when the addition request information is transmitted will be described in detail in a screen display example described later.

After receiving the question and answer, the storage/reading unit 29 of the input device 2 stores the received question and answer in a predetermined area of the storage unit 2000 (step S15).

After that, the display control unit 24 reads out the answer information indicating the answer stored in cooperation with the memory reading unit 29 or the additional request information from the storage unit 2000, and displays it on the display 206 (step S16).

The input device 2 with the question and answer displayed on the display 206 inputs information satisfying the additional request information (step S17). However, in step S16, the user or maintenance personnel confirms the question and answer, and if the question and answer is satisfactory, the process of step S17 may not be performed. The process of step S17 will be described in detail with a screen display example to be described later.

After the information satisfying the additional request information is input in step S17, the determination unit 25 determines whether or not the "send" button has been operated (step S18). If it is determined that the "send" button has been operated (step S18; YES), the determination unit 25 proceeds to the process of step S12, and the transmission/reception unit 21 transmits a new question request to the information processing device 3. and repeat the process until a unique answer is determined. That is, the transmission/reception unit 31 of the information processing device 3 receives a new question request from the input device 2 . The new question request received at this time is question information indicated by at least one of the image information and the voice information newly input for the additional request information in step S17. That is, in the process of step S12 from the second time onward, the transmitting/receiving unit 31 converts at least one of the image information and the audio information included in the predetermined event related to the predetermined question transmitted by the input device 2 to the additional request information. receive as additional information to

On the other hand, if it is determined that the "send" button has not been operated (step S18; NO), the determining section 25 terminates this sequence process. At this time, the determination unit 25 may repeat the determination process of step S18 for a predetermined period of time or a predetermined number of times, wait for an operation by the user or the person in charge of maintenance, and then exit this sequence.

In the question answering system according to this embodiment, for example, when the processes of steps S12 and S14 described above are executed, another device or the like may exist between the input device 2 and the information processing device 3 . That is, each piece of information (data) transmitted/received between the input device 2 and the information processing device 3 may be transmitted/received once via another device. The configuration and processing method described above are also applicable to other processing steps between the input device 2 and the information processing device 3 .

It should be noted that the above-described sequence diagrams and flowcharts are merely examples for describing one embodiment. Therefore, the present invention is not limited to the above-described embodiment, and can be modified within the range that a person skilled in the art can conceive of, such as processing using other sequence diagrams and flowcharts.

●Screen display example●
FIG. 8 is a screen display example of a response notification screen displaying the contents of the response notification. In FIG. 8, when desired answer information is uniquely determined in response to a predetermined question sent by a user or maintenance staff, a screen related to image information that caused an event that occurred is displayed on the input device 2. is displayed by the display control unit 24 of . As shown in FIG. 8, when receiving the question and answer transmitted by the information processing device 3, the display control unit 24 causes the display 206 to display a response notification screen 2102 including the following contents.

The response notification screen 2102 displays a title of "response notification screen" and a response message addressed to the user or maintenance personnel as an answer to the question. The question and answer generated by the answer generator 37 at this time includes, for example, a sentence explaining where the event occurred, details of the event that occurred, how to deal with it, and image information 2113 of the part dealt with. is included. The input device 2 receives the answer information included in the above question and answer from the information processing device 3 and displays it on the display 206 . As a result, the information processing device 3 can present the details of the answer information to be answered in response to the content of the question to the user or the person in charge of maintenance via the input device 2 . In other words, the information processing device 3 can respond including specific visual information such as image information when a predetermined question is input by the user or maintenance personnel. Therefore, the question answering system 1 can also improve the efficiency of repair work and the like for users or maintenance personnel.

The response notification screen 2102 also includes a “confirm” button 2114 . As a result, the user or maintenance personnel can switch to another screen by operating (pressing, tapping, etc.) the "confirm" button 2114 after confirming the contents displayed on the response notification screen 2102. can.

Note that the response notification screen 2102 shown in FIG. It is also possible to display a screen obtained by retrieving some information. In other words, screen information obtained using a browser or the like may be displayed on the display 206 instead of the mail reception screen.

●Screen display example●
FIG. 9 is a screen display example of an additional information acceptance screen for accepting input of additional information. FIG. 9 shows an example of a screen displayed on the input device 2 when desired answer information is not obtained in response to a predetermined question sent by the user or the person in charge of maintenance. As shown in FIG. 9, when receiving the question and answer transmitted by the information processing device 3, the display control unit 24 causes the display 206 to display an additional information reception screen 2103 including the following contents.

On the additional information reception screen 2103, a title of "additional information reception screen", a message requesting provision of additional information, a message as <additional question>, an "image" selection button 2115, a "voice" selection button 2116, an image A display area 2117, a microphone button 2118, a stop button 2119, a "send" button 2120 and a "cancel" button 2121 are displayed respectively. That is, the display control unit 24 displays on the display 206 at least one of the image information receiving means for providing image information and the audio information receiving means for providing audio information.

An "image" selection button 2115 is a button for inputting image information, which is one type of additional information. By operating the “image” selection button 2115 , the CPU 201 of the input device 2 may activate the camera 220 and the image sensor I/F 221 . As a result, the user or the person in charge of maintenance can photograph the vicinity of the location where the event has occurred and temporarily store it as image data in a predetermined area of the input device 2 as desired image information. Furthermore, by operating the “image” selection button 2115 , the CPU 201 may control the display 206 to display the image display area 2117 as a pop-up screen on the additional information reception screen 2103 . Thereby, the user or the person to be managed can temporarily store desired image data in a predetermined area of the input device 2 . With such a method, the user or the person in charge of maintenance can transmit (upload) desired image information to the information processing apparatus 3 .

A "voice" selection button 2116 is a button for inputting voice information, which is another type of additional information. By operating the "voice" selection button 2116, the CPU 201 of the input device 2 controls the microphone 218 to turn on the "microphone" button 2118. FIG. Therefore, the user or maintenance personnel can use this "microphone" button 2118 as a microphone. At this stage, the user or the person in charge of maintenance speaks desired voice (sound) information into the "microphone" button 2118, and after speaking, operates the "stop" button 2119 to set the input device 2 to the predetermined voice (sound) information. The content of the utterance is temporarily stored as voice data in the area. As a result, the user or the person in charge of maintenance can transmit (upload) desired audio information to the information processing device 3 .

After going through the above steps, the user or the person in charge of maintenance operates the "Send" button 2120 to send at least one of the image information and the sound information to the information processing device 3, and another screen is displayed. can transition to

With the configuration described above, the information processing device 3 can obtain detailed information associated with the content of a question related to a predetermined event from the user or the person in charge of maintenance. That is, the information processing device 3 can resolve the ambiguity of the question provided by the user who uses the input device 2 or the person in charge of maintenance at an early stage. In other words, when a predetermined question is input by the user or the maintenance staff, the information processing device 3 eliminates the possibility of giving an ambiguous response to the user or the maintenance staff as much as possible, It is possible to quickly provide a response requested by a user or a maintenance person. Therefore, the question answering system 1 can ensure the accuracy of deep learning for users or maintenance personnel.

Furthermore, in this embodiment, for example, a scene is assumed in which a new customer engineer (hereinafter referred to as CE) visits a customer. In such a situation, the CE is identifying the cause of failure, which is one of the phenomena occurring in the image forming apparatus such as the MFP installed at the customer's site. Therefore, when the CE asks the question-answering system 1 about the cause of the failure, the question-answering system 1 answers while resolving the ambiguity of the question. In such cases, information such as what kind of abnormal noise is occurring due to the failure and where the abnormal noise is occurring is important in identifying the problem, but in some cases it is difficult to verbalize it. many. In this embodiment, by using the multimodal information as it is in such a case, it is possible to efficiently resolve the ambiguity of the question.

In addition, in this embodiment, a user or a person in charge of maintenance is collectively called a user. In addition to maintenance personnel, users include service personnel who manage various services provided by managed devices, and repair personnel who repair managed devices.

[Main effects of the embodiment]
As described above, according to the present embodiment, the transmitting/receiving unit 31 of the information processing device 3 transmits the question and answer generated by the answer generating unit 37 to the input device 2 (step S14), and At least one of image information and audio information newly input in response to the additional request information received is received from the input device 2 (step S12). This has the effect of resolving the ambiguity of the additionally input question when the answer to the predetermined question cannot be answered.

[Supplement to the embodiment]
The functions of the above-described embodiment can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language. Programs for performing the functions of the embodiments can be distributed over telecommunications lines.

In addition, the program for executing the functions of one embodiment includes ROM, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), flash memory, flexible disk, CD (Compact Disc)- It can also be stored and distributed in recording media such as ROM, CD-RW (Re-Writable), DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO (Magneto-Optical disc) and the like.

Furthermore, some or all of the functions of one embodiment are, for example, ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (Field Programmable Gate Array), SOC (System on a chip), GPU (Graphics Processing Unit) can be implemented on a programmable device (PD) such as, and generates circuit configuration data (bitstream data) and circuit configuration data to be downloaded to the PD in order to realize the functions of each embodiment on the PD HDL (Hardware Description Language), VHDL (Very High Speed Integrated Circuits Hardware Description Language), Verilog-HDL, etc., for the purpose of distributing data on a recording medium.

Furthermore, the texts and various tables obtained by the above-described embodiments may be generated by the learning effect of machine learning. You don't have to. Here, machine learning is a technology for making a computer acquire human-like learning ability, and the computer autonomously generates algorithms necessary for judgment such as data identification from learning data taken in advance, It is a technology that applies this to new data and makes predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning. A combined learning method may be used, and any learning method for machine learning may be used.

Furthermore, the input device exemplified in one embodiment may have each function and means of an information processing device and function as an input response device. In that case, the input device 2 used in this embodiment includes each functional unit of the information processing device 3 .

A question answering system, an information processing apparatus, an information processing method, and a program according to one embodiment of the present invention have been described so far. Additions, changes, deletions, etc., can be made within the range that a person skilled in the art can conceive, and as long as the action and effect of the present invention are exhibited in any aspect, it is included in the scope of the present invention.

1 question answering system 2 input device 3 information processing device 21 transmission/reception unit (an example of receiving means, an example of transmitting means)
22 Operation reception unit (an example of operation reception means, an example of input means)
2114 Image button 2114 (an example of image information receiving means)
2115 Voice button 2115 (an example of voice information receiving means)
23 input/output unit (an example of input/output means)
31 Transceiver (an example of receiving means, an example of transmitting means)
32 Acquisition unit (an example of acquisition means)
33 analysis calculation unit (an example of analysis calculation means)
35 Determination unit (an example of determination means)
37 Answer generation unit (an example of generation means)

JP 2019-117568 A

Claims (10)

A question answering system comprising an input device for inputting a predetermined question and an information processing device for responding to an answer to the predetermined question transmitted by the input device,
The information processing device is
receiving means for receiving question information representing the predetermined question transmitted by the input device;
sending means for sending answer information representing an answer to the predetermined question or additional request information representing an additional request associated with the predetermined question to generate the answer information to the input device;
has
The input device is
display control means for displaying the addition request information transmitted by the information processing device on a display means;
transmitting means for transmitting at least one of image information and audio information input in response to the addition request information to the information processing apparatus;
having
A question answering system characterized by: The transmission means of the information processing device includes:
sending the additional request information to the input device when an answer to the predetermined question cannot be generated;
The question answering system according to claim 1, characterized by: The question answering system according to claim 1 or 2,
The receiving means
receiving at least one of image information and audio information included in a predetermined event related to the predetermined question transmitted by the input device as additional information to the additional request information;
A question answering system characterized by: The information processing device further includes
generating means for generating the response information by associating the pre-registered predetermined event, part information related to the part that has dealt with the predetermined event, and the additional information;
4. The question answering system according to claim 3, characterized by: The question answering system according to claim 4,
The information processing device further includes
A calculation means for calculating a similarity between the event and the additional information;
A question answering system characterized by: The display control means is
displaying on the display means at least one of image information receiving means for providing the image information and audio information receiving means for providing the audio information;
6. The question answering system according to any one of claims 1 to 5, characterized in that: The display control means is
displaying image information that caused the event on the display means when an answer to the predetermined question is uniquely determined;
6. The question answering system according to any one of claims 1 to 5, characterized in that: An information processing device that responds to an answer to the predetermined question transmitted by an input device for inputting a predetermined question,
receiving means for receiving question information representing the predetermined question transmitted by the input device;
sending means for sending answer information representing an answer to the predetermined question or additional request information representing an additional request associated with the predetermined question to generate the answer information to the input device;
has
The receiving means
receiving at least one of image information and audio information input in response to the addition request information;
The transmission means is
transmitting to the input device answer information representing an answer to the predetermined question generated based on at least one of the image information and the audio information received by the receiving means;
An information processing device characterized by: An information processing method executed by an information processing device that responds to an answer to a predetermined question transmitted by an input device for inputting a predetermined question,
a receiving step of receiving question information representing the predetermined question transmitted by the input device;
a sending step of sending answer information representing an answer to the predetermined question or additional request information representing an additional request associated with the predetermined question to generate the answer information to the input device;
has
The receiving step includes
including a process of receiving at least one of image information and audio information input in response to the addition request information;
The sending step includes:
A process of transmitting, to the input device, answer information representing an answer to the predetermined question generated based on at least one of the image information and the audio information received in the receiving step,
An information processing method characterized by: A program for causing a computer to function as each means according to claim 8.

Images