JP7230506B2 - Information processing system, information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing system, an information processing device, an information processing method, and an information processing program.

Today, AI voice assistants (AI: Artificial Intelligence) that operate devices by voice are known. Smart home appliances that can be operated by voice are also known, and the field of voice operation is expected to grow further in the future.

Here, in the case of image forming apparatuses such as MFPs (Multifunction Peripherals), which are widely used today, setting operations, printing execution operations, and the like are performed manually based on a GUI (Graphical User Interface). there is

However, with conventional image forming apparatuses, even a user who is accustomed to operating the apparatus needs to make detailed settings every time through the operation unit. In addition, users who are unfamiliar with the operation need to learn the operation method by referring to the manual. A conventional image forming apparatus has a problem in such operability.

The present invention has been made in view of the above-mentioned problems, and provides an information processing system, information processing apparatus, information processing method, and information processing program that enable quick and easy input without the need to operate an operation unit. aim.

In order to solve the above-described problems and achieve the object, the present invention provides an information processing system including an external device and an information processing device, wherein the information processing device acquires voice information for operating the external device. a voice recognition unit for recognizing voice information; an extraction unit for extracting instruction information indicating information processing to instruct an external device based on the recognition result of the voice information by the voice recognition unit; Associating a conversion unit that converts into operation information in an information format interpretable by an external device, first identification information that identifies an information processing device, and second identification information that identifies an external device associated with the information processing device and an output unit that refers to the associated information and outputs the operation information to the external device associated by the associated information with respect to the information processing apparatus that has acquired the voice information .

ADVANTAGE OF THE INVENTION According to this invention, there exists an effect that the operation by an operation part can be made unnecessary and quick and easy input can be performed.

FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. FIG. 2 is a hardware configuration diagram of the MFP provided in the voice operation system according to the first embodiment. FIG. 3 is a hardware configuration diagram of a mobile terminal provided in the voice operation system according to the first embodiment. FIG. 4 is a hardware configuration diagram of a voice recognition server device provided in the voice operation system of the first embodiment. FIG. 5 is a hardware configuration diagram of an AI assistant server device provided in the voice operation system according to the first embodiment. FIG. 6 is a functional block diagram of a mobile terminal provided in the voice operation system according to the first embodiment. FIG. 7 is a functional block diagram of a voice recognition server device provided in the voice operation system according to the first embodiment. FIG. 8 is a functional block diagram of an AI assistant server device provided in the voice operation system according to the first embodiment. FIG. 9 is a sequence diagram showing the overall operation flow of voice operation in the voice operation system according to the first embodiment. FIG. 10 is a diagram showing an example of entity information used for interpreting a user's input voice in the voice operation system according to the first embodiment. FIG. 11 is a diagram showing entity information registered based on an uttered phrase in the voice operation system according to the first embodiment. FIG. 12 is a diagram showing the flow of interactive input operations in the voice operation system according to the first embodiment. FIG. 13 is a sequence diagram showing the flow of the first half of the interactive input operation in the voice operation system according to the first embodiment. FIG. 14 is a sequence diagram showing the second half of the interactive input operation in the voice operation system according to the first embodiment. FIG. 15 is a diagram showing the flow of interactive input operations when an error occurs in the voice operation system according to the first embodiment. FIG. 16 is a sequence diagram showing the flow of interactive input operations when an error occurs in the voice operation system according to the first embodiment. FIG. 17 is a sequence diagram showing the flow of the first half of the operation of executing a scan according to the device capability of the MFP by an interactive input operation in the voice operation system according to the first embodiment. FIG. 18 is a sequence diagram showing the flow of the second half of the operation of executing a scan according to the equipment capability of the MFP by an interactive input operation in the voice operation system according to the first embodiment. FIG. 19 is a sequence diagram showing the first half of the operation of transmitting a scanned image to a user specified from an address book by an interactive input operation in the voice operation system according to the first embodiment. FIG. 20 is a sequence diagram showing the latter half of the operation of transmitting a scanned image to a user designated from an address book by an interactive input operation in the voice operation system according to the first embodiment. FIG. 21 is a sequence diagram showing the flow of the first half of the operation of printing a file specified from memory by an interactive input operation in the voice operation system according to the first embodiment. FIG. 22 is a sequence diagram showing the latter half of the operation of printing a file specified from memory by an interactive input operation in the voice operation system according to the first embodiment. FIG. 23 is a system configuration diagram of the voice operation system of the second embodiment. FIG. 24 is a diagram showing each function realized by the CPU of the smart speaker executing the operation voice processing program. FIG. 25 is a diagram for explaining the overall voice input operation in the voice operation system according to the second embodiment. FIG. 26 is a sequence diagram showing the flow of activation operation of the voice assistant program. FIG. 27 is a sequence diagram showing the flow of operations when instructing the copy function. FIG. 28 is a sequence diagram showing the flow of the number of copies setting operation. FIG. 29 is a sequence diagram showing the flow of the job execution instruction operation to the MFP. FIG. 30 is a sequence diagram showing the flow of such job execution operations and interactive operations when an error occurs. FIG. 31 is a sequence diagram showing the flow of operations until it is determined whether or not the MFP has the device capability required for the job instructed by the user. FIG. 32 is a sequence diagram showing the flow of operations when the job can be executed with the device capabilities of the MFP and when it is difficult to execute the job. FIG. 33 is a sequence diagram showing operations up to searching for a file specified by the user. FIG. 34 is a sequence diagram showing operations up to printing a file specified by the user.

An information processing system, an information processing apparatus, an information processing method, and a voice operation system as application examples of an information processing program will be described below.

(First embodiment)
(System configuration)
FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. As shown in FIG. 1, the voice operation system according to the first embodiment includes a multifunction peripheral (MFP) 1 as an example of an external device, a mobile terminal 2 (information processing device) such as a smartphone or a tablet terminal, and the like. example), the voice recognition server device 3 and the AI assistant server device 4 are formed by connecting them to each other via a predetermined network 5 such as a LAN (Local Area Network). However, the external device is not limited to the multifunction machine, and may be various electronic devices including office equipment such as an electronic blackboard and a projector.

Portable terminal 2 accepts voice input from the user for operating MFP 1 by voice. Also, the received operation is fed back to the user by voice. The mobile terminal 2 also relays data communication (communication of text data to be described later) between the voice recognition server device 3 and the AI assistant server device 4 . The speech recognition server device 3 analyzes the speech data received from the mobile terminal 2 and converts it into text data. Note that the voice recognition server device 3 corresponds to a first server device. The AI assistant server device 4 analyzes the text data, converts it into the pre-registered user's intention (job execution command of the MFP 1 ), and transmits it to the portable terminal 2 .

Note that the AI assistant server device 4 corresponds to a second server device. MFP 1 executes a job execution command transmitted from portable terminal 2 . Communication between the mobile terminal 2 and the MFP 1 may be wireless communication or wired communication. In other words, the mobile terminal 2 may be an operation terminal fixedly connected to the MFP 1 .

Also, in this example, two server devices, the voice recognition server device 3 and the AI assistant server device 4, are provided, but each of the server devices 3 and 4 may be physically one server device. . Alternatively, each of the server devices 3 and 4 may be further implemented by a plurality of server devices.

(MFP hardware configuration)
The MFP 1 has multiple functions such as a printer function and a scanner function. That is, the MFP 1 has a controller 7, a communication section 15, an operation section 16, a scanner engine 17 and a printer engine 18, as shown in FIG.

The controller 7 has a CPU 10 , an ASIC (Application Specific Integrated Circuit) 11 , a memory 12 , a HDD (Hard Disk Drive) 13 and a timer 14 . The CPU 10 to timer 14 are interconnected so as to be communicable via a bus line.

The communication unit 15 is connected to the network 5, and acquires a job execution command input by voice using the mobile terminal 2, such as a scan instruction or a print instruction, as will be described later.

The operation unit 16 is a so-called touch panel in which a liquid crystal display (LCD) and a touch sensor are integrally formed. When the operator uses the operation unit 16 to issue a command to execute a desired operation, the operator touches an operation button (software key) displayed on the operation unit 16 to specify the desired operation.

The scanner engine 17 controls the scanner unit to optically read the document. The printer engine 18 controls the image writing unit to print an image on transfer paper or the like. The CPU 10 comprehensively controls the image forming apparatus. The ASIC 11 is a so-called large-scale integrated circuit (LSI: Large-Scale Integration), and performs various image processing necessary for images processed by the scanner engine 17 and the printer engine 18 . The scanner engine 17 and printer engine 18, which are means for executing a job execution command acquired from the mobile terminal device 2, correspond to functional units.

The memory 12 stores various applications executed by the CPU 10 and various data used when executing the applications. The HDD 13 stores image data, various programs, font data, various files, and the like. An SSD (Solid State Drive) may be provided instead of the HDD 13 or together with the HDD 13 .

(Hardware configuration of mobile terminal)
The portable terminal 2 is formed by interconnecting a CPU 21, a RAM 22, a ROM 23, an interface section (I/F section) 24, and a communication section 25 via a bus line 26, as shown in FIG. The RAM 22 stores an address book that stores e-mail addresses of users to whom e-mails, scanned images, etc. are sent. The RAM 22 also stores files and the like that are image data to be printed.

The ROM 23 stores an operation voice processing program. The CPU 21 enables the voice input operation of the MFP 1 by executing this operation voice processing program.

A touch panel 27 , a speaker section 28 and a microphone section 29 are connected to the I/F section 24 . The microphone unit 29 collects (obtains) the input voice of a job execution command to the MFP 1 in addition to the call voice. The input speech is transmitted to the speech recognition server device 3 via the communication unit 25 and converted into text data.

(Hardware configuration of speech recognition server device)
As shown in FIG. 4, the speech recognition server device 3 connects a CPU 31, a RAM 32, a ROM 33, a HDD (Hard Disk Drive) 34, an interface section (I/F section) 35, and a communication section 36 to each other via a bus line 37. formed by connecting to A display unit 38 and an operation unit 39 are connected to the I/F unit 35 . The HDD 34 stores an operation voice conversion program for converting voice data into text data. By executing the operation voice conversion program, the CPU 31 converts the voice data transmitted from the mobile terminal 2 into text data and sends back to the mobile terminal 2 .

(Hardware configuration of AI assistant server device)
The AI assistant server device 4 is formed by interconnecting a CPU 41, a RAM 42, a ROM 43, an HDD 44, an interface section (I/F section) 45, and a communication section 46 via a bus line 47, as shown in FIG. ing. A display unit 48 and an operation unit 49 are connected to the I/F unit 45 . The HDD 44 stores an operation interpretation program for interpreting a job designated by the user. By executing this operation interpretation program, the CPU 41 interprets the job instructed by the user from the text data generated (converted) by the speech recognition server device 3 . This interpretation result is transmitted to the mobile terminal 2 . The mobile terminal converts the interpreted result into a job command and supplies it to the MFP1. As a result, the MFP 1 can be operated by voice input via the mobile terminal 2 .

(mobile terminal functions)
Next, the CPU 21 of the portable terminal 2 executes the operation voice processing program stored in the ROM 23 to obtain the acquisition unit 51, the communication control unit 52, the voice feedback unit 55, and the processing capacity acquisition unit as shown in FIG. 56 , an execution determination unit 57 and a search unit 58 .

Acquisition unit 51 is an example of an acquisition unit and a first acquisition unit, and acquires a user instruction voice for voice operation of MFP 1 collected via microphone unit 29 . The communication control unit 52 is an example of an output unit, a first output unit, a second output unit, a third output unit, a second acquisition unit, and a third acquisition unit. and the speech recognition server device 3, and between the mobile terminal 2 and the AI assistant server device 4. Interpretation result conversion unit 53 is an example of a conversion unit, and converts the interpretation result of the text data of the instruction voice of the user in AI assistant server device 4 into a job execution command for MFP 1 . Execution instructing unit 54 transmits a job execution instruction to MFP 1 to instruct execution of the job.

The voice feedback unit 55 is an example of a correction unit, and in order to realize an interactive voice input operation, for example, feeds back voice prompting input to compensate for missing data, or performs voice feedback for confirming input. . The processing capacity acquisition unit 56 is an example of a processing capacity detection unit, and acquires the processing capacity such as the maximum number of pixels that can be processed from the MFP 1 . Execution determination unit 57 determines whether or not MFP 1 can execute the job specified by the user by comparing the capability of MFP 1 with the job specified by the user. The search unit 58 is an example of a print information detection unit and an address detection unit, and searches a memory such as the RAM 22 for a destination, a file, or the like designated by voice from the user.

In this example, the acquiring unit 51 to the searching unit 58 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 51 to the search unit 58 may be realized by the operation voice processing program alone, may be executed by another program to execute a part of the processing, or may be indirectly obtained by using another program. Processing may be executed. For example, acquisition of information such as the processing capacity of MFP 1 is executed by another program, and the processing capacity acquiring unit 56 acquires the information acquired by the other program, thereby indirectly acquiring information possessed by MFP 1 . can.

(Function of voice recognition server device)
Next, the CPU 31 of the voice recognition server device 3 executes the operation voice conversion program stored in the HDD 34, thereby functioning as an acquisition unit 61, a text conversion unit 62, and a communication control unit 63 as shown in FIG. . The acquisition unit 61 acquires voice data input by the user, which is transmitted from the mobile terminal 2 . The text conversion unit 62 is an example of a speech recognition unit, and converts voice data input by the user into text data. The communication control unit 63 controls communication of the communication unit 36 so as to receive voice data input by the user and transmit text data to the mobile terminal 2 .

In this example, the acquisition unit 61 to the communication control unit 63 are realized by software, but some or all of them may be realized by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 61 to the communication control unit 63 may be realized by the operation voice conversion program alone, by causing another program to execute part of the processing, or indirectly by using another program. You can let the .

(Function of AI assistant server device)
Next, the CPU 41 of the AI assistant server device 4 functions as an acquisition unit 71, an interpretation unit 72, and a communication control unit 73 as shown in FIG. 8 by executing the operation interpretation program stored in the HDD 44. FIG. The acquisition unit 71 acquires text data of voice data input by the user, which is transmitted from the mobile terminal 2 . The interpretation unit 72 is an example of an extraction unit, and interprets an operation instruction from the user based on text data. The communication control unit 73 controls communication of the communication unit 46 so as to transmit interpretation results to the mobile terminal 2 of the user and receive text data of voice data input by the user.

In this example, the acquisition unit 71 to the communication control unit 73 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 71 to the communication control unit 73 may be realized by the operation interpretation program alone, or may be executed indirectly by using another program to execute a part of the processing by another program. Processing may be executed.

In addition, the operation voice processing program, the operation voice conversion program and the operation interpretation program are recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) as files in an installable format or an executable format. may be provided as Alternatively, it may be provided by being recorded on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), or a semiconductor memory. Moreover, it may be provided in a form of being installed via a network such as the Internet, or may be provided by being incorporated in advance in a ROM or the like in the device.

(Overall operation of voice input operation)
FIG. 9 illustrates the overall voice input operation in the voice operation system of the first embodiment. The example of FIG. 9 is an example in which the double-sided copy function of the MFP 1 is operated by voice input via the portable terminal 2. FIG. In this case, the user activates the operation voice processing program of the portable terminal and utters, for example, "copy on both sides". This user's voice is collected by the microphone unit 29 of the mobile terminal 2 and acquired by the acquisition unit 51 (step S1). The communication control unit 52 of the portable terminal 2 transmits the voice data of "copy on both sides" to the voice recognition server device 3, and controls the communication unit 25 to request text conversion (step S2).

The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "copy on both sides" into text data. Then, the communication control section 63 controls the communication section 36 to transmit the converted text data to the portable terminal 2 (step S3). The communication control unit 52 of the mobile terminal 2 transmits the text data "copy on both sides" to the AI assistant server device 4 (step S4).

In this example, the interpreting unit 72 of the AI assistant server device 4 interprets the action requested to the MFP 1 to be "copy (Action: Copy_Execute)", and interprets that "the printing surface is double-sided (printing surface = double-sided )” (step S5). In this manner, the interpretation unit 72 generates an interpretation result indicating the type (action) and content (parameters) of the job specified by the user, based on the text data. This interpretation result is transmitted to the portable terminal 2 via the communication section 46 by the communication control section 63 of the AI assistant server device 4 (step S6).

Interpretation result conversion unit 53 of mobile terminal 2 converts the interpretation result received from AI assistant server device 4 into a job command for MFP 1 (step S7). Table 1 below shows an example of an interpretation result and a job command converted from the interpretation result. In order to convert the interpretation result into a job command, the interpretation result conversion unit 53 may store information corresponding to Table 1 in the storage unit of the portable terminal device 2 and refer to it.

In the example of Table 1, "COPY_EXECUTE", "SCAN_EXECUTE", "PRINT_EXECUTE" and "FAX_EXECUTE" are shown as examples of actions. Also, "printing surface", "number of copies", etc. are shown as examples of parameters. The parameters include all parameters that can be specified as job setting values.

The interpretation result conversion unit 53 of the portable terminal 2 converts the interpretation result of "COPY_EXECUTE" into a job command of the MFP 1 "execute copy job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "SCAN_EXECUTE" into a job command of MFP 1, "execute scan job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "PRINT_EXECUTE" into a job command of MFP 1 "execute print job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "FAX_EXECUTE" into a job command of MFP 1, "execute FAX job".

Further, when the interpretation result includes the parameter of "printing surface", the interpretation result conversion unit 53 of the portable terminal 2 forms a job command for the MFP 1 to "change the setting value of the printing surface". Similarly, when the interpretation result includes a parameter of "number of copies", the interpretation result conversion unit 53 forms a job command for the MFP 1 to "change the set value of the number of copies".

That is, the interpretation result conversion unit 53 of the portable terminal 2 determines the type of job to be executed by the MFP 1 based on the information included in the "Action" of the interpretation result, and uses the value included in the "Parameter" as the setting value of the job. It judges and converts the result of interpretation into a job instruction.

The communication control unit 52 of the portable terminal 2 controls the communication unit 25 so as to transmit the job command thus formed to the MFP 1 (step S8). In the case of this example, a job command of "execute copy job (print side=both sides)" is transmitted to MFP1. As a result, MFP 1 executes double-sided printing.

(Details of the interpretation operation in the AI assistant server device)
The AI storage unit 40 of the HDD 44 of the AI assistant server device 4 stores AI assistant service information for interpreting a job instructed by a user's voice input. This AI assistant service information includes entity information, action information, and intent information. Entity information is information that associates job parameters with natural language. Multiple synonyms can be registered for one parameter. The action information is information indicating the type of job. The intent information is information that associates a user's uttered phrase (natural language) with entity information, and a user's uttered phrase (natural language) with action information, respectively. The intent information enables correct interpretation even if the utterance order or nuance of the parameters changes slightly. Also, the intent information makes it possible to generate a response text (interpretation result) based on the input content.

FIG. 10 shows an example of entity information. FIG. 10 shows entity information corresponding to Print Color. In FIG. 10, the characters "Print Color" indicate the entity name. In FIG. 10, characters such as “auto_color”, “monochrome”, “color”, etc. in the left column indicate parameter names. In FIG. 10, characters such as “auto_color”, “monochrome, black and white”, “color, full color”, etc. in the right column indicate synonyms.

As can be seen from FIG. 10, as entity information, parameters and synonyms are associated and stored. By registering synonyms along with parameters, for example, when instructing to copy in monochrome, even if you say "Please copy by black and white" or "Please copy by monochrome", you can set parameters can be made possible.

FIG. 11(a) shows an example of a user's uttered phrase, FIG. 11(b) shows an action name, and FIG. 11(c) shows entity information. As shown in FIGS. 11A to 11C, the user's utterance is dragged by operating the operation unit 49 on the screen displayed on the display unit 48 provided in the AI assistant server device 4. . Alternatively, the user's utterance is dragged by operating the operation unit of the device on the screen displayed on the display unit of the device that accessed the AI assistant server device 4 via the network.

This makes it possible to select entity information to be associated. Also, if a value (VALUE) is set in the selected entity information, the parameters entered in the response are changed. For example, when "Please copy by black and white" is uttered and the value is "SprintColor", "printColor=monochrome" is returned as the return value. On the other hand, if the value is "SprintColor.original", "printColor=black and white" is returned as the return value. Here, if the value is "SprintColor.original", the content of the user's utterance can be returned as it is as a parameter of the response.

(interactive behavior)
Next, the voice operation system of the first embodiment realizes an interactive system in which the system responds based on the contents of input from the user. In the voice operation system of the first embodiment, in addition to responding with fixed sentences necessary for dialogue, two types of responses, "input insufficient feedback" and "input confirmation feedback", are provided as responses specific to the operation of the MFP 1. By doing so, an interactive MFP operation system is realized.

"Insufficient input feedback" is a response that is output when the information required to execute a job is not complete. This is output when the user's input result cannot be heard, or when a required parameter is missing. In other words, there is no need to provide insufficient input feedback for parameters other than essential parameters, even if they are not indicated. In addition to the parameters, processing for confirming a function to be used, such as a copy function or a scan function, may also be included.

For example, the functions and parameters to be confirmed by the user may be changed according to the type of external device with which the mobile terminal device 2 is in communication connection. In this case, the processing capacity acquisition unit 56 acquires information indicating the type and function of the external device at a predetermined timing after communication with the external device is established, and based on the acquired information, determines the functions and parameters to be confirmed by the user. For example, the audio feedback unit 55 may determine. For example, when the type of the external device is MFP1, the user can confirm the functions of copy, print, scan, FAX, etc. that MPF1 has. As for the functions, the user may be asked which function to use.

"Input confirmation feedback" is a response that is output when the information necessary for executing a job is complete. That is, input confirmation feedback is provided when indicated for all required parameters. Also, the input confirmation feedback is provided to prompt the user to select whether to execute the job with the current set values or to change the set values. In addition, in order to confirm whether or not to execute the job with the current setting values, all parameters specified by the user (regardless of whether they are required parameters or parameters other than required parameters) can be output by voice. can be verified.

An example of interactive operation between the system and the user, including each of these feedbacks, is shown in FIG. The example of FIG. 12 is an example of operating the MFP 1 so as to make two copies of a monochrome image on both sides. In this example, the number of copies (=2 copies) is an essential parameter. Note that the essential parameter is not limited to the number of copies, and may include a plurality of parameters such as monochrome, color, or paper size.

It should be noted that which parameters among the parameters are essential parameters can be stored in advance in the storage unit of the AI assistant server device 4 . Further, the user may be able to appropriately change which parameter is the essential parameter by operating the operation unit 49 or by accessing the AI assistant server device 4 via the network.

In the example of FIG. 12, the utterances indicated by diagonal lines are user utterances, and the utterances without diagonal lines are system utterances. First, when the system outputs a voice saying "Do you want to copy? Do you want to scan?", the user pronounces "copy" to instruct the use of the copy function. On the system side, in order to request the input of the setting value of "copy" specified by the user, the voice output of "please input the setting value" is performed.

Assume that the user utters "monochrome" in response to this. In this example, the number of copies is an essential parameter as described above. Therefore, the system asks for input of the number of copies, such as "please specify the number of copies". This is an example of "insufficient input feedback". In response to such "insufficient input feedback", the user specifies the number of copies, such as "2 copies". As a result, the lack of input is resolved, so the system responds with "Monochrome, 2 copies. Are you sure?" and prompts an instruction to start copying. This is the "input confirmation feedback" that is output when the information necessary to execute the job is complete.

If the user responds with "Monochrome, 2 copies" in response to the voice output of "Please enter the setting value", the system side does not have the information necessary to execute the job. Since it is complete, the above-mentioned "input confirmation feedback" of "I will make two copies in black and white. Are you sure?" is performed.

Next, when changing the copy mode from single-sided copy to double-sided copy, the user performs voice input of "change to double-sided". In this case, since the information necessary for executing the job is complete, the system side provides "input confirmation feedback" such as "copy in monochrome, 2 copies, double-sided. Are you sure?".

Then, the user responds with "yes" to "input confirmation feedback" of "monochrome, 2 copies. Are you sure?" or "monochrome, 2 copies, double-sided. ', the system responds 'Job will be executed' and executes the job instructed by the user.

(Interactive operation flow)
13 and 14 are sequence diagrams showing the flow of such interactive operations. The sequence diagram of FIG. 13 shows the flow of the first half of the interactive operation, and the sequence diagram of FIG. 14 shows the flow of the second half of the interactive operation.

First, when the user activates the operation voice processing program of the portable terminal 2 (step S11), the voice feedback unit 55 provides voice feedback of "Do you want to copy? Do you want to scan?" (step S12). ). When the user pronounces "copy" (step S13), the communication control unit 52 of the mobile terminal 2 transmits voice data of "copy" to the voice recognition server device 3 and requests text conversion ( step S14). The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "copy" into text data, and transmits the text data to the portable terminal 2 (step S15).

The acquisition unit 51 of the mobile terminal 2 acquires this text data. Further, the communication control unit 52 of the mobile terminal 2 transmits the acquired text data to the AI assistant server device 4 (step S16). The interpretation unit 72 of the AI assistant server device 4 interprets the action and parameters based on the user's uttered phrase indicated by the received text data, as described with reference to FIGS. 10 and 11 . In this example, the user pronounces only "copy", so the number of copies and the like are unknown (insufficient input).

Therefore, the interpretation unit 72 forms an interpretation result by adding a response of "Please input a setting value" to the action of "Copy_Parameter_Setting" (step S17). The communication control unit 73 of the AI assistant server device 4 transmits this interpretation result to the mobile terminal 2 (step S18). The voice feedback unit 55 of the mobile terminal 2 outputs a voice message “Please input the setting value” through the speaker unit 29, and also outputs a text message “Please input the setting value” to the touch panel 27. Display is performed (step S19: input shortage feedback).

Next, since there is insufficient input feedback, the user utters, for example, "on both sides" (step S20). The communication control unit 52 of the portable terminal 2 transmits voice data of "on both sides" to the voice recognition server device 3, and requests text conversion (step S21). The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "on both sides" into text data, and transmits the text data to the portable terminal 2 (step S22).

The acquisition unit 51 of the mobile terminal 2 acquires this text data. Also, the communication control unit 52 of the mobile terminal 2 transmits the acquired text data to the AI assistant server device 4 (step S23). The interpretation unit 72 of the AI assistant server device 4 interprets the action and parameters based on the user's uttered phrase indicated by the received text data. In this example, the user only pronounces "copy" and "both sides", so the number of copies and the like are unknown (insufficient parameter).

Which parameter among the plurality of parameters is the essential parameter can be stored in advance in the storage unit of the AI assist server device 4 . In this case, the interpretation unit 72 determines whether or not the parameters acquired from the mobile terminal device 2 set all the required parameters based on the information of the required parameters stored in the storage unit. If the settings have not been made, the user can be prompted via the mobile terminal device 2 to set the essential parameters.

Therefore, the interpreting unit 72 forms an interpretation result by adding a response of "How many copies do you want to print?" (Step S24). The communication control unit 73 of the AI assistant server device 4 transmits this interpretation result to the portable terminal 2 (step S25). The voice feedback unit 55 of the mobile terminal 2 outputs a voice message “How many copies do you want to print?” Display is performed (step S26: input shortage feedback).

Next, since there is insufficient input feedback again, the user utters, for example, "2 copies" (step S27). The communication control unit 52 of the portable terminal 2 transmits the voice data of "second copy" to the voice recognition server device 3, and makes a text conversion request (step S28). The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "Part 2" into text data, and transmits the text data to the portable terminal 2 (step S29).

The acquisition unit 51 of the mobile terminal 2 acquires this text data. Further, the communication control unit 52 of the mobile terminal 2 transmits the acquired text data to the AI assistant server device 4 (step 30). The interpretation unit 72 of the AI assistant server device 4 interprets the action and parameters based on the user's uttered phrase indicated by the received text data. In this example, the lack of essential parameters for the copy job is resolved by the user transmitting the voice data of "two copies". Therefore, the interpretation unit 72 forms an interpretation result by adding the parameters of "print side=both sides" and "number of copies=2 copies" to the action of "Copy_Confirm" (step S31). The communication control unit 73 of the AI assistant server device 4 transmits this interpretation result to the portable terminal 2 (step S32).

The speech feedback unit 55 of the portable terminal 2 has resolved the shortage of essential parameters and is ready to start copying. (step S33). Here, the text may be generated by reading all or part of the text data stored in the storage unit of the mobile terminal 2 and combining them.

It should be noted that the voice feedback unit 55 may generate a feedback text in the same manner not only in step S33 but also in other steps as long as the interpretation result is obtained from the AI assistant server device 4. If the information is contained in the interpretation result response, there is no need to generate the text of the feedback. Next, the input confirmation feedback described above is performed (step S34). In response to this input confirmation feedback, the user performs voice input instructing to change the setting value or start copying.

FIG. 14 is a sequence diagram showing the flow of operations when steps S35 to S42 in the sequence diagram of FIG. 14 are voice instructions to change the set values. In FIG. 14, when the user makes an utterance to change the setting value (step S35), the text conversion unit 62 of the speech recognition server device 3 generates text data of the changed setting value, is transmitted to the AI assistant server device 4 (steps S36 to S38). The AI assistant server device 4 generates an interpretation result indicating the changed setting value based on the user's uttered phrase indicated by the received text data (step S39), and transmits it to the mobile terminal 2 (step S40).

The voice feedback unit 55 of the portable terminal 2 generates a feedback text based on the response included in the interpretation result (step S41), and the above-mentioned text such as "Copy with XX settings. Are you sure?" By performing input confirmation feedback, it is confirmed whether copying can be started with the changed set values (step S42).

Steps S43 to S50 in the sequence diagram of FIG. 14 are the operation flow of each unit when the start of copying is instructed. That is, when the user responds "yes" to the input confirmation feedback (step S43), the text is converted to text and sent to the AI assistant server device 4 via the mobile terminal 2 (steps S44 to S46). When the AI assistant server device 4 recognizes the copy start instruction based on the received text data, the interpretation result is obtained by adding the parameters of "print side=both sides" and "number of copies=2 copies" to the action of "Copy_Execute". is formed and transmitted to the portable terminal 2 (steps S47 and S48).

Interpretation result conversion unit 53 of portable terminal 2 converts the interpretation result into a job command for MFP 1 (step S49), and transmits it to MFP 1 (step S50). As a result, the MFP 1 can be copy-controlled by the voice input operation.

(Example of information fed back from AI assistant server device 4)
Table 2 below shows an example of interpretation results fed back from the AI assistant server device 4 to the mobile terminal 2 .

As shown in Table 2, for example, actions such as "COPY_PARAMETER_SETTING" for prompting input of job setting values, "COPY_CONFIRM" for prompting confirmation of job setting values, and "COPY_EXECUTE" for notifying start of job execution. is included in the interpretation result and fed back to the mobile terminal 2 .

The voice feedback unit 55 can determine feedback to the user according to actions, parameters, and responses included in the interpretation result. The audio feedback unit 55 may store information corresponding to Table 2 in the storage unit of the mobile terminal device 2 and refer to it in order to determine the content to be fed back. In Table 2, the case of copying was explained as an example, but in the case of printing, scanning, and facsimile, as in Table 2, actions are "PARAMETER_SETTING" for prompting input of job setting values and confirming job setting values. "CONFIRM" may be used to prompt.

Also, for example, a set value for the printing side such as double-sided or single-sided, or parameters such as the number of copies are included in the interpretation result and fed back to the portable terminal 2 . Furthermore, if the essential parameters are missing, a message prompting the input of the missing parameters is included in the interpretation result as a response and fed back to the portable terminal 2 .

(Job execution behavior and dialogue when an error occurs)
Next, FIG. 15 shows that the user responds "yes" to the above input confirmation feedback, such as "Run job. Are you sure?" An example of the interaction as it occurs is shown.

First, the job is executed in MFP 1 when the user responds "yes" to the above-mentioned input confirmation feedback such as "I will execute the job. Are you sure?" If the job is completed without incident, a completion message (speech and text) such as "Job completed. Please be careful not to forget to pick up the manuscript."

On the other hand, for example, when a paper shortage error occurs, job confirmation feedback such as "There is no paper. Please replenish paper and continue or cancel the job." will be In response to this confirmation feedback, the user replenishes the MFP 1 with paper and responds by uttering "continue". On the other hand, when the user gives an instruction to cancel the job by uttering "cancel" according to the confirmation feedback, the MFP 1 is notified of the cancellation of the job, and the job after the occurrence of the error is canceled. Then, via the portable terminal 2, confirmation feedback is provided by outputting voice and text such as "Job has been canceled".

(Job Execution Operation and Flow of Interactive Operation When an Error Occurs)
FIG. 16 is a sequence diagram showing the flow of such a job execution operation and an interactive operation when an error occurs. By executing the instructed job, the MFP 1 transmits to the portable terminal 2 an execution result indicating that the job has been completed normally or an execution result indicating that an error has occurred. When receiving the execution result indicating that the job has been completed normally, the voice feedback unit 55 of the portable terminal 2 outputs a voice message such as "Job completed" (step S52). On the other hand, when receiving an execution result indicating that an error has occurred, the voice feedback unit 55 of the mobile terminal 2 issues, for example, "An error has occurred. Do you want to continue or cancel the job?" Then, a voice message asking whether to continue the job is output (step S53).

Steps S54 to S61 in FIG. 16 are the operation flow of each unit when the user designates continuation. That is, in response to the inquiry "An error has occurred. Do you want to continue or cancel the job?" ), converted into text and transmitted to the AI assistant server device 4 (steps S55 to S57).

The AI assistant server device 4 generates an interpretation result with the action as "COPY_JOB_CONTINUE" and the response as "continue the job", and transmits this to the portable terminal 2 (steps S58 to S59). The execution instructing unit 54 of the portable terminal 2 instructs the MFP 1 to continue the job (step S60). Further, the voice feedback unit 55 of the portable terminal 2 performs input confirmation feedback of "job will be continued" (step S61).

On the other hand, steps S62 to S69 in FIG. 16 are the operation flow of each section when the user designates cancellation of the job. That is, when the user responds "Cancel" to the inquiry "An error has occurred. Do you want to continue or cancel the job?" It is transmitted to the device 4 (steps S63 to S65).

The AI assistant server device 4 generates an interpretation result with the action as "Copy_Job_Cancel" and the response as "Job is canceled", and transmits it to the mobile terminal 2 (steps S66 and S67). The execution instructing unit 54 of the portable terminal 2 instructs the MFP 1 to cancel the job (step S68). Also, the voice feedback unit 55 of the portable terminal 2 performs input confirmation feedback of "Job is canceled" (step S69).

(Adjustment operation of parameters according to device capability of MFP by dialogue)
Next, the voice operation system according to the first embodiment realizes execution of jobs corresponding to the device capabilities of the MFP 1 by interactive operations. FIG. 17 is a sequence diagram showing the flow of operations until it is determined whether or not the MFP 1 has the device capabilities necessary for the job instructed by the user. FIG. 18 is a sequence diagram showing the flow of operations when the job can be executed with the device capabilities of the MFP 1 and when it is difficult to execute the job.

First, in the sequence diagram of FIG. 17, when the mobile voice processing program of the mobile terminal 2 is started (step S71), the processing capability acquisition unit 56 inquires of the MFP 1, for example, about the resolution that can be processed (capability acquisition). : step S72). In response to this inquiry, the MFP 1 responds to the mobile terminal 2 with a processable resolution such as "200dpi to 600dpi" (step S73). As a result, the portable terminal 2 recognizes the resolution that can be processed by the MFP 1 .

Note that the processing capacity acquisition unit 56 may acquire information on the processing capacity in advance at a predetermined timing after the communication between the mobile terminal device 2 and the MFP 1 is established. Also, the acquisition of the processing capacity may be executed by another program, and the processing capacity acquisition unit 56 may indirectly acquire the information of the MFP 1 by acquiring the information acquired by the other program.

Next, the voice feedback unit 55 of the portable terminal 2 provides voice feedback prompting a job instruction such as "Do you want to copy? Do you want to scan?" (step S74). When the user utters "scan" (step S75), the text is converted to text and sent to the AI assistant server device 4 (steps S76 to S78). Since the scan instruction alone is insufficient for the required parameters, the interpreting unit 72 transmits to the portable terminal 2 an interpretation result with an action of "Scan_Parameter_Setting" and a response of "Please enter a setting value" (step S79 to step S80). As a result, the voice feedback unit 55 of the portable terminal 2 performs insufficient input feedback of "Please input the setting value" (step S81).

Next, when the user utters "addressed to Mr. Tanaka at 1000 dpi" in response to such insufficient input feedback, the text is converted to text and sent to the AI assistant server device 4 (steps S82 to S85). The interpreting unit 72 of the AI assistant server device 4 transmits to the portable terminal 2 an interpretation result with the action set to "Scan_Confirm" and the parameters set to "resolution=1000 dpi, destination=Tanaka" (steps S86 to S87). The execution determination unit 57 of the mobile terminal 2 determines whether or not the job specified by the user can be executed by the MFP 1 based on the parameters included in the interpretation result (step S88).

If the MFP 1 is capable of image processing with a resolution of 1000 dpi, as shown in step S89 of FIG. Is it?" is given as input confirmation feedback.

When the user responds "yes" to this input confirmation feedback (step S91), the text is converted to text and sent to the AI assistant server device 4 (steps S92 to S94). The AI assistant server device 4 sets the action to "Scan_Execute" and transmits the interpretation result to the portable terminal 2 with "resolution=1000, destination=Tanaka" (steps S95 to S96).

The interpretation result converter 53 of the portable terminal 2 converts the interpretation result into a job command (step S97). The communication control unit 52 of the mobile terminal 2 transmits the job command to the MFP 1 (step S98).

On the other hand, if 600 dpi is the limit of the processing capability of the MFP 1, it will be difficult for the MFP 1 to process a 1000 dpi image specified by the user. In this case, in step S90, the portable terminal 2 gives confirmation feedback, "1000 dpi cannot be specified. Scan at 600 dpi and send to Mr. Tanaka. Are you sure?"

Execution determination unit 57 can select the function or value closest to the function or processing capacity value instructed by the user within the range of processing capacity of MFP 1 . For example, if the resolution that can be processed by the MFP 1 is in the range of 200-600 dpi, 600 dpi, which is the closest to 1000 dpi specified by the user, is selected. Then, the voice feedback unit 55 outputs the ability or value selected by the execution determination unit 57 as confirmation feedback.

When the user says "yes" to such confirmation feedback (step S91), the text is converted to text and sent to the AI assistant server device 4 (steps S92 to S94). The AI assistant server device 4 sets the action to "SCAN_EXECUTE" and transmits the interpreted result to the portable terminal 2 with "resolution=600, destination=Tanaka" (steps S95 to S96).

The interpretation result converter 53 of the portable terminal 2 converts the interpretation result into a job command (step S97). The communication control unit 52 of the mobile terminal 2 transmits the job command to the MFP 1 (step S98). Thereby, the MFP 1 can be operated by the voice input operation so that the job is executed within the processing capacity.

(Operation of sending a scanned image to a destination searched from the mobile terminal's address book)
Next, the operation of transmitting the scanned image to the destination searched from the address book of the mobile terminal 2 will be described. The sequence diagram of FIG. 19 shows the flow of the first half of the operation, and the sequence diagram of FIG. 20 shows the flow of the second half of the operation.

First, in the sequence diagram of FIG. 19, when the portable voice processing program of the portable terminal 2 is started (step S101), the voice feedback unit 55 issues a job instruction such as "Do you want to copy? Do you want to scan?" Prompting voice feedback is performed (step S102). When the user utters "scan" (step S103), the text is converted to text and sent to the AI assistant server device 4 (steps S104 to S106). Since the scan instruction alone is insufficient for the required parameters, the interpreting unit 72 transmits to the portable terminal 2 an interpretation result in which the action is "SCAN_PARAMETER_SETTING" and the response is "Please enter a setting value" (step S107 to step S108). As a result, the voice feedback unit 55 of the portable terminal 2 performs insufficient input feedback such as "Please input the setting value" (step S109).

Next, when the user utters "addressed to Mr. Tanaka" in response to such insufficient input feedback (step S110), the text is converted to text and sent to the AI assistant server device 4 (steps S111 to S113). The interpreting unit 72 of the AI assistant server device 4 transmits to the portable terminal 2 the interpretation result with the action set to "SCAN_CONFIRM" and the parameter set to "destination=Tanaka" (steps S114 to S115). The search unit 58 of the portable terminal 2 searches for Mr. Tanaka's e-mail address from the storage area of the address book stored in the RAM 22 based on the parameters included in the interpretation result (step S116).

The search target is not limited to the RAM 22, but may be a server connected to the mobile terminal device via a network.

If only one Mr. Tanaka is recorded in the storage area of the address book, in step S117 of FIG. Create an e-mail with the address set as the destination. The e-mail is sent with the scanned image attached. As a result, the scanned image is sent to Mr. Tanaka.

When a plurality of Mr. Tanaka are recorded in the storage area of the address book, that is, when there are a plurality of destinations including "Tanaka" as information for identifying the destination, the audio feedback unit 55 of the mobile terminal 2 outputs " Mr. Tanaka is registered multiple times.Please select a destination." Then, the search unit 58 of the mobile terminal 2 displays a list of Mr. Tanaka's names, e-mail addresses, etc. searched from the storage area of the address book on the touch panel 27 (step S119).

The user selects a desired Mr. Tanaka from this list. The communication control unit 52 of the mobile terminal 2 creates an e-mail in which Mr. Tanaka's e-mail address selected by the user is set as the destination. The e-mail is sent with the scanned image attached. As a result, the scanned image is transmitted to Mr. Tanaka corresponding to the destination desired by the user.

If Mr. Tanaka is not recorded in the storage area of the address book, the voice feedback unit 55 of the mobile terminal 2 provides insufficient input feedback such as "Mr. Tanaka is not registered. Please specify the destination." (Step S120). Then, the search unit 58 of the mobile terminal 2 displays a list of Mr. Tanaka's name and e-mail address of each user searched from the storage area of the address book on the touch panel 27, for example, in the order of the Japanese syllabary (step S121).

Further, by displaying a keyboard on the touch panel 27, the user can manually input the destination e-mail address, thereby obtaining the name and the e-mail address. The acquired name and e-mail address may be registered in the address book.

The user selects a desired destination from this list. The communication control unit 52 of the mobile terminal 2 creates an e-mail address of the destination selected by the user. The e-mail is sent with the scanned image attached. As a result, the scanned image is transmitted to the destination desired by the user.

That is, when Mr. Tanaka or another user to whom the scanned image is to be sent is selected, the voice feedback unit 55 of the mobile terminal 2 will say, for example, "Scan and send to Mr. Tanaka. Are you sure? ” is given as confirmation feedback (step S122).

When the user responds "yes" to this confirmation feedback (step S123), the text is converted to text and sent to the AI assistant server device 4 (steps S124 to S126). The AI assistant server device 4 sets the action to "Scan_Execute" and transmits the interpretation result to the mobile terminal 2 with "resolution=600, destination=Tanaka" (steps S127 to S128).

The interpretation result converter 53 of the portable terminal 2 converts the interpretation result into a job command (step S129). The communication control unit 52 of the portable terminal 2 transmits the job command to the MFP 1 together with destination information such as the name and e-mail address of the destination (step S130). As a result, the e-mail attached with the scanned image is sent to the destination desired by the user.

(Operation of searching for print target from mobile terminal memory and printing)
Next, the operation of printing a file retrieved from the memory such as the RAM 22 of the portable terminal 2 will be described. The sequence diagram of FIG. 21 shows the flow of the first half of the operation, and the sequence diagram of FIG. 22 shows the flow of the second half of the operation.

First, in the sequence diagram of FIG. 21, when the portable voice processing program of the portable terminal 2 is started (step S141), the voice feedback unit 55 asks, for example, "Do you want to copy? Scan? Do you want to print?" A voice feedback prompting for a job instruction is provided (step S142). When the user utters "print" (step S143), the text is converted to text and sent to the AI assistant server device 4 (steps S144 to S146). Since the command "print" alone lacks the required parameters, the interpreting unit 72 transmits to the portable terminal 2 an interpretation result in which the action is "PRINT_PARAMETER_SETTING" and the response is "Please enter a setting value." (steps S147 and S148). As a result, the voice feedback unit 55 of the portable terminal 2 performs insufficient input feedback of "Please input the setting value" (step S149).

Next, when the user utters "print file A" in response to such insufficient input feedback (step S150), the text is converted to text and sent to the AI assistant server device 4 (steps S151 to S153). The interpreting unit 72 of the AI assistant server device 4 transmits to the portable terminal 2 the result of interpretation with the action set to "PRINT_CONFIRM" and the parameter set to "print target=file A" (steps S154 to S155). The search unit 58 of the mobile terminal 2 searches for the file A from the memory such as the RAM 22 based on the parameters included in the interpretation result (step S156).

The search target is not limited to the RAM 22, but may be a server connected to the mobile terminal device via a network.

When only one file A is recorded in the memory, in step S157 of FIG. 22, the communication control unit 52 of the mobile terminal 2 sets this file A as print data. The print data of this file A is sent to the MFP 1 and printed.

When two or more files A are present in the memory, the voice feedback unit 55 of the mobile terminal 2 provides insufficient input feedback such as "Multiple files A are registered. Please select data to be printed." (Step S158). Then, the search unit 58 of the mobile terminal 2 displays on the touch panel 27 a list of files including "file A" in the file name searched from the memory (step S159). The user selects the desired file A from this list. The communication control unit 52 of the portable terminal 2 transmits the file A selected by the user to the MFP 1 and requests printing, as will be described later.

If the file A is not recorded in the memory, the voice feedback unit 55 of the mobile terminal 2 provides insufficient input feedback such as "File A is not registered in the memory. Please specify the file to be printed." (Step S160). Then, the search unit 58 of the mobile terminal 2 displays a list of each file searched from the memory on the touch panel 27 (step S161). The user selects a desired file from this list. The communication control unit 52 of the mobile terminal 2 transmits the file selected by the user to the MFP 1 and requests printing.

That is, when the file A desired to be printed or another file is selected, the voice feedback unit 55 of the mobile terminal 2 gives confirmation feedback such as "Printing of file A is started. Are you sure?" step S162).

When the user responds "yes" to this confirmation feedback (step S163), the text is converted to text and sent to the AI assistant server device 4 (steps S164 to S166). The AI assistant server device 4 sets the action to "Print_Execute" and transmits to the portable terminal 2 the interpreted result of "print target=file A" (steps S167 to S168).

The interpretation result converter 53 of the portable terminal 2 converts the interpretation result into a job command (step S169). The communication control unit 52 of the mobile terminal 2 transmits the job command to the MFP 1 together with the file selected by the user (step S170). As a result, the file desired by the user is printed in MFP 1 .

(Effect of the first embodiment)
As is clear from the above description, the voice operation system of the first embodiment installs an operation voice processing program, which is a platform application program, in the portable terminal 2, and this application is the MFP 1, the voice recognition server device 3, and the voice recognition server device 3. It relays communication between AI assistant server devices 4 . When the user speaks into the microphone unit 29 provided in the portable terminal 2, the content of the user's speech is analyzed and the MFP 1 is operated to perform the operation instructed by the user (voice input operation).

This eliminates the need for GUI (Graphical User Interface) operations such as the touch panel 27 by commanding complicated operations by voice. input operation can be made possible. In addition, even the elderly or inexperienced users can quickly and easily perform operations desired by the user, such as complex network settings, advanced job settings, or introduction of new applications, through interactive operation support. be executable.

In addition, since the user's intention is analyzed based on the text of the content uttered by the user, the AI assistant server device 4 side can determine the processing content based on the user's utterance.

(Second embodiment)
Next, the voice operation system of the second embodiment will be explained.

(System configuration)
FIG. 23 is a system configuration diagram of the voice operation system of the second embodiment. As shown in FIG. 23, the voice operation system of the second embodiment includes one or more multifunction peripherals (MFP) 1, one or more smart speakers 100 (or smart phones, tablet terminals, etc.), 1 It has the above voice recognition server device 3 and one or more AI assistant server devices 4 . The MFP 1, the smart speaker 100, the voice recognition server device 3, and the AI assistant server device 4 are interconnected via a network 5 such as a LAN. As a result, the voice recognition server device 3 and the AI assistant server device 4 can provide cloud services.

In addition to the MFP 1, other electronic equipment such as an electronic blackboard or a projector may be provided as an external device.

The AI assistant server device 4 has a management database (management DB) 101 and a linking database 102 . The management DB 101 is a database that stores content (data) such as text data, image data, and voice data provided by the AI assistant server device 4 as a cloud service. Further, in the linking DB 102, for example, each device ID specifying each smart speaker 100 and the device ID of the MFP 1 associated with each smart speaker 100 are associated and stored. That is, the device ID of each smart speaker 100 and the device ID of the MFP 1 are associated and stored in the linking DB so that the MFP 1 that can be voice-operated by each smart speaker 100 can be specified.

As the management DB 101 and the linking DB 102, a storage unit such as an HDD that the cloud service has on the network 5 can be used. In addition, one or both of the management DB 101 and the linking DB 102 may be stored in another server device that can be accessed via the network 5 using a cloud service.

The smart speaker 100 has a microphone unit (corresponding to the microphone unit 29 in FIG. 3) for obtaining voice input from the user for operating the MFP 1 by voice. The smart speaker 100 also includes a speaker unit (corresponding to the speaker unit 28 in FIG. 3) for performing audio feedback prompting the user to input missing information. The smart speaker 100 also performs data communication such as voice data, text data, and image data between the voice recognition server device 3 and the AI assistant server device 4 that provide cloud services. The speech recognition server device 3 analyzes the speech data received from the smart speaker 100 and converts it into text data. It also interprets the user's intention based on text data and pre-registered dictionary information, and transmits the result of interpretation to AI assistant server device 4 .

AI assistant server device 4 converts the interpretation result received from speech recognition server device 3 into a job execution command in a format interpretable by MFP 1 , and transmits the job execution command to MFP 1 . The MFP 1 executes job execution instructions sent from such cloud services.

(MFP hardware configuration)
The MFP 1 has multiple image processing functions such as a printer function and a scanner function. That is, the MFP 1 has a controller 7, a communication section 15, an operation section 16, a scanner engine 17 and a printer engine 18, as described with reference to FIG.

The controller 7 has a CPU 10 , an ASIC 11 , a memory 12 , a HDD (Hard Disk Drive) 13 and a timer 14 . The CPU 10 to timer 14 are interconnected so as to be communicable via a bus line.

The communication unit 15 is connected to the network 5, and acquires a job execution command input by voice using the mobile terminal 2, such as a scan instruction or a print instruction, as will be described later.

The operation unit 16 is a so-called touch panel in which a liquid crystal display (LCD) and a touch sensor are integrally formed. When the operator uses the operation unit 16 to issue a command to execute a desired operation, the operator touches an operation button (software key) displayed on the operation unit 16 to specify the desired operation.

The scanner engine 17 controls the scanner unit to optically read the document. The printer engine 18 controls the image writing unit to print an image on transfer paper or the like. The CPU 10 comprehensively controls the image forming apparatus. The ASIC 11 is a so-called large-scale integrated circuit (LSI: Large-Scale Integration), and performs various image processing necessary for images processed by the scanner engine 17 and the printer engine 18 . The scanner engine 17 and printer engine 18, which are means for executing a job execution command acquired from the mobile terminal device 2, correspond to functional units.

The memory 12 stores various applications executed by the CPU 10 and various data used when executing the applications. The HDD 13 stores image data, various programs, font data, various files, and the like. An SSD (Solid State Drive) may be provided instead of the HDD 13 or together with the HDD 13 .

(Device hardware configuration)
The smart speaker 100 is formed by connecting a CPU 21, a RAM 22, a ROM 23, an interface unit (I/F unit) 24, and a communication unit 25 to each other via a bus line 26, similar to the mobile terminal 2 shown in FIG. It is

The ROM 23 stores an operation voice processing program. By executing this operation voice processing program, the CPU 21 acquires voice information from the microphone unit 29 and transmits the voice information to the cloud service. In addition, the CPU 21 controls display of data (audio data, text data, image data, etc.) acquired from the cloud service on the touch panel 27 or controls audio output via the speaker section 28 .

A touch panel 27 , a speaker section 28 and a microphone section 29 are connected to the I/F section 24 . The microphone unit 29 collects (obtains) voice input from the user that serves as a job execution command to the MFP 1, in addition to the call voice. The input voice is transmitted to the cloud service via the communication unit 25 .

(Hardware configuration of cloud service)
As shown in FIG. 4, the speech recognition server device 3 and the AI assistant server device 4 that constitute the cloud service include a CPU 31, a RAM 32, a ROM 33, an HDD 34, an interface unit (I/F unit) 35, and a communication unit 36 connected to a bus line. 37 are connected to each other.

A display unit 38 and an operation unit 39 are connected to the I/F unit 35 . The HDD 34 converts the voice data into text data, judges whether or not the text data matches predefined dictionary information, and converts the text data to an intent and job An operation voice conversion program is stored for conversion into parameters indicating variables such as conditions. The HDD 34 also stores a voice assistant program that holds dictionary information, and a management program that converts job execution instructions into a format interpretable by the MFP based on intents and parameters and transmits the job execution instructions to registered MFPs. ing.

The CPU 31 executes these programs. Note that the operation voice conversion program, the voice assistant program, and the management program may be executed by one server device, or may be executed by different server devices. Furthermore, these programs may be executed by cooperation of a plurality of server devices.

(Each function based on each program)
FIG. 24 is a diagram showing functions realized by the CPU 21 of the smart speaker 100 executing the operation voice processing program. Also, FIG. 24 is a diagram showing functions realized by the CPU 31 of the voice recognition server device 3 executing the operation voice conversion program and the voice assistant program. Furthermore, FIG. 24 is a diagram showing each function realized by the CPU 31 of the AI assistant server device 4 executing the management program.

(Function of operation voice processing program)
The CPU 21 of the smart speaker 100 functions as an acquisition unit 151 , a communication control unit 152 and a feedback unit 153 by executing an operation voice processing program stored in a storage unit such as the ROM 23 .

Acquisition unit 51 acquires a user's instruction voice for voice-operating MFP 1 , collected via microphone unit 29 . Note that the acquisition unit 151 may acquire user operations and instructions via a touch panel, a physical switch, or the like. The communication control unit 152 controls communication between the smart speaker 100 and cloud services. The communication control unit 152 transmits the information acquired by the acquisition unit 151 to the cloud service, and acquires text data, image data, and audio data from the cloud service. Further, when transmitting the information acquired by the acquisition unit 151 to the cloud service, the communication control unit 152 transmits the device ID that identifies the smart speaker 100 .

The feedback unit 153 feeds back a voice prompting an input operation for compensating for missing data, or feeds back a voice confirming the input operation, in order to realize an interactive voice input operation. Note that the feedback unit 153 may provide the user with a text or image that serves as feedback on the display of the touch panel 27 .

In this example, the acquisition unit 151 to the feedback unit 153 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). . In addition, the acquisition unit 151 to the feedback unit 153 may be realized by the operation voice processing program alone, may be executed by another program to execute part of the processing, or may be indirectly executed by using another program. can be

(Function of operation voice processing program)
Next, the CPU 31 of the speech recognition server device 3 functions as an acquisition unit 155 , a text conversion unit 156 , an analysis unit 157 and an output unit 158 by executing the operation speech conversion program stored in the HDD 34 . The acquisition unit 155 acquires voice data input by the user, which is transmitted from the smart speaker 100 . In addition, the acquisition unit 155 may acquire user operations such as the touch panel, buttons, or switches of the smart speaker 100 .

The text conversion unit 156 converts voice data input by the user into text data. Interpretation unit 157 interprets the instruction from the user based on the text data. Specifically, the interpreting unit 157 checks whether or not the words included in the text data match the dictionary information based on the dictionary information provided from the voice assistant program. If there is a match, it is converted into an intent indicating the user's intention and a parameter indicating a variable such as a job condition. The interpretation unit 157 transmits the intent and parameters to the management program, and at this time, also transmits the device ID of the smart speaker 100 for specifying the smart speaker 100 that acquires the audio data. The output unit 158 controls communication of the communication unit 36 so as to transmit data such as text data, audio data, and image data to the smart speaker 100 .

In this example, the acquisition unit 155 to the output unit 158 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, each function of the acquisition unit 155 to the output unit 158 may be realized by the operation voice conversion program alone, may be executed by another program for part of the processing, or may be performed by using another program. It may be executed indirectly. For example, part or all of the functions of the interpretation unit 157 of the operation voice conversion program may be executed by the voice assistant program. In this case, for example, it is necessary to check whether words contained in the text data match the dictionary information, and if they match, convert intents that indicate the user's intentions and parameters that indicate variables such as job conditions. , a voice assistant application, and the interpreting unit 157 acquires the intent and parameters from the voice assistant program.

(Function of voice assistant program)
Next, CPU 31 of voice recognition server device 3 functions as providing unit 159 by executing the voice assistant program stored in HDD 34 . The providing unit 159 manages dictionary information that defines in advance the relationship between text data, intents, and parameters stored in the HDD 34, and provides it to the operation voice conversion program. Also, the operation instruction from the user may be interpreted based on the text data. Specifically, the text data is acquired from the operation voice conversion program, and it is checked whether the words contained in the text data match the dictionary information. and parameters. The intent and parameters are then provided to the operation speech conversion program.

In this example, the providing unit 159 is implemented by software, but part or all of these may be implemented by hardware such as an IC (Integrated Circuit), etc., as described above. Like any other program.

(Function of management program)
By executing the management program stored in the HDD 34, the CPU 31 of the AI assistant server device 4 obtains the acquisition unit 160, the interpretation result conversion unit 161, the execution instruction unit 162, the device information acquisition unit 163, the execution determination unit 164, the notification It functions as a unit 165 , a management unit 166 and a search unit 167 .

The acquisition unit 160 acquires the intent, the parameter, the device ID of the smart speaker 100, and the like, which are transmitted from the speech recognition server device 3 . The interpretation result conversion unit 161 converts the interpretation result such as the intent and parameters converted by the operation voice conversion program into a job execution command that can be interpreted by the MFP 1 . Execution instruction unit 162 transmits a job execution instruction to MFP 1 to instruct execution of the job.

Note that the execution instruction unit 162 refers to the linking DB 102 and transmits a job execution instruction to the MFP 1 associated with the smart speaker 100 . That is, the execution instructing unit 162 acquires the device ID specifying the smart speaker 100 used by the user together with the intent and the parameter, searches the MFP 1 associated with the acquired device ID from the linking DB 102, and searches the A job execution command is transmitted to MFP1.

Device information acquisition unit 163 acquires device information indicating the processing capability of MFP 1 , such as the maximum number of pixels that can be processed, from MFP 1 . The device information acquisition unit 163 stores and manages the device information acquired from a plurality of external devices such as the MFP 1 in a storage unit such as an HDD in association with specific information such as a device ID for specifying each external device.

Further, the device information acquisition unit 163 acquires a connection state indicating whether or not a communication connection with the MFP 1 is established, a power state such as power ON/OFF of the MPF 1 or sleep mode, presence/absence of errors and types of errors, paper Also, the device status including remaining status of consumables such as toner, user's login status, and authorization information indicating functions permitted to be used by the logged-in user is obtained.

The execution determination unit 57 compares the processing capability of the MFP 1 indicated by the device information with the job specified by the user (that is, the intent and parameters generated by the operation voice conversion program). It is determined whether or not the MFP 1 can execute the job. If it is determined that the job specified by the user can be executed, it transmits a job execution command to the MFP. Further, when it is determined that the operation cannot be performed, response information such as an error message is fed back to the smart speaker 100 via the operation voice conversion program.

The notification unit 165 notifies the operation voice conversion program of text data, voice data, image data, etc. as a response to the user's job execution instruction. Also, if the parameters indicating the job conditions for executing the job are insufficient, the user is prompted to input the parameters by providing feedback to the terminal via the operation voice conversion program. Here, parameter information may be sent as information necessary to confirm missing parameters, or text data, voice data, and image data may be sent as information necessary to prompt the user to specify parameters. You can send it.

The management unit 166 associates and links the device ID of the smart speaker 100 and the device ID of the MFP 1 based on information and instructions input to the MFP 1, the smart speaker 100, or a client device connected to the cloud service via a network. Register in DB 102 for use. Note that the linking DB 102 holds information that associates the device ID of the smart speaker 100 and the device ID of the MFP in, for example, a management table.

In this example, the acquiring unit 160 to the searching unit 167 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). is similar to the other programs mentioned above.

In addition, the above-mentioned operation voice processing program, operation voice conversion program, voice assistant program and management program can be read by computer devices such as CD-ROM and flexible disk (FD) as files in installable format or executable format. may be recorded on a suitable recording medium and provided. Alternatively, it may be provided by being recorded on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), or a semiconductor memory. Moreover, it may be provided in a form of being installed via a network such as the Internet, or may be provided by being incorporated in advance in a ROM or the like in the device.

(Overall operation of voice input operation)
FIG. 25 is a diagram for explaining the overall voice input operation in the voice operation system according to the second embodiment. The example of FIG. 25 is an example in which the double-sided copy function of the MFP 1 is operated by voice input via the smart speaker 100 . In this case, the user activates the operation voice processing program of the smart speaker 100 and utters, for example, "copy on both sides" (step S181). This user's voice is collected by the microphone unit 29 of the smart speaker 100 and acquired by the acquisition unit 151 . Note that the operation voice processing program may be started at the same time when the smart speaker 100 is started, or may be started when a predetermined operation or a predetermined sound is performed on the smart speaker 100 .

The communication control unit 152 of the smart speaker 100 transmits the voice data of "copy on both sides" to the voice recognition server device 3 of the cloud service (step S182). At this time, the communication control unit 152 transmits the device ID of the smart speaker 100 for identifying the smart speaker 100 to the voice recognition server device 3 . The text conversion unit 156 by the operation voice conversion program of the voice recognition server device 3 converts the voice data of "copy on both sides" into text data (step S183).

Based on the dictionary information acquired from the voice assistant program (steps S184 and S185), the interpreting unit 157 checks whether or not the words included in the text data match the dictionary information (step S186). If there is a match, the AI is converted into a parameter indicating variables such as an intent indicating an operation instructed by the user and a job condition, and a device ID for specifying the smart speaker 100 that acquires the voice data. It is transmitted to the assistant server device 4 (step S187).

Specifically, in this example, the interpreting unit 157 interprets the operation requested to the MFP 1 to be "copy (intent: Copy_Execute)", and interprets "the printing surface is double-sided (printing surface = double-sided"). )” (step S186). In this way, the interpretation unit 157 generates an interpretation result indicating the type (intent) and content (parameters) of the job specified by the user, based on the text data. This interpretation result is sent to the management program (step S187). At this time, the interpreting unit 157 may transmit to the AI assistant server device 4 a device ID that specifies the smart speaker 100 that is the transmission source of the audio data in association with the interpretation result.

The interpretation result conversion unit 161 of the management program of the AI assistant server device 4 converts the interpretation result obtained from the smart speaker 100 into a job command of the MFP 1 (step S188). An example of the interpretation result and job instructions converted from the interpretation result are as described with reference to Table 1 above. Note that the interpretation result conversion unit 161 may store information corresponding to Table 1 in a storage unit such as the HDD 34 of the AI assistant server device 4 and refer to it in order to convert the interpretation result into a job command.

In the example of Table 1 above, "COPY_EXECUTE", "SCAN_EXECUTE", "PRINT_EXECUTE" and "FAX_EXECUTE" are examples of intents. In addition, "print surface" and "number of copies" are examples of parameters. The parameters include all parameters that can be specified as job setting values.

The interpretation result conversion unit 161 of the AI assistant server device 4 converts the interpretation result of "COPY_EXECUTE" into a job command of the MFP 1 "execute copy job". Similarly, the interpretation result conversion unit 161 converts the interpretation result of “SCAN_EXECUTE” into a job command of the MFP 1 “execute scan job”. Similarly, the interpretation result conversion unit 161 converts the interpretation result of "PRINT_EXECUTE" into a job command of the MFP 1 "execute print job". Similarly, the interpretation result conversion unit 161 converts the interpretation result of "FAX_EXECUTE" into a job command of the MFP 1 "execute FAX job".

The above description is an example of a basic job executed by the MFP 1, but jobs that can be interpreted by the cloud service are not limited to such basic jobs. For example, it is possible to instruct a job to cause a communication target to collect device information and transmit it to a cloud service, a job to display predetermined information stored in a storage unit of the communication target on a display unit of the communication target, and the like.

Further, when the interpretation result includes the parameter of "printing surface", the interpretation result conversion unit 161 forms a job command for the MFP 1 to "change the setting value of the printing surface". Similarly, when the interpretation result includes the parameter "number of copies", the interpretation result conversion unit 161 forms a job command for the MFP 1 to "change the set value of the number of copies".

That is, the interpretation result conversion unit 161 determines the user's intention, for example, the type of job to be executed by the MFP 1, based on the information included in the "Intent" of the interpretation result, and converts the value included in the "Parameter" to the set value of the job. Then, the interpreted result is converted into a job instruction.

The interpretation result conversion unit 161 of the AI assistant server device 4 transmits the job command thus formed to the MFP 1 via the communication unit 36 (step S189). In the case of this example, a job command of "execute copy job (print side=both sides)" is transmitted to MFP1. Communication control unit 52 transmits a job command to the MFP specified by the management unit. In other words, it is possible to instruct the execution of the job to the MFP identified by the ID associated with the device ID that identifies the terminal that transmitted the audio data. As a result, MFP 1 executes double-sided printing.

(Details of the interpretation operation in the cloud service device)
The HDD 34 of the speech recognition server device 3 stores dictionary information. The operation voice conversion program generates intents and parameters based on this dictionary information. More specifically, the operation voice conversion program determines whether or not words included in text data converted from voice data match dictionary information. and generate an interpretation result containing parameters.

Dictionary information may be in any form as long as intents and parameters can be generated. As an example, in this example, the dictionary information includes entity information, intent information, and association information. Entity information is information that associates job parameters with natural language. Multiple synonyms can be registered for one parameter. The intent information is information indicating the type of job. The association information is information that associates a user's uttered phrase (natural language) and entity information, and a user's uttered phrase (natural language) and intent information, respectively. The association information allows correct interpretation even if the utterance order or nuance of the parameters changes slightly. Also, the association information may be used to generate a response text (interpretation result) based on the input content.

FIG. 10 described above shows an example of entity information. The example in FIG. 10 is entity information corresponding to print color. In FIG. 10, the characters "print Color" indicate the entity name. In FIG. 10, characters such as “auto_color”, “monochrome”, “color”, etc. in the left column indicate parameter names. In FIG. 10, characters such as “auto_color”, “monochrome, black and white”, “color, full color”, etc. in the right column indicate synonyms.

As can be seen from FIG. 10, as entity information, parameters and synonyms are associated and stored. By registering synonyms along with parameters, for example, when instructing to copy in monochrome, even if you say "Please copy by black and white" or "Please copy by monochrome", you can set parameters can be made possible.

FIG. 11(a) shows an example of a user's uttered phrase, FIG. 11(b) shows an intent name, and FIG. 11(c) shows entity information. As shown in FIGS. 11A to 11C, the user's utterance is dragged by operating the operation unit 39 on the screen displayed on the display unit 38 provided in the cloud service 4. FIG. Alternatively, the user's utterance is dragged by operating the operation unit of the device on the screen displayed on the display unit of the device that accessed the cloud service via the network.

This makes it possible to select entity information to be associated. Also, if a value (VALUE) is set in the selected entity information, the parameters entered in the response are changed. For example, when "Please copy by black and white" is uttered and the value is "SprintColor", "printColor=monochrome" is returned as the return value. On the other hand, if the value is "SprintColor.original", "printColor=black and white" is returned as the return value. Here, if the value is "SprintColor.original", the content of the user's utterance can be returned as it is as a parameter of the response.

(interactive behavior)
Next, the voice operation system of the embodiment realizes an interactive system in which the system responds based on the contents of input from the user. In the voice operation system according to the embodiment, in addition to responding with fixed sentences necessary for dialogue, two types of responses, "input insufficient feedback" and "input confirmation feedback", are given as responses specific to the operation of the MFP 1. , realizes an interactive MFP operation system that enables interactive MFP operation.

"Insufficient input feedback" is a response that is output when the information required to execute a job is not complete. This is output when the user's input result cannot be heard, or when a required parameter is missing. In other words, there is no need to provide insufficient input feedback for parameters other than essential parameters, even if they are not indicated. In addition to the parameters, processing for confirming a function to be used, such as a copy function or a scan function, may also be included.

For example, the cloud service device may change the functions and parameters for confirmation from the user according to the type of external device with which the cloud service device is communicating. In this case, the device information acquisition unit 163 acquires information indicating the type and function of the external device at a predetermined timing after communication with the external device is established, and based on the acquired information, determines the function and parameters to be confirmed by the user. For example, the feedback unit 153 may decide.

For example, if the type of the external device is MFP1, the user can confirm the functions of copy, print, scan, FAX, etc. that MPF1 has. However, the user may be asked which function to use. Also, the essential parameters may be changed according to the setting conditions specified by the user. In other words, if the printing conditions specified by the user are variable-magnification printing, the required parameter is the paper size to be printed. Settings such as the finished size and the number of pages included in one page are set as essential parameters.

"Input confirmation feedback" is a response that is output when the information necessary for executing a job is complete. That is, input confirmation feedback is provided when indicated for all required parameters. Also, the input confirmation feedback is provided to prompt the user to select whether to execute the job with the current set values or to change the set values. In addition, in order to confirm whether or not to execute the job with the current setting values, all parameters specified by the user (regardless of whether they are required parameters or parameters other than required parameters) can be output by voice. can be verified.

(Example of interactive operation of the MFP to make two copies on both sides)
26 to 34 are sequence diagrams showing the flow of such interactive operations.

(Flow of voice assistant program startup operation)
First, the sequence diagram of FIG. 26 shows the flow of the activation operation of the voice assistant program. In FIG. 26, first, the user activates the operation voice processing program of the smart speaker 100, and then instructs activation of the voice assistant program by voice input, for example. As an example, the voice assistant program of the voice recognition server device 3 can be activated by the user uttering an activation word for activating the voice assistant program (step S191).

The communication control unit 152 of the smart speaker 100 transmits the voice data of the activation word to the voice recognition server device 3 (step S192). The text conversion unit 156 of the speech recognition server device 3 converts the speech data of the activation word into text data (step S193). Further, the interpretation unit 157 of the voice recognition server device 3, based on the dictionary information (dictionary information stored in the HDD 34) provided by the voice assistant program (steps S194 and S195), converts the text data of the activation word into The intent and parameters are converted (text interpretation) (step S196), and transmitted to the AI assistant server device 4 together with the device ID of the smart speaker 100 (step S197).

The execution determination unit 164 of the AI assistant server device 4 determines whether or not all the conditions necessary for job execution are met based on the intent and parameters (steps S198 to S202). For example, the execution determination unit 164 determines whether or not the essential parameters are satisfied (step S202), and if the essential parameters are not satisfied, the operation voice conversion program of the speech recognition server device 3, the smart speaker 100 (step S203, step S204).

The response information includes, for example, text data, voice data, image data, and the like. Here, as an example, the text data “Do you want to copy? Do you want to scan?” is sent to the smart speaker 100 . Note that the content of the message is not limited to this as long as the message prompts the user to enter the type of job or setting conditions for the job. Also, if the smart speaker 100 cannot convert text data into voice data, it may be transmitted in the form of voice data. Furthermore, feedback to the user may be provided by displaying text or images on the touch panel 27 in addition to voice output. As a result, the feedback unit 153 of the smart speaker 100 provides audio feedback of "Do you want to copy? Do you want to scan?" (step S205).

It should be noted that the above steps may be omitted if the type of job and setting conditions have been specified when instructing to activate the voice assistant program of the voice recognition server device 3 . Also, the execution determination unit 164 of the AI assistant server device 4 may search the MFP 1 corresponding to the acquired device ID of the smart speaker 100 from the linking DB 102 before transmitting the response information.

At this time, if the MFP 1 associated with the device ID of the smart speaker 100 is not stored in the association DB 102, the execution determination unit 164 notifies the user that the smart speaker 100 is not associated with the communication target. For example, the execution determination unit 164 forms response information including a response that "this device is not associated with equipment". Here, the execution determination unit 164 may include in the response a method for associating the device with the communication target. Note that the retrieval of the MFP 1 from the linking DB 102 and the formation of the response information may be performed at any timing other than when the device ID of the smart speaker 100 is obtained.

Also, the execution determination unit 164 may check the state of the device to be communicated with prior to sending the response information. The execution determination unit 164 acquires the device information by the device information acquisition unit 163 or refers to the device information acquired in advance by the device information acquisition unit 163 to determine whether communication with the communication target is possible, and Determine whether the communication target is available. At this time, if the connection with the MFP 1 associated with the device ID of the smart speaker 100 has not been established, or if the MFP cannot be used because it is being activated, the execution determination unit 164 notifies the user.

For example, the execution determination unit 164 forms response information including a response that "the device is offline" or "the device is being prepared". Here, the execution determination unit 164 may include the countermeasure method in the response. Note that the confirmation of the device status may be performed at any other timing when the intent, parameters, and device ID are obtained from the operation speech conversion program of the speech recognition server device 3 .

Also, the execution determination unit 164 may check the state of the application that executes the function specified by the user as the communication target prior to the transmission of the response information. The execution determination unit 164 determines whether the application is installed by acquiring the device information by the device information acquisition unit 163 or by referring to the device information previously acquired by the device information acquisition unit 163. It is determined whether or not the

For example, if the application related to the copy function whose execution has been instructed is not installed in the MFP 1 associated with the device ID of the smart speaker 100, or if the application related to the copy function cannot be used because it is running, the execution determination unit 164 prompts the user to to notify. For example, the execution determination unit 164 forms response information including a response that "the application is not installed" or "the application is currently unavailable". Here, the execution determination unit 164 may include the countermeasure method in the response.

Note that confirmation of the status of the application may be performed at any other timing after acquiring the intent, parameter, and device ID from the operation voice conversion program. Further, the execution determination unit 164 may acquire device information by acquiring the device information by the device information acquisition unit or by reading the device information previously acquired by the device information acquisition unit prior to the transmission of the response information. good. The acquired device information is used, for example, when determining whether or not the job type and job conditions specified by the user can be executed by the communication target.

(Flow of voice assistant program startup operation)
Next, the sequence diagram of FIG. 27 is a diagram showing the flow of operations when instructing the copy function. In the above step S205, when the user pronounces "copy" in response to the voice feedback "Do you want to copy? Scan?" , the communication control unit 152 of the smart speaker 100 transmits voice data of "copy" to the voice recognition server device 3 . (Step S212). The text conversion unit 156 of the operation speech conversion program of the speech recognition server device 3 converts the speech data of "copy" into text data (step S213).

The interpretation unit 157 of the speech recognition server device 3 interprets the user's uttered phrase indicated by the text data (steps S214 to S216), as described with reference to FIGS. and parameters are sent to the AI assistant server device 4 together with the device ID of the smart speaker 100 . Here, an intent of "COPY_EXECUTE" is generated as an interpretation result and transmitted to the management program (step S217).

Next, in the case of this example, since the user pronounces only "copy", the number of copies and the type of copy such as single-sided or double-sided are unknown (insufficient input). Therefore, the execution determination unit 164 of the management program of the AI assistant server device 4 determines that the required parameters are not satisfied (step S218). As a result, the AI assistant server device 4 inquires about the missing parameter to the smart speaker 100 (steps S219 and S220).

Specifically, the execution determination unit 164 of the AI assistant server device 4 forms response information (Response), for example, "Please input a setting value." The notification unit 165 of the management program of the AI assistant server device 4 transmits this response information to the smart speaker 100 via the speech conversion program of the speech recognition server device 3 (step S220). The feedback unit 153 of the smart speaker 100 outputs a voice message “Please enter the setting value” via the smart speaker 100, and displays a text message “Please enter the setting value” on the touch panel 27. (step S221: input shortage feedback). However, either the voice output or the text display on the touch panel may be omitted.

Further, the notification unit 165 of the management program of the AI assistant server device 4 may transmit state information indicating that the session is continuing when transmitting response information to the operation voice conversion program. The state information is information indicating the state of the session. For example, when a job execution command has not yet been sent to an external device such as the MFP 1 and the job setting process by the user is continuing, the session status information is sent indicating that the Also, when the transmission of the job execution command to the external device is completed, status information indicating that the session has ended is transmitted.

However, the state information is not limited to this, and may be in the form of information indicating the state of the session in more detail. The interpretation unit 157 of the operation voice conversion program of the voice recognition server device 3 determines whether to continue or end the session according to the state information. In other words, when the session is ongoing, the interpretation unit 157 of the operation voice conversion program interprets the job setting conditions as belonging to one job even if the user designates job setting conditions by dividing them into a plurality of utterances. If it is determined that there is, and the session has ended, it can be determined that the job setting conditions included in the user's utterance belong to the new job. As a result, the operation voice conversion program of the voice recognition server device 3 is able to recognize whether the job type and job setting condition instructions belong to the same job, It can be determined whether it belongs to a new job.

Note that the operation voice conversion program of the voice recognition server device 3 may determine that the session has ended when no signal is received from the smart speaker 100 for a predetermined time or longer. Even if the speech recognition server device 3 manages the session, the communication target (MFP 1) may execute the job regardless of the session. In this case, when the MFP 1 to be communicated acquires the job execution command, it overwrites the job conditions currently held by the MFP 1 with the job conditions included in the acquired job execution command. At this time, the job conditions contained in the job execution command may be set by deleting all the job conditions held in the communication target or returning them to the default conditions.

Also, for job conditions that conflict with the job conditions held by the communication target, the job conditions included in the job execution command are overwritten with priority, and for non-conflicting job conditions, the job conditions held in advance by the communication target are maintained. can be Note that when the communication target executes a job, the job conditions included in the job execution command can be deleted, and the default state registered in advance in the communication target can be set.

Next, since there is insufficient input feedback, the user utters, for example, "both sides" to indicate the copy mode (step S222). The communication control unit 152 of the smart speaker 100 transmits the voice data "on both sides" to the voice recognition server device 3, and requests text conversion (step S223). The text conversion unit 156 of the operation speech conversion program of the speech recognition server device 3 converts the speech data of "on both sides" into text data (step S224).

The interpreting unit 157 of the speech recognition server device 3 interprets the user's uttered phrase indicated by the text data (steps S225 to S227), and the intent and parameters that are the result of the interpretation are sent to the smart speaker 100 together with the device ID of AI. It is transmitted to the assistant server device 4 (step S228).

In this example, a parameter of "print side=both sides" is generated as an interpretation result. As a result, the interpreting unit 157 sends the interpretation result including the intent of "COPY_EXECUTE" and the parameter of "print side = both sides" to the AI assistant server device 4 together with the intent and parameters included in the previous utterance. Send.

Next, in the case of this example, since the user only pronounces "copy" and "double-sided", the number of copies and the like are unknown (again lack of essential parameters). In the above description, the operation voice conversion program of the voice recognition server device 3 generates an intent and parameters by integrating the interpretation result of the previous utterance and the interpretation result of the current utterance. It is not limited to this. That is, the management program of the AI assistant server device 4 stores the interpretation result of the previous utterance, and integrates the interpretation result of the previous utterance and the interpretation result of the current utterance by the management program to obtain the intent and parameters. may be generated. In this case, the interpreting unit 157 of the speech recognition server device 3 may transmit only the parameter “print side=both sides” newly acquired by the current speech to the management program of the AI assistant server device 4 .

Which parameter among the plurality of parameters is the essential parameter can be stored in advance in a storage unit such as the HDD 24 of the AI assistant server device 4 . In this case, the execution determination unit 164 of the AI assistant server device 4 determines whether or not all essential parameters have been set using the parameters acquired from the smart speaker 100 based on the information on the essential parameters stored in the storage unit. If the required parameters have not been set, the user can be prompted via the smart speaker 100 to set the required parameters.

Therefore, the execution determination unit 164 of the management program of the AI assistant server device 4 forms response information (Response) asking "How many copies do you want to print?" (step S229). The notification unit 165 of the AI assistant server device 4 transmits this response information to the smart speaker 100 via the operation voice conversion program of the voice recognition server device 3 (steps S230 and S231). The feedback unit 153 of the smart speaker 100 outputs a voice message "How many copies do you want to print?" (step S232: input shortage feedback).

(Number of copies setting operation)
Next, FIG. 28 is a sequence diagram showing the flow of the number of copies setting operation. In step S232 of the sequence diagram of FIG. 27, the input shortage feedback "How many copies do you want to print?" S241). The communication control unit 152 of the smart speaker 100 transmits the voice data of "second copy" to the voice recognition server device 3 and requests text conversion (step S242). The text conversion unit 156 of the operation voice conversion program of the voice recognition server device 3 converts the voice data of "second copy" into text data (step S243).

The interpretation unit 72 of the operation voice conversion program interprets the user's uttered phrase indicated by the text data based on the dictionary information (steps S244 to S246). In this example, the interpretation unit 72 generates a parameter of "number of copies=2 copies" as an interpretation result. The interpreting unit 72 outputs the interpretation result including the intent of "COPY_EXECUTE" and the parameter of "printing side = double-sided, number of copies = 2 copies" together with the intent and parameters included in the previous utterance to the AI assistant server. It is transmitted to the device 4 (step S247).

The execution determination unit 164 of the AI assistant server device 4 receives the parameter "print side = double-sided, number of copies = 2 copies", thereby solving the shortage of the required parameters for the copy job in the required parameter sufficiency determination in step S248. It is judged that it did. For this reason, the execution determination unit 164 forms the response information "I will copy two copies on both sides. Are you sure?". This response information is transmitted to the smart speaker 100 via the notification unit 165 and the voice recognition server device 3 (steps S249 and S250).

Based on the Response included in the response information, the audio feedback unit 55 of the smart speaker 100 indicates that the shortage of the essential parameters has been resolved and the preparation to start copying is complete. Are you sure?” is output as input confirmation feedback voice and text (step S251).

Here, instead of outputting the text data and voice data included in the response information, the text data generated based on the text data stored in the storage unit of the smart speaker 100 based on the information included in the response information is output. It may be output (audio output or display). In response to this input confirmation feedback, the user gives an instruction to change the setting value or an instruction to start copying.

(Setting value change operation)
When instructing to change the setting value, the user performs voice input of the setting value to be changed to the smart speaker 100 (step S252). For example, when changing the number of copies to be printed from 2 to 3, the user inputs "3 copies" by voice. Also, when changing the printing side from double-sided to single-sided, the user performs voice input of "single-sided". Based on this voice input, the AI assistant server device 4 determines whether the required parameters are satisfied, and based on the response information, the smart speaker 100 reads, for example, "(changed content) will be copied. Are you sure?" and input confirmation feedback is performed (steps S253 to S262).

Such a setting value changing operation (steps S252 to S262) is repeated until the user performs voice input (text input, button input, etc.) that allows the setting value for which input confirmation feedback is provided. That is, such a setting value changing operation is repeated until the user voices "yes" to accept the setting value fed back for input confirmation.

(Operation to instruct MFP to execute job)
Next, FIG. 29 is a sequence diagram showing the flow of the job execution instruction operation to the MFP 1. As shown in FIG. When the user inputs a voice response of "yes" by the above-described input confirmation feedback (step S271), it is converted into text by the operation voice conversion program of the voice recognition server device 3 (step S272). When the interpreting unit 157 of the operation voice conversion program of the voice recognition server device 3 recognizes the copy start instruction based on the text data, the intent of "COPY_EXECUTE" includes "printing side=both sides" and "number of copies=2". and parameters are added and sent to the management program (steps S274 to S277).

When the execution determination unit 164 of the management program of the AI assistant server device 4 determines that the interpretation result satisfies the essential parameters (step S278), the interpretation result conversion unit 161 converts the interpretation result into a job command of the MFP 1. (step S279). Execution instruction unit 162 then transmits the converted execution instruction information to MFP 1 (step S280). As a result, the operation of the MFP 1 can be controlled by setting the functions of the MFP 1 such as copying and scanning, the number of copies to be executed such as 1 copy and 2 copies, and the execution mode such as single-sided and double-sided by voice input.

When the MFP 1 is instructed to execute a job, the execution instructing unit 162 of the AI assistant server device 4 sends information to the effect that the MFP 1 has been instructed to execute the job, and the current job execution state (the current job execution state of the MFP 1). state). The notification unit 165 of the AI assistant server device 4 transmits this response information to the smart speaker 100 via the voice recognition server device 3 (steps S281, 282). When the feedback unit 153 of the smart speaker 100 detects the information indicating that the execution of the job has been instructed from the response information, the feedback unit 153 of the smart speaker 100 performs voice feedback and text feedback such as "(the instructed job will be started)" (step S283). ). This allows the user to recognize that the MFP 1 has started executing the instructed job.

Note that if the execution determination unit 164 of the management program of the AI assistant server device 4 determines that the interpretation result satisfies the essential parameters (step S278), the feedback operation to the smart speaker 100 may be omitted.

(Example of information fed back from AI assistant server device)
Next, in the above description, the feedback unit 153 of the smart speaker 100 outputs text data and voice data included in the response information. However, the feedback unit 153 may form text data corresponding to the response information based on the text data stored in the storage unit of the smart speaker 100 and perform feedback output (audio output and text output).

Specifically, as described above with reference to Table 2, intents such as "COPY_PARAMETER_SETTING" for prompting input of job setting values, "COPY_CONFIRM" for prompting confirmation of job setting values, etc. are included in the response information. , and fed back from the AI assistant server device 4 to the smart speaker 100 .

The feedback unit 153 determines feedback to the user according to the intent, parameter, and response included in the response information. The feedback unit 153 may store information corresponding to Table 2 in the storage unit of the smart speaker 100 and refer to it in order to determine the content to be fed back.

In Table 2, the case of copying was explained as an example, but in the case of printing, scanning, and facsimile, as in Table 2, actions are "PARAMETER_SETTING" for prompting input of job setting values and confirming job setting values. "CONFIRM" may be used to prompt. The response information should include at least one of intent and parameters or response.

Further, parameters such as the set value of the printing surface such as double-sided or single-sided, or the number of copies are included in the response information and fed back to the terminal 2 . Furthermore, if the required parameters are missing, a message prompting the input of the missing parameters is included in the signal as a response and fed back to the smart speaker 100 .

(Operation when an error occurs in the MFP)
Next, the operation when a job is executed in MFP 1 or an error occurs will be described. If the MFP 1 executes the job and the job is completed without incident, a completion message (voice and text) such as "The job has been completed. Please be careful not to forget to pick up the manuscript." output via 100.

On the other hand, if a paper shortage error occurs, for example, the AI assistant server device 4 sends to the smart speaker 100, for example, "There is no paper. Please replenish paper and continue or cancel the job." ” and other job confirmation feedback. In response to this confirmation feedback, the user refills the MFP 1 with paper and responds by uttering "continue". The job is restarted in MFP1.

On the other hand, when the user utters "cancel" as confirmation feedback to instruct cancellation of the job, the execution instructing unit 162 of the AI assistant server device 4 instructs the MFP 1 to cancel the job. The MFP 1 cancels the execution of the job after the occurrence of the error. In this case, confirmation feedback is provided via the smart speaker 100 by outputting voice and text, for example, "Job canceled."

FIG. 30 is a sequence diagram showing the flow of such job execution operations and interactive operations when an error occurs. When the smart speaker 100 acquires the job execution result notified from the MFP 1 (step S350) via the AI assistant server device 4 (step S351), it feeds back to the user that the job has been completed normally (step S352). ), or feedback indicating that an error has occurred (step S353).

When receiving the execution result indicating that the job has been completed normally, the feedback unit 153 of the smart speaker 100 outputs a voice message such as "Job completed" (step S352). On the other hand, when an execution result indicating that an error has occurred is received, the feedback unit 153 of the smart speaker 100 outputs, for example, "An error has occurred. Do you want to continue or cancel the job?" , a voice message asking whether to continue the job is output (step S353).

Specifically, when the job is completed, MFP 1 transmits a completion notification to the management program of AI assistant server device 4 . Further, when an error occurs, the MFP 1 transmits error information regarding the error such as the content of the error to the AI assistant server device 4 . In addition, the management program of the AI assistant server device 4 responds to the information acquired from the MFP 1, for example, "Job completed" or "An error has occurred. Do you want to continue or cancel the job? , and transmits it to the smart speaker 100 via the speech recognition server device 3 .

Steps S354 to S361 in FIG. 30 are the operation flow of each unit when the user designates continuation. That is, in response to the inquiry "An error has occurred. Do you want to continue or cancel the job?" ), the voice of "Continue" is converted into text and interpreted by the operation voice conversion program of the soot speaker 100 as described above (steps S355 and S356).

The smart speaker 100 operation voice conversion program generates an interpretation result with the intent as "COPY_JOB_CONTINUE" and transmits it to the management program of the AI assistant server device 4 (step S357). The notification unit 165 of the management program of the AI assistant server device 4 generates response information indicating that the job will be continued (step S358), and transmits it to the smart speaker 100 via the voice recognition server device 3 (step S359). ). Further, the execution instructing unit 162 of the management program of the AI assistant server device 4 instructs the MFP 1 to continue the job (step S361). In addition, the feedback unit 153 of the smart speaker 100 performs input confirmation feedback to "continue the job" (step S360).

On the other hand, steps S362 to S368 in FIG. 30 are the operation flow of each unit when the user designates cancellation of the job. That is, when the user responds "cancel" to the inquiry "An error has occurred. Do you want to continue or cancel the job?" It is transmitted to the voice recognition server device 3 (step S363). The interpretation unit 157 of the operation voice conversion program of the voice recognition server device 3 converts this voice data into text and interprets it (step S364).

The interpretation unit 157 generates an interpretation result with the intent as "COPY_JOB_CANCEL" and transmits it to the management program of the AI assistant server device 4 (step S365). The execution determination unit 164 of the management program of the AI assistant server device 4 generates response information to "cancel the job". The notification unit 165 transmits this response information to the smart speaker 100 (step S366). As a result, the feedback unit 153 of the smart speaker 100 performs input confirmation feedback of "job is canceled" (step S368). Further, the execution instructing unit 54 of the management program of the AI assistant server device 4 instructs the MFP 1 to cancel the job (step S367).

(Adjustment operation of parameters according to device capability of MFP)
Next, an example of executing a job corresponding to the device capability of the MFP 1 by interactive operation will be described with reference to the sequence diagrams of FIGS. 31 and 32. FIG. FIG. 31 is a sequence diagram showing the flow of operations until it is determined whether or not the MFP 1 has the device capabilities required for the job instructed by the user. FIG. 32 is a sequence diagram showing the flow of operations when the job can be executed with the device capabilities of the MFP 1 and when it is difficult to execute the job.

First, in the sequence diagram of FIG. 31, when the user instructs “start” (step S371), the device information acquisition unit 163 of the management program of the AI assistant server device 4 sends the MFP 1, for example, a resolution that can be processed. (Acquisition of Capability: step S372). In response to this inquiry, the MFP 1 responds to the AI assistant server device 4 with a processable resolution such as "200dpi to 600dpi" (step S373). AI assistant server device 4 transmits information indicating the resolution that MFP 1 can process to smart speaker 100 . Thereby, the smart speaker 100 recognizes the resolution that can be processed by the MFP 1 .

Note that the device information acquiring unit 163 of the AI assistant server device 4 may acquire the information on the processing capability from the MFP 1 at a predetermined timing after the communication line with the MFP 1 is established. Further, when the smart speaker 100 and the MFP 1 are associated and registered in the management table of the linking DB 102, information on the processing capability of the MFP 1 may be acquired and registered. Further, the device information acquisition unit 163 of the AI assistant server device 4 may periodically acquire information on the processing capability from the MFP 1 . In addition, the device information acquisition unit 163 detects the processing capacity from the MFP 1 at the timing when the voice recognition server device 3 receives activation of the voice assistant program from the smart speaker 100 or receives a job execution instruction from the smart speaker 100. You can get information.

The acquired processing capacity information may be stored in the storage unit such as the HDD 44 of the AI assistant server device 4, or may be stored in the management table of the linking DB 102 in association with the MFP 1. FIG. Further, the acquisition of the information on the processing capacity may be executed by another program, and the device information acquisition unit 163 may indirectly acquire the information possessed by the MFP 1 by acquiring the information acquired by the other program. . The information acquired by the device information acquisition unit 163 from the MFP 1 is not limited to information on the processing capability, and for example, the type and version of the application installed in the MFP, the printing speed, the file format that can be processed, options including the finisher, etc. Information about the types of jobs that can be executed by the MFP 1 or job setting conditions, such as the connection status of devices, may be acquired.

Next, after the user activates the operation voice processing program of the smart speaker 100, for example, when "activate" is instructed by voice input, the communication control unit 152 of the smart speaker 100 instructs the voice recognition server device 3 to: This "activation" voice data is transmitted. As a result, the speech data of "activation" is converted into text and interpreted, and the result of interpretation is notified to the AI assistant server device 4. FIG. The execution determination unit 164 of the AI assistant server device 4 determines whether essential parameters are satisfied.

At this point, the job type has not been specified, and the required parameters are not satisfied. ” or the like to prompt the user to enter the type of job. Accordingly, the feedback unit 153 of the smart speaker 100 provides voice feedback prompting a job instruction such as "Do you want to copy? Do you want to scan?" (step S374).

Next, when the user utters "scan" (step S375), it is converted into text and interpreted by the operation speech conversion program of the speech recognition server device 3 (steps S376 and S377). Further, the interpretation unit 377 of the speech recognition server device 3 generates an interpretation result with the intent as "SCAN_EXECUTE" and transmits it to the management program of the AI assistant server device 4 (step S378).

The execution determination unit 164 of the AI assistant server device 4 determines whether the required parameters are satisfied based on the received interpretation result (step S379), but the required parameters are insufficient with only the scan instruction. For this reason, the execution determination unit 164 forms response information to "input a setting value" and transmits this to the smart speaker 100 (step S380). Based on this response information, the feedback unit 153 of the smart speaker 100 performs insufficient input feedback such as "Please enter a setting value" (step S381).

Next, when the user utters "addressed to Mr. Tanaka at 1000 dpi" in response to such input insufficient feedback (step S382), this voice data is converted into text and interpreted by the voice recognition server device 3 (step S383). , step S384). The interpretation unit 157 forms an interpretation result with the intent as "SCAN_EXECUTE" and the parameters as "resolution=1000 dpi, destination=Tanaka", and transmits it to the management program of the AI assistant server device 4 (step S385).

The execution determination unit 57 of the management program of the AI assistant server device 4 refers to the parameters included in the interpretation result and the information indicating the processing capability of the MFP 1 acquired in advance by the device information acquisition unit 161 to determine the communication target. (step S386). Then, the execution determination unit 164 forms response information corresponding to the determination result, and the notification unit 165 notifies the smart speaker 100 (step S387). As a result, feedback corresponding to the determination result is provided via the smart speaker 100 (step S388).

Specifically, in the sequence diagram of FIG. 32, if the MFP 1 has a processing capability for a resolution of 1000 dpi, the execution determination unit 164 will say, "Scan at 1000 dpi and send to Mr. Tanaka. ?” is formed. The notification unit 165 then transmits this response information to the smart speaker 100 . As a result, the feedback unit 153 of the smart speaker 100 provides input confirmation feedback such as "Scan at 1000 dpi and send to Mr. Tanaka. Are you sure?" (step S389).

When the user responds "yes" to this input confirmation feedback (step S391), the interpretation unit 157 of the speech recognition server device 3 converts this speech data into text and interprets it (steps S392 and S393). Then, the interpreting unit 157 transmits the interpretation result with the intent as "SCAN_EXECUTE" and the parameters as "resolution=1000, destination=Tanaka" to the management program of the AI assistant server device 4 (step S394).

The interpretation result converter 161 of the management program of the AI assistant server device 4 converts the interpretation result into a job command (step S395). Then, the execution instructing unit 162 of the management program transmits to the MFP 1 a job instruction to "transmit the image scanned at 1000 dpi to Mr. Tanaka" (step S396). As a result, the specified job is executed by MFP1. The execution determination unit 164 of the AI assistant server device 4 forms response information indicating that execution of the instructed job has started, and the notification unit 165 transmits this response information to the smart speaker 100 (step S397). The feedback unit 153 of the smart speaker 100 provides the user with input confirmation feedback of "Start".

On the other hand, if 600 dpi is the limit of the processing capability of the MFP 1, it is difficult for the MFP 1 to execute image processing of 1000 dpi specified by the user. The execution determination unit 164 of the AI assistant server device 4 determines that the image processing specified by the user is difficult for the MFP 1, based on the information indicating the processing capability of the MFP 1 acquired in advance by the device information acquisition unit 161. do. In this case, the execution determination unit 164 forms response information saying "1000 dpi cannot be specified. Scan at 600 dpi and send to Mr. Tanaka. Are you sure?". The notification unit 165 transmits this response information to the smart speaker 100 . As a result, the smart speaker 100 gives confirmation feedback to the user that "1000 dpi cannot be specified. Scan at 600 dpi and send to Tanaka-san. Are you sure?"

More specifically, the execution determination unit 164 of the AI assistant server device 4 selects the function or value closest to the function or processing capacity value instructed by the user within the processing capacity of the MFP 1 . For example, when the processable resolution of MFP 1 is in the range of 200-600 dpi, execution determination unit 164 selects 600 dpi, which is the closest to 1000 dpi specified by the user. Then, the feedback unit 153 of the smart speaker 100 outputs the ability or value selected by the execution determination unit 57 as confirmation feedback.

That is, when execution determination unit 164 determines that it is difficult to execute the job indicated by the intent and parameters included in the interpretation result of the user's instruction, execution determination unit 164 determines whether MFP 1 can execute the job based on the information indicating the processing capability of MFP 1 . Select a setting condition. Then, response information is generated based on the selected setting conditions.

Note that the management program of the AI assistant server device 4 adds a device ID (voice input source device ID). Therefore, the execution determination unit 164 refers to the linking DB 102 to identify the MFP 1 associated with the voice input source device ID, and refers to the processing capability of the identified MFP to determine whether the job can be executed. You may Further, when the user instructs execution of a function that MFP 1 does not have, execution determination unit 164 may transmit response information indicating that the job cannot be executed to smart speaker 100 . Note that the execution determination unit 164 may transmit information about the selected setting condition to the operation voice conversion program of the voice recognition server device 3 .

When the user utters "yes" in response to such confirmation feedback (step S391), the text conversion unit 156 of the speech recognition server device 3 converts it into text and interprets it (steps S392 and S393). Based on the text data, the interpretation unit 157 of the speech recognition server device 3 forms an interpretation result with the intent set to "SCAN_EXECUTE" and the parameters set to "resolution=600, destination=Tanaka". Send to the management program (step S394).

The interpretation result conversion unit 161 of the management program of the AI assistant server device 4 converts the interpretation result into a job execution instruction for the MFP 1 (step S395). The execution instruction unit 162 of the management program of the AI assistant server device 4 transmits a job execution instruction to the MFP 1 (steps S396 to S398). As a result, even if the user's instruction exceeds the processing capacity of the MFP 1 and is difficult to execute, the content of the instruction is automatically changed to an instruction that can be executed by the processing capacity of the MFP 1, and the job instructed by the user is executed. can do.

Here, MFP 1 searches for destination information corresponding to "Tanaka" by referring to the address book stored in the storage unit such as HDD 13 in MFP 1 . Destination information is information such as a mail address and a FAX number. If the destination information corresponding to "Tanaka" exists in the address book, the image data scanned by the MFP 1 is transmitted to that destination.

Even if the destination information is displayed on the operation unit 16 of the MFP 1 to prompt the user to confirm that the destination is correct, and the scanned image data is sent to the destination when the user operates the OK button. good. Also, if there are multiple destinations corresponding to "Tanaka" in the address book, information on all the corresponding destinations may be displayed on the operation unit 16 of the MFP 1 to allow the user to select one.

(Operation to search for print target and print)
Next, the operation of printing a file designated by the user will be described with reference to the sequence diagrams of FIGS. 33 and 34. FIG. FIG. 33 is a sequence diagram showing operations up to searching for a file specified by the user. FIG. 34 is a sequence diagram showing operations up to printing a file specified by the user.

First, in FIG. 33 , after the operation voice processing program of the smart speaker 100 is activated, the user instructs activation of the voice assistant program by voice input, for example. As a result, the communication control unit 152 of the smart speaker 100 transmits voice data of "start" to the voice recognition server device 3 (step S441). The text conversion unit 156 of the speech recognition server device 3 converts the voice data of “activate” into text data, converts them into intent and parameters, and transmits them to the management program of the AI assistant server device 4 .

The execution determination unit 164 of the management program of the AI assistant server device 4 determines whether essential parameters are satisfied. At the time when the voice of "Start" is input, the required parameters such as the type of job, for example, are not satisfied. ' is formed as response information prompting a job instruction. The notification unit 165 transmits this response information to the smart speaker 100 . As a result, the feedback unit 153 of the smart speaker 100 provides voice feedback prompting a job instruction such as "Copy? Scan? Print? Print?" (step S442). .

Next, when the user utters "print" (step S443), this voice is converted into text by the text conversion unit 156 of the operation voice conversion program of the voice recognition server device 3 and interpreted by the interpretation unit 157 (steps S444, step S445). In this case, the interpretation unit 157 forms an interpretation result with the intent as "PRINT_EXECUTE" and transmits it to the management program of the AI assistant server device 4 (step S446).

The execution determination unit of the management program of the AI assistant server device 4 determines whether the essential parameters are satisfied based on the received interpretation result (step S447). In this case, since the required parameters are insufficient with only the “print” instruction, the execution determination unit 164 forms the response information “Please enter the setting value” and It is transmitted to the speaker 100 (step S448). Based on this response information, the feedback unit 153 of the smart speaker 100 performs insufficient input feedback of "Please enter a setting value" (step S449).

Next, when the user utters "print file A" in response to such insufficient input feedback (step S450), this voice data is transmitted to the voice recognition server device 3 (step S451). The voice recognition server device 3 interprets this voice data as text (step S452), forms an interpretation result with the intent as "Print_Execute" and the parameter as "print target = file A", and AI assistant server device 4 (step S453).

The search unit 167 of the management program of the AI assistant server device 4 searches the file A from the management DB 101 based on the parameters included in the interpretation result (step S454).

Note that the search target is not limited to the management DB 101, and may be a database connected to the cloud service device of the voice recognition server device 3 or the AI assistant server device 4 via a network. Moreover, the search unit 167 may search not only for a file name including a character string included in the parameter, but also for a file including the character string included in the parameter in the file data. Further, the search unit 167 may search based on file attributes such as file creation date and time and file creator.

(When only one file A is recorded)
By performing such a search, if only one file A is recorded in the management DB 101, in the sequence diagram of FIG. (step S455). Then, the execution instructing unit 162 of the AI assistant server device 4 transmits the print data of the file A to the MFP 1 (step S456), and instructs printing (step S457). As a result, printing of the print data of file A is executed in MFP 1 .

(When there are two or more files A)
On the other hand, when two or more files A are retrieved from the management DB 101, that is, when a plurality of files are retrieved as a result of the retrieval by the retrieval unit 167, the execution determination unit 164 of the AI assistant server device 4 states, " Multiple files A are registered. Please select a file to print." The notification unit 165 transmits this response information to the smart speaker 100 (step S458). At this time, as information for specifying each file, the date and time of creation of the file, the creator, the thumbnail image, and the like may be included in the response information.

The feedback unit 153 of the smart speaker 100 displays a list of searched files on the touch panel 27 based on the response information (step S459). Further, the feedback unit 153 provides insufficient input feedback such as "A plurality of files A are registered. Please select a file to be printed" (step S460).

When displaying the list of files, in order to facilitate the selection of a desired file from among the searched files, the date and time of creation and the creator of the file may be given as voice feedback or displayed on the touch panel 27. can be Alternatively, thumbnail images may be displayed as a file list to prompt the user to make a selection.

When selecting file A, the file may be selected by uttering information about the file such as the file creation date and time or the creator, or a desired file may be selected from the file list displayed on the touch panel 27. You can select it by operation.

(When file A is not recorded)
Next, when the file A specified by the user is not recorded in the management DB 101 (when the search unit 167 cannot search for the file A), the execution determination unit 164 of the AI assistant server device 4 determines that "File in memory A is not registered" is generated. Further, the execution determination unit 164 acquires list information of files stored in the management DB 101 from the management DB 101 and includes it in the response information. The notification unit 165 transmits such response information to the smart speaker 100 (step S461).

Based on this response information, the feedback unit 153 of the smart speaker 100 provides feedback that "File A is not registered. Please specify the file to be printed" (step S462). Along with this, the feedback unit 153 displays a list of files stored in the management DB 101 on the touch panel 27 based on the file list information added to the response information (step S463). Based on this list of files, the user selects a file to be printed and designates it again.

(Print operation of selected file)
Next, when only one file A exists in the management DB 101 and the user instructs to print the file A, the execution determination unit 164 asks, "Printing of the file A will start. Are you sure?" It generates response information and transmits it to the smart speaker 100 (step S464). The feedback unit 153 of the smart speaker 100 provides confirmation feedback such as "Printing of file A will start. Are you sure?" (step S465).

When the user responds "yes" to this confirmation feedback (step S466), this response voice data is transmitted from the smart speaker 100 to the voice recognition server device 3 (step S467). The text conversion unit 156 of the speech recognition server device 3 converts the voice data into text, and the interpretation unit 157 interprets the textualized voice data (step S468). The interpretation unit 157 forms an interpretation result with the intent set to "PRINT_EXECUTE" and the parameter set to "printing target=file A, file specifying information X", and transmits it to the management program of the AI assistant server device 4 (step S469). . The file specifying information X is information added to specify the file A selected by the user when two or more files A exist in the management DB 101. For example, the file creation date and time or the creator etc.

The execution determination unit 164 of the management program of the AI assistant server device 4 determines whether the essential parameters are satisfied based on the interpretation result (step S470). In this case, since the essential parameters are complete, the interpretation result conversion unit 161 converts the interpretation result into a job command for the MFP 1 (step S471). The execution instruction unit 162 transmits the file A selected by the user to the MFP 1 and instructs execution of printing (step S472). As a result, MFP 1 executes printing of file A desired by the user. When the MFP 1 is instructed to execute printing, the notification unit of the AI assistant server device 4 transmits to the smart speaker 100 response information indicating that printing has started, indicating that printing has started (step S473). ). As a result, the feedback section 153 of the smart speaker 100 provides a feedback "starting", and notifies the user that printing has started (step S474).

In the above description, the case where the search unit 167 of the AI assistant server device 4 searches for a print target and transmits the searched file to the MFP 1 has been described as an example. However, for example, a job command specifying "print target=file A" may be transmitted to the MFP 1, and the MFP 1 may retrieve the file A from the HDD 13, the management DB 101, or the like, and execute printing. If there are two or more retrieved files, a file list may be displayed on the display section of the MFP 1 for the user to select.

(Effect of Second Embodiment)
As is clear from the above description, the voice operation system of the second embodiment installs an operation voice processing program, which is a platform application program, in the smart speaker 100, and this application communicates with the cloud service device. . When the user speaks into the microphone unit 29 provided in the smart speaker 100, the cloud service device analyzes the contents of the user's speech and operates the MFP 1 to perform the operation instructed by the user (voice input operation). .

As a result, complicated operations can be commanded by voice, and operations using a GUI (Graphical User Interface) such as the touch panel 27 can be made unnecessary. Therefore, even a user accustomed to the operation can perform the input operation more quickly and easily. In addition, even the elderly or inexperienced users can quickly and easily perform operations desired by the user, such as complex network settings, advanced job settings, or introduction of new applications, through interactive operation support. be executable.

In addition, since the user's intention is analyzed based on the text of the content uttered by the user, the processing content based on the user's utterance can be determined and processed on the AI assistant server device 4 side.

Finally, each embodiment described above is presented as an example and is not intended to limit the scope of the present invention. Each novel embodiment can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention.

For example, in the description of the first embodiment above, the speech recognition server device 3 generates text data corresponding to the user's utterance, and the AI assistant server device 4 recognizes the user's intention based on the text data. interpreted the operation. However, the mobile terminal 2 may be provided with such a speech recognition function and an interpretation function, and the mobile terminal 2 may interpret the intended operation from the user's utterance. This eliminates the need for the voice recognition server device 3 and the AI assistant server device 4, thereby simplifying the system configuration.

Such each embodiment and modifications of each embodiment are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1 multifunction machine (MFP)
2 mobile terminal 3 voice recognition server device 4 AI assistant server device 5 network 51 acquisition unit 52 communication control unit 53 interpretation result conversion unit 54 execution instruction unit 55 voice feedback unit 56 processing capacity acquisition unit 57 execution determination unit 58 search unit 61 acquisition unit 62 text conversion unit 63 communication control unit 71 acquisition unit 72 interpretation unit 73 communication control unit

JP 2014-203024 A

Claims (11)

An information processing system including an external device and an information processing device,
an acquisition unit included in the information processing device that acquires voice information for operating the external device;
a voice recognition unit that recognizes the voice information;
an extracting unit that extracts instruction information indicating information processing to instruct the external device based on the result of recognition of the speech information by the speech recognition unit;
a conversion unit that converts the instruction information into operation information in an information format interpretable by the external device;
Regarding the information processing device that acquires the voice information by referring to the association information that associates the first identification information that identifies the information processing device with the second identification information that identifies the external device associated with the information processing device An information processing system comprising: an output unit that outputs the operation information to the external device associated by the association information . The speech recognition unit converts the speech information into text information,
The information processing system according to claim 1, wherein the extraction unit extracts the instruction information based on the text information. a processing capability detection unit that detects the information processing capability of the external device;
Correction information for forming correction information for prompting correction of the information processing indicated by the instruction information so that the information processing indicated by the instruction information can be executed by the information processing capability of the detected external device. a forming unit;
3. The information processing system according to claim 1, further comprising a correction output unit that outputs an output prompting correction of the information processing based on the correction information. 4. The information processing system according to claim 3, wherein the information processing device is a smart speaker including a microphone unit that is the acquisition unit and a speaker unit that is the correction output unit. further comprising a deficiency judgment unit that judges the instruction information lacking in the operation of the external device,
The correction information forming unit performs an output prompting the input of the instruction information to be the shortage determined by the shortage determination unit,
The conversion unit converts the instruction information for operating the external device to the operation information when the shortage determination unit determines that all the instruction information required for operating the external device has been input. converted into information
5. The information processing system according to claim 3, wherein the output unit outputs the operation information to the external device. further comprising a print information detection unit for detecting print information to be printed, which is instructed by the instruction information, from the storage unit;
The information processing system according to any one of claims 1 to 5, wherein the output unit outputs the detected print information together with the operation information to the external device. 7. The output unit outputs to the external device, together with the operation information, an e-mail address to which the image information formed by the external device is to be sent. or the information processing system according to item 1. Further comprising a notification unit that notifies when the information processing device that has acquired the voice information is not associated with any external device according to the association information.
The information processing system according to any one of claims 1 to 7, characterized by: an acquisition unit included in a smart speaker that acquires audio information for operating an external device;
a voice recognition unit that recognizes the voice information;
an extracting unit that extracts instruction information indicating information processing to instruct the external device based on the result of recognition of the speech information by the speech recognition unit;
a conversion unit that converts the instruction information into operation information in an information format interpretable by the external device;
With reference to association information that associates first identification information identifying a smart speaker with second identification information identifying an external device associated with the smart speaker, the association information is obtained for the smart speaker from which the audio information has been obtained. and an output unit configured to output the operation information to the external device associated with the information processing apparatus. an acquisition step in which an acquisition unit provided in a smart speaker acquires audio information for operating an external device;
a speech recognition step in which a speech recognition unit recognizes the speech information;
an extracting step in which an extraction unit extracts instruction information indicating information processing to instruct the external device based on a recognition result of the speech information by the speech recognition unit;
a conversion step in which the conversion unit converts the instruction information into operation information in an information format interpretable by the external device;
The output unit refers to association information that associates first identification information that identifies a smart speaker with second identification information that identifies an external device associated with the smart speaker, and acquires the audio information. and an output step of outputting the operation information to the external device associated by the association information . the computer,
an acquisition unit that acquires voice information for operating an external device , which is acquired by a smart speaker ;
a voice recognition unit that recognizes the voice information;
an extracting unit that extracts instruction information indicating information processing to instruct the external device based on the result of recognition of the speech information by the speech recognition unit;
a conversion unit that converts the instruction information into operation information in an information format interpretable by the external device;
With reference to association information that associates first identification information identifying a smart speaker with second identification information identifying an external device associated with the smart speaker, the association information is obtained for the smart speaker from which the audio information has been obtained. an information processing program that causes the external device associated with the above to function as an output unit that outputs the operation information.

Images