JP2020087381A - Information processing system, program, and information processing method - Google Patents

Abstract

To provide an information processing system configured to facilitate difficult-to-understand setting instruction, a program, and an information processing method.SOLUTION: An information processing system including an information processing apparatus and an external device comprises: an acquisition unit which acquires voice information including setting instruction for operating the external device; a voice recognition unit which recognizes the voice information; a notification unit which displays operation information based on a result recognized by the voice recognition unit recognizing the voice information, on a screen of the information processing apparatus; an interpretation unit which interprets settings lacking in the operation information; and an output unit which outputs the operation information to the external device. The notification unit displays options on the settings lacking in the operation information on the screen of the information processing apparatus.SELECTED DRAWING: Figure 13

Description

The present invention relates to an information processing system, a program, and an information processing method.

2. Description of the Related Art Conventionally, there is known an operation method of giving an instruction to an image forming apparatus such as a multifunction peripheral (MFP) by voice. For example, Patent Document 1 discloses an image forming apparatus that can be operated by voice.

However, when adopting the interactive type in the operation method for giving instructions to the external device by voice, simple settings such as “double-sided” and “black and white” are good, but settings that are difficult to call such as “staple position setting” The problem is that it is difficult to give verbal instructions.

The present invention has been made in view of the above, and an object thereof is to provide an information processing system, a program, and an information processing method capable of facilitating a setting instruction that is difficult to call.

In order to solve the above-mentioned problems and achieve the object, the present invention is an information processing system including an information processing device and an external device, and an acquisition unit for acquiring voice information including a setting instruction for operating the external device. A voice recognition unit that recognizes the voice information, a notification unit that notifies operation information based on a recognition result of the voice information by the voice recognition unit on a screen of the information processing device, and a setting that is insufficient in the operation information. And an output unit that outputs the operation information to the external device, and the notification unit notifies the option related to the setting lacking in the operation information on the screen of the information processing device. , Is characterized.

According to the present invention, for example, it is easy to give a setting instruction such as "position setting of staples" which is difficult to call by notifying "choice" and a setting option which is not suitable for verbal instruction. There is an effect that can be.

FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. FIG. 2 is a hardware configuration diagram of the MFP. FIG. 3 is a hardware configuration diagram of the mobile terminal device. FIG. 4 is a hardware configuration diagram of the voice recognition server device. FIG. 5 is a hardware configuration diagram of the AI assistant server device. FIG. 6 is a functional block diagram of the mobile terminal device. FIG. 7 is a functional block diagram of the voice recognition server device. FIG. 8 is a functional block diagram of the AI assistant server device. FIG. 9 is a sequence diagram showing the flow of the overall operation of the voice operation in the voice operation system. FIG. 10 is a diagram showing an example of entity information used for interpreting a voice input by the user. FIG. 11 is a diagram showing entity information registered based on the utterance phrase. FIG. 12 is a diagram showing a flow of an interactive input operation. FIG. 13 is a diagram showing an example of a screen display when the processing shown in FIG. 12 is executed. FIG. 14 is a sequence diagram showing the first half flow of the interactive input operation. FIG. 15 is a sequence diagram showing the latter half flow of the interactive input operation. FIG. 16 is a system configuration diagram of the voice operation system according to the second embodiment. FIG. 17 is a hardware configuration diagram of the smart speaker. FIG. 18 is a hardware configuration diagram of the cloud service device. FIG. 19 is a schematic explanatory diagram showing the overall functions of the cloud. FIG. 20 is a diagram showing an example of the configuration of functional blocks of a smart speaker. FIG. 21 is a diagram showing an example of the configuration of each function of the cloud service. FIG. 22 is a sequence diagram showing the flow of operations at startup. FIG. 23 is a sequence diagram showing the flow of the interactive operation after the activation. FIG. 24 is a sequence diagram showing the flow of the interactive operation after the activation. FIG. 25 is a sequence diagram showing the flow of the interactive operation after the activation. FIG. 26 is a diagram showing an example of a screen display.

Embodiments of an information processing system, a program, and an information processing method will be described in detail below with reference to the accompanying drawings.

(First embodiment)
(System configuration)
FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. As shown in FIG. 1, a voice operation system according to the first embodiment, which is an information processing system, includes a multifunction peripheral (MFP) 1 which is an example of an external device, a mobile terminal device such as a smartphone or a tablet terminal. 2 (an example of an information processing apparatus), a voice recognition server apparatus 3 and an AI (Artificial Intelligence) assistant server apparatus 4 are formed by connecting them to each other via a predetermined network 5 such as a LAN (Local Area Network). ing. However, the external device is not limited to the multifunction peripheral, and may be various electronic devices including office devices such as an electronic blackboard and a projector.

The mobile terminal device 2 receives a voice input from the user for voice-operating the MFP 1. Further, the received operation is fed back to the user by voice or screen display. In addition, the mobile terminal device 2 relays data communication (communication of text data described later) between the voice recognition server device 3 and the AI assistant server device 4. The voice recognition server device 3 analyzes the voice data received from the mobile terminal device 2 and converts it into text data. The voice recognition server device 3 corresponds to the first server device. The AI assistant server device 4 analyzes the text data, converts the text data into a pre-registered user's intention (job execution command of the MFP 1), and transmits it to the mobile terminal device 2.

The AI assistant server device 4 corresponds to the second server device. The MFP 1 executes the job execution instruction transmitted from the mobile terminal device 2. The communication between the mobile terminal device 2 and the MFP 1 may be wireless communication or wired communication. That is, the mobile terminal device 2 may be an operation terminal fixedly connected to the MFP 1.

Further, in this example, two server devices, the voice recognition server device 3 and the AI assistant server device 4, are provided, but each server device 3, 4 may be physically one server device. .. Alternatively, each server device 3, 4 may be realized by a plurality of server devices.

(MFP hardware configuration)
FIG. 2 is a hardware configuration diagram of the MFP 1 provided in the voice operation system. The MFP 1 has a plurality of functions such as a printer function and a scanner function. That is, the MFP 1 has a controller 19, a communication unit 15, an operation unit 16, a scanner engine 17, and a printer engine 18, as shown in FIG.

The controller 19 includes a CPU 10, an ASIC (Application Specific Integrated Circuit) 11, a memory 12, an HDD (Hard Disk Drive) 13, and a timer 14. The CPU 10 to the timer 14 are connected to each other via a bus line so that they can communicate with each other.

The communication unit 15 is connected to the network 5, and acquires a job execution command, such as a scan instruction or a print instruction, which is input by voice using the mobile terminal device 2, as described later.

The operation unit 16 is a so-called touch panel in which a liquid crystal display (LCD) and a touch sensor are integrally formed. When the operator uses the operation unit 16 to give an instruction to execute a desired operation, the operator touches an operation button (software key) displayed on the operation unit 16 to specify the desired operation.

The scanner engine 17 controls the scanner unit to optically read a document. The printer engine 18 controls the image writing unit to print an image on, for example, transfer paper. The CPU 10 centrally controls the image forming apparatus. The ASIC 11 is a so-called large-scale integrated circuit (LSI), and performs various kinds of image processing necessary for an image processed by the scanner engine 17 and the printer engine 18. The scanner engine 17 and the printer engine 18, which are means for executing the job execution command acquired from the mobile terminal device 2, correspond to functional units.

The memory 12 stores various applications executed by the CPU 10 and various data used when executing the applications. The HDD 13 stores image data, various programs, font data, various files, and the like. An SSD (Solid State Drive) may be provided instead of or together with the HDD 13.

(Hardware configuration of mobile terminal)
FIG. 3 is a hardware configuration diagram of the mobile terminal device 2 provided in the voice operation system. As shown in FIG. 3, the mobile terminal device 2 is formed by connecting a CPU 21, a RAM 22, a nonvolatile ROM 23, an interface unit (I/F unit) 24, and a communication unit 25 to each other via a bus line 26. There is. The RAM 22 stores an address book in which electronic mail addresses of users who are destinations of electronic mail and scanned images are stored. Further, the RAM 22 stores files such as image data to be printed.

An operation voice processing program is stored in the ROM 23. The CPU 21 enables the voice input operation of the MFP 1 by executing this operation voice processing program.

A touch panel 27, a speaker unit 28, and a microphone unit 29 are connected to the I/F unit 24. The microphone unit 29 collects (acquires) the input voice of the job execution command for the MFP 1 in addition to the call voice. The input voice is transmitted to the voice recognition server device 3 via the communication unit 25 and converted into text data.

(Hardware configuration of the voice recognition server device)
FIG. 4 is a hardware configuration diagram of the voice recognition server device 3 provided in the voice operation system. As shown in FIG. 4, the voice recognition server device 3 includes a CPU 31, a RAM 32, a ROM 33, an HDD (Hard Disk Drive) 34, an interface unit (I/F unit) 35, and a communication unit 36 via a bus line 37. Is formed by connecting to. A display unit 38 and an operation unit 39 are connected to the I/F unit 35. The HDD 34 stores an operation voice conversion program for converting voice data into text data. The CPU 31 executes the operation voice conversion program to convert the voice data transmitted from the mobile terminal device 2 into text data, and returns the text data to the mobile terminal device 2.

(Hardware configuration of AI assistant server device)
FIG. 5 is a hardware configuration diagram of the AI assistant server device 4 provided in the voice operation system. As shown in FIG. 5, the AI assistant server device 4 is formed by connecting a CPU 41, a RAM 42, a ROM 43, an HDD 44, an interface unit (I/F unit) 45, and a communication unit 46 to each other via a bus line 47. ing. A display unit 48 and an operation unit 49 are connected to the I/F unit 45. The HDD 44 stores an operation interpretation program for interpreting a job instructed by the user. The CPU 41 executes the operation interpretation program to interpret the job instructed by the user from the text data generated (converted) by the voice recognition server device 3. The interpretation result is transmitted to the mobile terminal device 2. The mobile terminal converts the interpretation result into a job command and supplies it to the MFP 1. As a result, the MFP 1 can be operated by the voice input via the mobile terminal device 2.

(Function of mobile terminal)
FIG. 6 is a functional block diagram of the mobile terminal device 2 provided in the voice operation system. The CPU 21 of the mobile terminal device 2 executes the operation voice processing program stored in the ROM 23 to thereby obtain the acquisition unit 51, the communication control unit 52, the feedback unit 55, the processing capacity acquisition unit 56, and the execution unit as illustrated in FIG. 6. It functions as the determination unit 57 and the search unit 58.

The acquisition unit 51 is an example of an acquisition unit, and acquires the user's instruction voice for performing voice operation of the MFP 1 collected via the microphone unit 29. The communication control unit 52 is an example of an output unit, and is between the mobile terminal device 2 and the MFP 1, between the mobile terminal device 2 and the voice recognition server device 3, and between the mobile terminal device 2 and the AI assistant server device 4. Control the communication between each. The interpretation result conversion unit 53 converts the interpretation result of the text data of the user's instruction voice in the AI assistant server device 4 into a job execution command for the MFP 1. The execution instruction unit 54 sends a job execution instruction to the MFP 1 to instruct execution of the job.

The feedback unit 55 is an example of a notifying unit, and for example, to realize an interactive voice input operation, for example, a voice or a screen display that prompts an input that supplements the insufficient data is fed back, or a voice or a screen display that confirms the input. Give feedback, etc. The processing capacity acquisition unit 56 acquires the processing capacity such as the maximum number of pixels that can be processed from the MFP 1. The execution determination unit 57 determines whether the job designated by the user can be executed by the MFP 1 by comparing the capability of the MFP 1 with the job designated by the user. The search unit 58 searches a memory such as the RAM 22 for a destination, a file, or the like, which is voice-instructed by the user.

In this example, the acquisition unit 51 to the search unit 58 are implemented by software, but some or all of these may be implemented by hardware such as an IC (Integrated Circuit). Further, the functions realized by the acquisition unit 51 to the search unit 58 may be realized by the operation voice processing program alone, or by causing another program to execute a part of the processing, or indirectly by using another program. The processing may be executed. For example, the acquisition of information such as the processing capacity of the MFP 1 is executed by another program, and the processing capacity acquisition unit 56 can acquire the information possessed by the MFP 1 indirectly by acquiring the information acquired by the other program. it can.

(Function of the voice recognition server device)
FIG. 7 is a functional block diagram of the voice recognition server device 3 provided in the voice operation system. By executing the operation voice conversion program stored in the HDD 34, the CPU 31 of the voice recognition server device 3 functions as the acquisition unit 61, the text conversion unit 62, and the communication control unit 63 as illustrated in FIG. 7. The acquisition unit 61 acquires the voice data input by the user, which is transmitted from the mobile terminal device 2. The text conversion unit 62 is an example of a voice recognition unit, and converts the voice data input by the user into text data. The communication control unit 63 controls communication of the communication unit 36 so as to receive voice data input by the user, send text data to the mobile terminal device 2, and the like.

In this example, the acquisition unit 61 to the communication control unit 63 are realized by software, but some or all of them may be realized by hardware such as an IC (Integrated Circuit). Further, the functions realized by the acquisition unit 61 to the communication control unit 63 may be realized by a single operation voice conversion program, or may be executed by another program to execute a part of the processing or indirectly by using another program. May be made to perform processing.

(Function of AI assistant server device)
FIG. 8 is a functional block diagram of the AI assistant server device 4 provided in the voice operation system. By executing the operation interpretation program stored in the HDD 44, the CPU 41 of the AI assistant server device 4 functions as the acquisition unit 71, the interpretation unit 72, and the communication control unit 73 as illustrated in FIG. 8. The acquisition unit 71 acquires text data of voice data input by the user, which is transmitted from the mobile terminal device 2. The interpretation unit 72 is an example of an interpretation unit, and interprets an operation instruction from the user based on the text data. The communication control unit 73 controls the communication of the communication unit 46 so as to transmit the interpretation result to the mobile terminal device 2 of the user, receive the text data of the voice data input by the user, and the like.

In this example, the acquisition unit 71 to the communication control unit 73 are realized by software, but some or all of them may be realized by hardware such as an IC (Integrated Circuit). Further, the functions realized by the acquisition unit 71 to the communication control unit 73 may be realized by a single operation interpretation program, or by causing another program to execute a part of the processing, or indirectly by using another program. The processing may be executed.

Further, the operation voice processing program, the operation voice conversion program, and the operation interpretation program are recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) as files in an installable format or an executable format. May be provided. Alternatively, the program may be provided by being recorded in a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray disc (registered trademark), or a semiconductor memory. Further, it may be provided by being installed via a network such as the Internet, or may be provided by being incorporated in a ROM or the like in the device in advance.

(Overall operation of voice input operation)
Next, the overall operation of the voice input operation in the voice operation system of the embodiment will be described. FIG. 9 is a sequence diagram showing the flow of the overall operation of voice operation in the voice operation system. The example of FIG. 9 is an example of performing a voice input operation on the double-sided copy function of the MFP 1 via the mobile terminal device 2. In this case, the user activates the operation voice processing program of the mobile terminal device 2 and utters, for example, “copy on both sides”. The voice of the user is collected by the microphone unit 29 of the mobile terminal device 2 and acquired by the acquisition unit 51 (step S1). The communication control unit 52 of the mobile terminal device 2 transmits the voice data “copy on both sides” to the voice recognition server device 3 and controls the communication unit 25 to make a text conversion request (step S2).

The text conversion unit 62 of the voice recognition server device 3 converts the voice data “copy on both sides” into text data. Then, the communication control unit 63 controls the communication unit 36 so as to transmit the converted text data to the mobile terminal device 2 (step S3). The communication control unit 52 of the mobile terminal device 2 transmits the text data "copy on both sides" to the AI assistant server device 4 (step S4).

In the case of this example, the interpretation unit 72 of the AI assistant server device 4 interprets that the operation requested to be executed by the MFP 1 is “copy (Action:Copy_Execcute)”, and “print side is both sides (print side=double side). )” (step S5). In this way, the interpretation unit 72 generates an interpretation result indicating the type (action) and content (parameter) of the job designated by the user based on the text data. The interpretation result is transmitted to the mobile terminal device 2 via the communication unit 46 by the communication control unit 63 of the AI assistant server device 4 (step S6).

The interpretation result conversion unit 53 of the mobile terminal device 2 converts the interpretation result received from the AI assistant server device 4 into a job command of the MFP 1 (step S7). Table 1 below shows an example of the interpretation result and a job command converted from the interpretation result. Note that the interpretation result conversion unit 53 may store the information corresponding to Table 1 in the storage unit (ROM 23) of the mobile terminal device 2 in order to convert the interpretation result into a job command so that it can be referred to.

In the case of the example in Table 1, "COPY_EXECUTE", "SCAN_EXECUTE", "PRINT_EXECUTE", and "FAX_EXECUTE" are shown as an example of an action. Further, “print surface”, “number of copies” and the like are shown as an example of parameters (Parameter). Note that the parameters include all parameters that can be designated as job setting values.

The interpretation result conversion unit 53 of the mobile terminal device 2 converts the interpretation result of “COPY_EXECUTE” into a job command of the MFP 1 “execute copy job”. Similarly, the interpretation result conversion unit 53 converts the interpretation result of “SCAN_EXECUTE” into a job command of “execute scan job” of the MFP 1. Similarly, the interpretation result conversion unit 53 converts the interpretation result of “PRINT_EXECUTE” into a job command of the MFP 1 that “executes a print job”. Similarly, the interpretation result conversion unit 53 converts the interpretation result of “FAX_EXECUTE” into a job command of the MFP 1 “execute FAX job”.

In addition, the interpretation result conversion unit 53 of the mobile terminal device 2 forms a job command of the MFP 1 that “changes the setting value of the printing surface” when the interpretation result includes the parameter of “printing surface”. Similarly, the interpretation result conversion unit 53 forms a job command of the MFP 1 to “change the set value of the number of copies” when the interpretation result includes the parameter “number of copies”.

That is, the interpretation result conversion unit 53 of the mobile terminal device 2 determines the type of the job to be executed by the MFP 1 based on the information included in the “Action” of the interpretation result, and the value included in “Parameter” is set to the job setting value. Then, the interpretation result is converted into a job command.

The communication control unit 52 of the mobile terminal device 2 controls the communication unit 25 to transmit the job command thus formed to the MFP 1 (step S8). In the case of this example, a job command of “execute copy job (printing side=duplex)” is transmitted to the MFP 1. This causes the MFP 1 to execute double-sided printing.

(Details of Interpretation Operation in AI Assistant Server Device)
The AI storage unit 40 of the HDD 44 of the AI assistant server device 4 stores AI assistant service information for interpreting a job instructed by a user by voice input. The AI assistant service information is configured to include entity information, action information, and intent information. The entity information is information that associates a job parameter with a natural language. A plurality of synonyms can be registered in one parameter. The action information is information indicating the type of job. The intent information is information that associates a user's utterance phrase (natural language) with entity information, and a user's utterance phrase (natural language) with action information. The intent information allows correct interpretation even if the utterance order or nuance of the parameters changes slightly. Further, the indent information makes it possible to generate a response text (interpretation result) based on the input content.

FIG. 10 is a diagram showing an example of entity information used for interpreting a voice input by the user. FIG. 10 is entity information corresponding to the print color (Print Color). In FIG. 10, the characters "Print Color" indicate entity names. Also, in FIG. 10, characters such as “auto_color”, “monochrome”, “color”... In the left column indicate parameter names. In FIG. 10, characters such as “auto_color”, “monochrome, black and white”, “color, full color”... In the right column indicate synonyms.

As can be seen from FIG. 10, as entity information, parameters and synonyms are stored in association with each other. By registering synonyms together with parameters, for example, when instructing copy in monochrome, whether you say "Please copy by black and white" or "Please copy by monochrome", you can set the parameters. Can be possible.

FIG. 11 is a diagram showing entity information registered based on the utterance phrase. FIG. 11A shows an example of a user's utterance phrase, FIG. 11B shows an action name, and FIG. 11C shows entity information. As shown in FIGS. 11A to 11C, by operating the operation unit 49 on the screen displayed on the display unit 48 included in the AI assistant server device 4, the user's utterance is dragged. .. Alternatively, the user's utterance is dragged by operating the operation unit of this device on the screen displayed on the display unit of the device that has accessed the AI assistant server device 4 via the network.

Thereby, the entity information to be associated can be selected. Further, when the value (VALUE) is set in the selected entity information, the parameter entered in the response is changed. For example, when uttering "Please copy by black and white" and the value is "$printColor", "printColor=monochrome" is returned as a return value. On the other hand, if the value is "$printColor.original", "printColor=black and white" is returned as the return value. Here, when the value is “$printColor.original”, the utterance content of the user can be directly returned as a response parameter.

(Interactive operation)
Next, the voice operation system according to the embodiment realizes an interactive system in which the system responds based on the input content from the user. In the voice operation system according to the embodiment, two types of responses, “input shortage feedback” and “input confirmation feedback”, are provided as the response peculiar to the operation of the MFP 1 in addition to the response of the fixed text necessary for the dialogue. , An interactive MFP operation system is realized.

“Input shortage feedback” is a response output when information necessary for executing a job is not available. This is output when the user's input result cannot be heard or when the required parameters are insufficient. That is, for parameters other than the essential parameters, it is not necessary to perform input shortage feedback even when not instructed. In addition to the parameters, a process of confirming a function to be used, such as a copy function or a scan function, may be included.

For example, the function and parameter to be confirmed by the user may be changed according to the type of the external device that the mobile terminal device 2 is connected to for communication. In this case, the processing capacity acquisition unit 56 acquires information indicating the type and function of the external device at a predetermined timing after the communication with the external device is established, and the function and parameter to be confirmed to the user based on the acquired information are set. For example, the feedback unit 55 may make the determination. For example, when the type of the external device is the MFP 1, the user can confirm the functions of the MPF 1 such as copy, print, scan, and FAX, and the MFP 1 of the copy, print, scan, and FAX has the function. For the function only, the user may be asked which function to use.

The “input confirmation feedback” is a response output when the information necessary for executing the job is gathered. That is, the input confirmation feedback is performed when all the required parameters are instructed. The input confirmation feedback is provided to prompt the user to select whether to execute the job with the current setting value or change the setting value. In order to confirm whether to execute the job with the current set values, all parameters (regardless of whether they are mandatory parameters or non-essential parameters) instructed by the user are output by voice, Can be confirmed.

FIG. 12 is a diagram showing a flow of an interactive input operation. FIG. 12 shows an example of an interactive operation between the system and the user, including such feedbacks. The example of FIG. 12 is an example of operating the MFP 1 so that two copies of a monochrome image are copied on both sides and the upper one place is stapled. Also, in this example, the staple position is an essential parameter. Note that the essential parameter is not limited to the staple position, and may include a plurality of parameters such as monochrome, color, and paper size.

13 is a diagram showing an example of a screen display when the process shown in FIG. 12 is executed. That is, the mobile terminal device 2 displays the content uttered by the user (recognition result) and the content fed back from the AI assistant server device 4 (operation information) on the screen of the touch panel 27. Note that, in FIG. 13, the comment displayed in a balloon from the right side of the screen of the touch panel 27 of the mobile terminal device 2 indicates a comment indicating the content that the user uttered to the mobile terminal device 2. Further, in FIG. 13, the comment displayed in a balloon from the left side of the screen of the touch panel 27 of the mobile terminal device 2 is a comment indicating the content fed back from the AI assistant server device 4 to the user. That is, when the mobile terminal device 2 receives the feedback from the AI assistant server device 4, the mobile terminal device 2 provides the feedback to the user by voice output, and at the same time, performs the feedback by the screen display of the touch panel 27. However, the feedback of the audio output may be omitted.

Note that which of the parameters is an indispensable parameter can be stored in advance in the storage unit included in the AI assistant server device 4. Further, which parameter is an essential parameter may be appropriately changed by the user operating the operation unit 49 or accessing the AI assistant server device 4 via the network.

In the example of FIG. 12, the utterances indicated by the diagonal lines are the utterances of the user, the utterances without the hatched lines are the utterances of the system, and the conversations indicated by hatching are the utterances (voices) of the screen of the mobile terminal device 2 or the system. First, when the system outputs a voice message "Do you want to copy? Do you want to scan?", the user pronounces "copy" and gives an instruction to use the copy function. On the system side, in order to request the input of the setting value of "copy" designated by the user, the mobile terminal device 2 outputs a voice message "Please input the setting value."

On the other hand, it is assumed that the user utters "two-sided, two copies, staple". In the case of this example, the staple position is an essential parameter as described above. Therefore, the system side prompts the user to input the staple position, such as "Please specify the staple position". This is an example of "input shortage feedback".

As shown in FIG. 13, by displaying one or more images (icons) showing items that can be set as the staple position on the mobile terminal device 2, the user can be prompted to input the insufficient setting items.

As shown in FIG. 13, an image (icon) may be displayed in addition to the comment, or the comment may be omitted and only the image (icon) may be displayed. Further, images (icons) may be displayed for all the settable items, or only representative items may be displayed as images (icons).

In FIG. 13, as the items that can be set as the staple position, images (icons) indicating four items of “upper one place”, “oblique”, “two left places”, and “upper two places” are displayed. . In addition, the names of the setting items (“upper one place”, “diagonal”, “left two places”, “upper two places”) are displayed in text on each image.

The displayed name corresponds to dictionary information (AI assistant service information shown in FIG. 10) registered in advance in the AI assistant server device 4. With this, the user can specify a desired staple position by speaking the text displayed together with the image (icon).

Although the staple position has been described as an example of the job condition for displaying an image in the above description, the present invention is not limited to this. For example, images can be displayed for arbitrary printing conditions such as aggregation, bookbinding, punch positions, dithering settings, and the like. Further, not only the conditions for copying and printing, but also the setting conditions for other functions such as scanning and FAX can be displayed as an image.

In response to such "input shortage feedback", the user specifies the staple position, such as "at the upper one place". This eliminates the input shortage, so the system side responds with "duplexing, copying in two copies. Staples in one place. Are you sure?", and prompts you to start copying. This is the "input confirmation feedback" that is output when the information necessary to execute the job is gathered.

Then, when the user responds "Yes" to "Input confirmation feedback," "Copy on both sides, copy in two copies. Staple in one place. Are you sure?" Execute job" and execute the job instructed by the user.

(Flow of interactive operation)
14 and 15 are sequence diagrams showing the flow of such an interactive operation. The sequence diagram of FIG. 14 shows the flow of the first half of the interactive operation, and the sequence diagram of FIG. 15 shows the flow of the second half of the interactive operation.

First, when the user activates the operation voice processing program of the mobile terminal device 2 (step S11), the feedback unit 55 provides feedback with a voice and a screen display such as "Do you want to copy or scan?" Is performed (step S12).

The mobile terminal device 2 displays the comment “Do you want to copy? Scan?” on the screen of the touch panel 27 together with the voice feedback in step S12. That is, the mobile terminal device 2 displays the text data stored in advance in the ROM 23 or the like of the mobile terminal device 2.

When the user pronounces "copy" (step S13), the communication control unit 52 of the mobile terminal device 2 transmits the voice data of "copy" to the voice recognition server device 3 to make a text conversion request. (Step S14). The text conversion unit 62 of the voice recognition server device 3 converts the voice data of "copy" into text data, and transmits the text data to the mobile terminal device 2 (step S15).

The mobile terminal device 2 displays the comment of “copy” on the screen of the touch panel 27 at the timing when the mobile terminal device 2 receives the text data from the voice recognition server device 3 in step S15. At this time, the mobile terminal device 2 may feed back "copy" by voice or may omit it.

The acquisition unit 51 of the mobile terminal device 2 acquires this text data. In addition, the communication control unit 52 of the mobile terminal device 2 transmits the acquired text data to the AI assistant server device 4 (step S16). As described with reference to FIGS. 10 and 11, the interpretation unit 72 of the AI assistant server device 4 interprets the action and the parameter based on the utterance phrase of the user indicated by the received text data. In the case of this example, since the user pronounces only "copy", the number of copies and the like becomes unknown (insufficient input).

Therefore, the interpretation unit 72 forms an interpretation result by adding a response (Response) of “please input the setting value” to the action of “Copy_Parameter_Setting” (step S17). The communication control unit 73 of the AI assistant server device 4 transmits the interpretation result to the mobile terminal device 2 (step S18). The feedback unit 55 of the mobile terminal device 2 outputs the voice "Please input the setting value" through the speaker unit 28 and the text "Please input the setting value" on the touch panel 27. Display is performed (step S19: input shortage feedback).

Next, since there is insufficient input feedback, the user speaks, for example, "double-sided, 2-copy, staple" (step S20). The communication control unit 52 of the mobile terminal device 2 transmits the voice data of “two-sided, two sets, staple” to the voice recognition server device 3 and makes a text conversion request (step S21). The text conversion unit 62 of the voice recognition server device 3 converts the voice data of “double-sided, double copy, staple” into text data and transmits the text data to the mobile terminal device 2 (step S22).

The acquisition unit 51 of the mobile terminal device 2 acquires this text data. Further, the communication control unit 52 of the mobile terminal device 2 transmits the acquired text data to the AI assistant server device 4 (step S23). The interpretation unit 72 of the AI assistant server device 4 interprets the action and the parameter based on the utterance phrase of the user indicated by the received text data. In the case of this example, since the user pronounces only “copy” and “double-sided, two copies, staple”, the staple position and the like are unknown (insufficient parameters are insufficient).

Which of the plurality of parameters is an essential parameter can be stored in advance in the storage unit of the AI assist server device 4. In this case, the interpretation unit 72 determines whether or not the parameters acquired from the mobile terminal device 2 set all the essential parameters based on the information of the essential parameters stored in the storage unit. If the setting has not been made, the user can be prompted via the mobile terminal device 2 to set the essential parameters.

Therefore, the interpreting unit 72 of the AI assistant server device 4 says "Please specify the staple position" in the parameter "Copy_Parameter_Setting" and the parameters "printing surface=duplex, number of copies=2". The interpretation result is formed by adding the response (Response) (step S24-1).

At the same time, the interpretation unit 72 of the AI assistant server device 4 sets all the settings that can be selected as the insufficient setting items (for example, when the insufficient setting item is the staple position, “upper one place”, “oblique”, “two left places”). , "Upper two places", etc. are specified in the AI assistant server 4 (step S24-2).

Here, the selectable settings can be determined based on, for example, dictionary information (AI assistant service information shown in FIG. 10) registered in advance in the AI assistant server device 4. As a result, only the setting items that can be interpreted by the AI assistant server device 4 can be specified.

Next, the interpretation unit 72 of the AI assistant server device 4 displays an image image corresponding to the specified setting (from a storage unit such as the HDD 44 in the AI assistant server device 4 or from an external server that can communicate with the AI assistant server device 4). ) Search (step S24-3). For example, when the shortage setting item is the staple position, the staple position is used as a keyword for the search.

Here, there are four items that can be set as the staple position: "upper one place", "oblique", "left two places", and "upper two places". In this case, the image image corresponding to each of the four setting items can be searched. It should be noted that one image in which the above four setting items are collected may be searched for as the staple position.

Next, the communication control unit 73 of the AI assistant server device 4 transmits the interpretation result including the searched image image to the mobile terminal device 2 (step S25). The feedback unit 55 of the mobile terminal device 2 outputs a voice "Please specify the position of the staple" through the speaker unit 28, and at the same time, "Please specify the position of the staple" on the touch panel 27. Is displayed (feedback for insufficient input) (step S26). Further, the feedback unit 55 of the mobile terminal device 2 displays the received image on the screen of the touch panel 27 of the mobile terminal device 2 as described later.

As shown in FIG. 13, the mobile terminal device 2 can prompt the user to input the insufficient setting item by displaying an image (icon) on the screen of the touch panel 27 of the mobile terminal device 2. Here, the input method of the insufficient setting items can be instructed by voice, but may be instructed by touching the screen of the touch panel 27 of the mobile terminal device 2. In this case, the mobile terminal device 2 displays an image image (icon) on the touch panel 27 of the mobile terminal device 2 in a selectable manner, and the AI assistant receives the setting information corresponding to the image (icon) selected by touching the screen. It may be configured to transmit to the server device 4 or the MFP 1.

The mobile terminal device 2 displays a plurality of image images (icons) on the screen of the touch panel 27 so that they can be selected. For example, when the shortage setting item is the stapling position, the mobile terminal device 2 has four images in total, which show the setting items of “upper one place”, “oblique”, “two left places”, and “upper two places”. Each of the images is independently selectable.

When any one of the image images (icons) displayed on the screen of the touch panel 27 is selected, the mobile terminal device 2 selects the set value (or which image image is selected) corresponding to the selected image image (icon). Information for identifying whether or not the voice recognition server device 3 is transmitted to the AI assistant server device 4 (text conversion is not required, so that the voice recognition server device 3 is not used).

That is, the mobile terminal device 2 transmits the input information (voice data) to the voice recognition server device 3 when a voice input is performed by the user, and the input information (selected when the screen operation input is performed. Information indicating the set item) is transmitted to the AI assistant server device 4.

On the screen as shown in FIG. 13, the user specifies the staple position by uttering “at one place on top” or by touching the screen on the touch panel 27 at “one place on top” (step S27). In the case of designation by touching the screen, the communication control unit 52 of the mobile terminal device 2 transmits the text data “upper one place” to the AI assistant server device 4 (step S28). The interpretation unit 72 of the AI assistant server device 4 interprets the action and the parameter based on the utterance phrase of the user indicated by the received text data (step S29). In the case of this example, the user touches the screen of the touch panel 27 at "one place", and the lack of essential parameters for the copy job is resolved. Therefore, the interpretation unit 72 forms an interpretation result by adding parameters such as “print surface=double-sided”, “number of copies=2 copies”, and “stapling position=1 upper position” to the action “Copy_Confirm”. The communication control unit 73 of the AI assistant server device 4 transmits the interpretation result to the mobile terminal device 2 (step S30).

The feedback unit 55 of the mobile terminal device 2 has solved the shortage of essential parameters and is ready to start copying. For example, "duplexing, copying in two copies. Staple in one place. Are you sure?" The text of the feedback is generated based on the Response included in the interpretation result (step S31). Here, the text may be generated by reading all or part of the text data stored in the storage unit of the mobile terminal device 2 and combining them.

It should be noted that the feedback unit 55 may generate the feedback text in other steps as well as in the case where the interpretation result is acquired from the AI assistant server device 4, not limited to step S31. If is included in the interpreted response, it is not necessary to generate feedback text.

Next, the above-mentioned input confirmation feedback is performed (step S32). In response to the input confirmation feedback, the user inputs a voice to instruct to change the setting value or start copying.

Which of the plurality of parameters is an essential parameter can be stored in advance in the storage unit of the AI assist server device 4. In this case, the interpretation unit 72 determines whether or not the parameters acquired from the mobile terminal device 2 set all the essential parameters based on the information of the essential parameters stored in the storage unit. If the setting has not been made, the user can be prompted via the mobile terminal device 2 to set the essential parameters.

As described above, the operation voice processing program makes a comment based on the text data previously stored in the mobile terminal device 2, the text data received from the voice recognition server device 3, and the Response received from the AI assistant server device 4. It is displayed on the screen of the touch panel 27 of the device 2.

Steps S35 to S42 of the sequence diagram of FIG. 15 are sequence diagrams showing the flow of operations when a voice instruction is given to change the set value. In FIG. 15, when the user speaks to change the set value (step S35), the text conversion unit 62 of the voice recognition server device 3 generates the text data of the changed set value, and the mobile terminal device 2 operates. It transmits to the AI assistant server apparatus 4 via (step S36-step S38). The AI assistant server device 4 generates an interpretation result indicating the changed setting value based on the utterance phrase of the user indicated by the received text data (step S39) and transmits it to the mobile terminal device 2 (step S40). .

The feedback unit 55 of the mobile terminal device 2 generates a feedback text based on the Response included in the interpretation result (step S41), for example, "black and white, copy two-sided, both sides. Are you sure?" By performing the above-described input confirmation feedback, it is confirmed whether or not copying may be started with the changed set value (step S42).

Steps S43 to S50 in the sequence diagram of FIG. 15 are the flow of the operation of each unit when the start of copying is instructed. That is, when the user responds "yes" by the input confirmation feedback described above (step S43), it is converted into text and transmitted to the AI assistant server device 4 via the mobile terminal device 2 (steps S44 to S46). When the AI assistant server device 4 recognizes the copy start instruction based on the received text data, the interpretation result of adding the parameters of “printing surface=duplex” and “copies=1 copy” to the action “Copy_Execute” Is formed and transmitted to the mobile terminal device 2 (steps S47 to S48).

The interpretation result conversion unit 53 of the mobile terminal device 2 converts the interpretation result into a job command of the MFP 1 (step S49), and transmits the job command to the MFP 1 (step S50). Thus, the MFP 1 can be copy-controlled by the voice input operation.

After the mobile terminal device 2 sends a job command to the MFP 1, when the linked copy mode is ON in the MFP 1 and the staple mode is ON, the mobile terminal device 2 as the parent device sends to the MFP 1 as the child device. , Request the connection status of peripheral devices such as finishers. When the peripheral device is not connected to the MFP 1, the touch panel 27 of the mobile terminal device 2 displays a message indicating that linked copy cannot be performed.

(Example of information fed back from the AI assistant server device 4)
Table 2 below shows an example of the interpretation result fed back from the AI assistant server device 4 to the mobile terminal device 2.

As shown in Table 2, actions such as “Copy_Parameter_Setting” for prompting the input of the setting value of the job, “Copy_Confirm” for prompting the confirmation of the setting value of the job, “Copy_Execute” for notifying the start of the execution of the job Is included in the interpretation result and fed back to the mobile terminal device 2.

The feedback unit 55 can determine feedback to the user according to the action, parameter, and response included in the interpretation result. The feedback unit 55 may store the information corresponding to Table 2 in the storage unit of the mobile terminal device 2 so as to be referred to in order to determine the content to be fed back. In Table 2, the case of copying has been described as an example, but as with Table 2, print, scan, and FAX are also actions such as “Parameter_Setting” for prompting the input of the setting value of the job and confirmation of the setting value of the job. “Confirm” for prompting may be used.

Further, for example, the setting value of the printing surface such as double-sided or single-sided or the parameter such as the number of copies is included in the interpretation result and fed back to the mobile terminal device 2. Further, when the required parameters are insufficient, a message prompting the user to input the insufficient parameters is included in the interpretation result as a response and fed back to the mobile terminal device 2.

As described above, according to the present embodiment, for example, “calling” is difficult, and by giving an option about settings that are not suitable for verbal instructions, setting instructions such as “staple position setting” that is difficult to call Can be facilitated. Further, the user can empirically learn the “calling” of the setting, and the user can instruct the “calling” from the next time.

In this embodiment, the AI assistant server device 4 searches for the image image corresponding to the specified setting, but the present invention is not limited to this. As another form, the mobile terminal device 2 may search for an image image corresponding to the specified setting.

In this case, the AI assistant server device 4 inquires of the mobile terminal device 2 about the insufficient setting item. That is, in step S25 of FIG. 14, the communication control unit 73 of the AI assistant server device 4 transmits the interpretation result that does not include the image image to the mobile terminal device 2. In this case, the interpretation result of Action:Copy_Parameter_Setting (Parameter::printing side=both sides, number of copies=2 copies, post-processing=stapling, Response: select staple position) is transmitted. At this time, an interpretation result including an instruction for requesting the display of the image image to the mobile terminal device 2 may be transmitted, or when a predetermined Response is received (for example, “select staple position”). The mobile terminal device 2 may determine to display the image image (in response to the reception of the text data).

When the mobile terminal device 2 receives an inquiry from the AI assistant server device 4, the mobile terminal device 2 selects all settings (“upper one place”, “diagonal”, “two left places”) that can be selected as insufficient setting items. “Upper two places” are specified from within the mobile terminal device 2, and an image image corresponding to the specified setting is stored (a storage unit such as the ROM 23 of the mobile terminal device 2 or an external server or the MFP 1 accessible by the mobile terminal device 2). Search) and display the searched image.

In the description of the above embodiment, the voice recognition server device 3 generates text data corresponding to the user's utterance, and the AI assistant server device 4 interprets the operation intended by the user based on the text data. did. However, the mobile terminal device 2 may be provided with such a voice recognition function and an interpretation function, and the mobile terminal device 2 may interpret the intended operation from the user's utterance. As a result, the voice recognition server device 3 and the AI assistant server device 4 can be eliminated, and the system configuration can be simplified.

(Second embodiment)
Next, a second embodiment will be described.

The second embodiment is different from the first embodiment in that a smart speaker is applied instead of the mobile terminal device 2. In the following description of the second embodiment, the description of the same parts as those of the first embodiment will be omitted, and only the parts different from those of the first embodiment will be described.

FIG. 16 is a system configuration diagram of the voice operation system according to the second embodiment. As shown in FIG. 16, the voice operation system according to the second embodiment employs a smart speaker 50 (an example of an information processing device) instead of the mobile terminal device 2 described in FIG. The smart speaker is also called an AI speaker and has an AI assistant function that supports interactive voice operations.

The voice operation system connects the MFP 1, which is an example of an external device, the smart speaker 50 (an example of an information processing device), and the cloud service device 60 to each other via a predetermined network 5 such as a LAN (Local Area Network). It is formed by that. However, the external device is not limited to the multifunction peripheral, and may be various electronic devices including office devices such as an electronic blackboard and a projector.

The smart speaker 50 receives a voice input from the user for voice-operating the MFP 1. The smart speaker 50 is installed near the MFP 1. Further, the smart speaker 50 and the MFP 1 have a one-to-one correspondence. Therefore, the smart speaker 50 basically provides the function to the user operating in front of the MFP 1. However, the present invention is not limited to this, and the smart speaker 50 may correspond to the plurality of MFPs 1 and other electronic devices.

The cloud service device 60 may be physically one server device or may be realized by a plurality of server devices. The cloud service device 60 is a control device in which an operation voice conversion program for converting voice data into text data and further interpreting a user's intention is installed. The cloud service device 60 is a control device in which a management program for managing the MFP 1 is installed. Therefore, the cloud service device 60 exhibits the same functions as the voice recognition server device 3 and the AI assistant server device 4 of the first embodiment.

The operation voice conversion program creates/registers an operation voice dictionary and an operation for the MFP 1. The management program manages the entire system by associating the accounts/devices of the smart speaker 50 and the MFP 1.

(Hardware configuration of smart speaker 50)
FIG. 17 is a hardware configuration diagram of the smart speaker 50 provided in the voice operation system. As shown in FIG. 17, the smart speaker 50 includes a CPU 21, a RAM 22, a ROM 23, an interface unit (I/F unit) 24, a communication unit 25, and a bus line 26, as in the mobile terminal device 2 described in FIG. It is formed by being connected to each other through.

An operation voice processing program is stored in the ROM 23. The CPU 21 enables the voice input operation of the MFP 1 by executing this operation voice processing program.

A touch panel 27, a speaker unit 28, and a microphone unit 29 are connected to the I/F unit 24. The microphone unit 29 collects (acquires) the input voice of the job execution command for the MFP 1 in addition to the call voice. The input voice is transmitted to the cloud service device 60 via the communication unit 25 and converted into text data.

(Hardware configuration of cloud service device 60)
FIG. 18 is a hardware configuration diagram of the cloud service device 60 provided in the voice operation system. Note that, in FIG. 18, the cloud service device 60 is assumed to be physically configured by one server device. As shown in FIG. 18, the cloud service device 60, like the voice recognition server device 3 described in FIG. 4, includes a CPU 31, a RAM 32, a ROM 33, an HDD (Hard Disk Drive) 34, and an interface unit (I/F unit) 35. The communication unit 36 and the communication unit 36 are connected to each other via a bus line 37. A display unit 38 and an operation unit 39 are connected to the I/F unit 35. The HDD 34 stores a voice dictionary for operating the MFP 1 and an operation voice conversion program for creating/registering an operation. Further, the HDD 34 stores a management program for managing the entire system by associating the accounts/devices of the smart speaker 50 and the MFP 1. By executing the operation voice conversion program and the management program, the CPU 31 enables the MFP 1 to be operated based on the voice data transmitted from the smart speaker 50.

(Overall functional configuration)
FIG. 19 is a schematic explanatory diagram of the entire function. FIG. 19 shows main functions of providing a cloud service. Details of main functions and description of functions of the smart speaker 50 shown in FIG. 19 will be described later with reference to FIGS. 20 to 21.

The function of the cloud 100 is realized by one cloud service device 60 or a plurality of cloud service devices 60. These functions are appropriately set in one or a plurality of cloud service devices 60, and may be one cloud service device 60 or a plurality of cloud service devices 60.

The CPU 31 of the cloud service device 60 functions as the operation voice conversion unit 310 by reading the operation voice conversion program of the HDD 34 into the RAM 32 and executing it. The operation voice conversion unit 310 has a function of converting voice data into text data. Further, the operation voice conversion unit 310 has a function of determining whether or not the text data matches the predefined dictionary information. Furthermore, the operation voice conversion unit 310 has a function of converting the text data into a parameter indicating a variable such as an action indicating the user's intention and a job condition when the operation voice converting unit 310 matches.

Further, the CPU 31 of the cloud service device 60 functions as the voice assistant unit 320 by reading the voice assistant program of the HDD 34 into the RAM 32 and executing it. The voice assistant unit 320 has a function of holding dictionary information.

Further, the CPU 31 of the cloud service device 60 functions as the management unit 330 by reading the management program of the HDD 34 into the RAM 32 and executing it. The management unit 330 has a function of converting the job execution instruction into a format that can be interpreted by the MFP 1 based on the action and the parameter, and transmitting the job execution instruction to the registered MFP 1.

As described above, the cloud 100 provides the cloud service 300 with at least the functions of the operation voice conversion unit 310, the voice assistant unit 320, and the management unit 330.

The cloud service 300 stores various kinds of information in a DB based on communication with the MFP 1 and the information processing device. As an example, the management unit 330 manages various information using the management DB 340, the linking DB 350, the device information DB 360, and the like.

The management DB 340 is a database that stores data related to the content provided by the cloud service 300, such as text data, image data, and audio data.

The association DB 350 is a database that stores an external device associated with the information processing device. In this example, the association DB 350 stores the device ID of the smart speaker 50 used as the information processing device and the ID of the external device (MFP 1 in this example) associated with the smart speaker 50 in association with each other. The smart speaker 50 and the external device may be linked one-to-one, but the smart speaker 50 and a plurality of external devices may be linked. That is, the type and number of external devices associated with the device ID are not limited. Further, the method of associating the external device with the smart speaker 50 is not limited to the above method. That is, the configuration may be such that the information for identifying the user such as the user account and the user ID is linked to the external device. In this case, the information for specifying the smart speaker 50 transmitted from the smart speaker 50 to the cloud, such as the device ID, and the information for specifying the user are stored in the linking DB of the cloud 100, and the management unit 330 stores the information. The external device may be specified based on the information specifying the user associated with the device ID. Alternatively, instead of the device ID, information for identifying the user may be transmitted from the smart speaker 50. Further, instead of the information for specifying the user, information for specifying an organization such as a department or a company, or information for specifying a place such as a room or a building, and the external device may be linked. In this case, one or more smart speakers 50 may be associated with one or more external devices.

The device information DB 360 is a database that stores the IDs of the external devices including the MFP 1 and the device information in association with each other.

(Function of smart speaker 50)
FIG. 20 is a diagram showing an example of a functional block configuration of the smart speaker 50. By executing the operation processing program stored in the ROM 23, the CPU 21 of the smart speaker 50 functions as the acquisition unit 211, the communication control unit 212, and the feedback unit 213 as illustrated in FIG.

The acquisition unit 211 acquires the user's instruction voice for voice-operating the MFP 1, which is collected via the microphone unit 29 (see FIG. 3). The acquisition unit 211 may acquire a user operation via the touch panel 27 (see FIG. 3) or a physical switch (not shown). The communication control unit 212 controls communication with the cloud 100. The communication control unit 212 communicates with the cloud 100, transmits the information acquired by the acquisition unit 211 to the cloud 100, and acquires text data, image data, and voice data from the cloud 100. In addition, when transmitting the information acquired by the acquisition unit 211 to the cloud 100, the communication control unit 212 may also transmit a device ID that identifies the smart speaker 50.

The feedback unit 213 feeds back to the user, for example, a voice prompting an input to make up for insufficient data, a voice confirming the input, or the like in order to realize an interactive voice input operation. Further, the feedback unit 213 may provide feedback to the user as a text or an image by controlling the display display on the touch panel 27.

In this example, the acquisition unit 211 to the feedback unit 213 are implemented by software, but some or all of these may be implemented by hardware such as an IC (Integrated Circuit). Further, each function of the acquisition unit 211 to the feedback unit 213 may be realized by a single operation processing program, or may cause another program to execute a part of the processing, or indirectly perform the processing by using another program. It may be executed.

(Details of cloud service functions)
FIG. 21 is a diagram showing an example of the configuration of each function of the cloud service. As shown in FIG. 21, the operation voice conversion unit 310 includes functions such as an acquisition unit 311, a text conversion unit 312, an interpretation unit 313, and an output unit 314. The acquisition unit 311 acquires voice data (voice data input by the user) transmitted from the smart speaker 50. The acquisition unit 311 may also acquire data indicating an operation performed by the user on the touch panel 27 of the smart speaker 50, a physical switch (including a button, etc.), or the like. The text conversion unit 312 includes STT (Speech To Text) that converts voice data (user's voice data input in the smart speaker 50) into text data. The interpretation unit 313 interprets the content of the user's instruction based on the text data converted by the text conversion unit 312. Specifically, the interpretation unit 313 confirms whether or not a word or the like included in the text data converted by the text conversion unit 312 matches the dictionary information provided by the voice assistant unit 320, and the words match. In this case, it is converted into an action indicating a job type and a parameter indicating a variable such as a job condition. Then, the interpretation unit 313 transmits the action and the parameter to the management unit 330 together with, for example, the device ID that specifies the smart speaker 50 that is the acquisition source of the audio data. The output unit 314 includes TTS (Text To Speech) that synthesizes text data into voice data. The output unit 314 controls the communication of the communication unit 36 (see FIG. 4) and transmits data such as text data, voice data, and image data to the smart speaker 50.

In this example, the acquisition unit 311 to the output unit 314 are implemented by software, but some or all of these may be implemented by hardware such as an IC (Integrated Circuit). Further, each function realized by the acquisition unit 311 to the output unit 314 may be realized by the operation voice conversion program alone, or may be caused to cause another program to execute a part of the processing, or indirectly by using another program. May perform processing. In addition, a part or all of the functions of the interpretation unit 313 of the operation voice conversion program may be executed by the voice assistant program. In this case, for example, it is confirmed whether a word included in the text data matches the dictionary information, and if they match, the action indicating the user's intention and the conversion to the parameter indicating the variable such as the job condition are performed. The voice assistant unit 320 does this. The interpreting unit 313 only needs to acquire the action and the parameter from the voice assistant unit 320.

The voice assistant unit 320 includes the function of the providing unit 321 as shown in FIG. The providing unit 321 manages the dictionary information in which the relationship between the text data, the action, and the parameter is defined in advance, and provides the operation voice converting unit 310 with the dictionary information. The voice assistant unit 320 may receive the text data from the operation voice conversion unit 310 and interpret the user's operation instruction from the text data. For example, the voice assistant unit 320 acquires the text data from the interpretation unit 313, confirms whether a word or the like included in the text data matches the dictionary information, and if the words match, the text data is regarded as an action. Convert to a parameter. Then, the action and the parameter are provided to the interpretation unit 313.

In this example, the voice assistant unit 320 (including the providing unit 321) is realized by software, but a part or all of it may be realized by hardware such as an IC (Integrated Circuit). Further, the function of the providing unit 321 or the like may be realized by the voice assistant program alone, or may be caused to execute a part of the processing by another program or indirectly execute the processing by using another program. Good.

As shown in FIG. 21, the management unit 330 includes an acquisition unit 331, an interpretation result conversion unit 332, an execution instruction unit 333, a device information acquisition unit 334, an execution determination unit 335, a notification unit 336, and a DB management. Functions of the unit 337 and the like are included.

The acquisition unit 331 acquires the interpretation result from the interpretation unit 313.

The interpretation result conversion unit 332 converts the interpretation result of the action, the parameter, and the like converted by the operation voice conversion unit 310 into a job execution command that can be interpreted by the MFP 1.

The execution instruction unit 333 gives an instruction to execute the job by transmitting a job execution instruction to the MFP 1. Specifically, the execution instruction unit 333 acquires the device ID of the smart speaker 50 instructed by the user, along with the action and the parameter. The execution instruction unit 333 searches the association DB 350 (see FIG. 19) for the MFP 1 corresponding to the acquired device ID, and sends a job execution command to the MFP 1 obtained by the search.

The device information acquisition unit 334 acquires device information from each registered external device (MFP 1 in this example). For example, the device information acquisition unit 334 acquires information indicating the processing capacity such as the maximum number of pixels that can be processed. Further, the device information acquisition unit 334 has a connection state indicating whether or not a communication connection is established with the MFP 1, a power state indicating whether the power of the MFP 1 is ON/OFF or a sleep mode, and whether or not there is an error. And information about the device status including the type of error, the remaining status of consumables such as paper and toner, the login status of the user, authority information indicating the functions that the logged-in user is permitted to use, etc. are also acquired according to the settings. To do.

When the device information acquisition unit 334 acquires the device information such as the processing capability from the plurality of MFPs 1, the device information acquisition unit 334 associates the device information DB 360 (see FIG. 19) with the information such as the ID that identifies each external device. Manage device information.

The execution determination unit 335 compares the processing capability of the MFP 1 with the job specified by the user (that is, the action and the parameter generated by the operation voice conversion unit 310), so that the job specified by the user is executed by the MFP 1. Determine if it is feasible. When it is determined that the job execution designated by the user can be executed, the job execution instruction is transmitted to the MFP 1. When it is determined that the notification cannot be executed, the notification unit 336 may feed back an error message or the like as response information to the smart speaker 50 via the operation voice conversion unit 310.

The notification unit 336 transmits text data, voice data, image data, and the like to the operation voice conversion unit 310 as a response to the user's job execution instruction. Further, when the parameters indicating the job conditions for executing the job are insufficient, the user is urged to further instruct the parameters by feeding back to the smart speaker 50 via the operation voice conversion unit 310. Here, parameter information may be transmitted as information necessary for confirming the missing parameter, or text data, voice data, image data may be transmitted as information necessary for prompting the user to specify the parameter. You may send it.

The DB management unit 337 manages the management DB 340, the linking DB 350, and the device information DB 360. Specifically, various tables are set, and data is registered, searched, deleted, and updated in various tables. For example, the DB management unit 337 links and associates the device ID of the smart speaker 50 with the ID of the MFP 1 based on the information and the instruction input to the client device of the MFP 1, the smart speaker 50, or the cloud service device 60. It is registered in the business DB 350. The association DB 350 holds information in which the device ID of the smart speaker 50 and the ID of the MFP 1 are associated with each other as table data.

(Flow of interactive operation)
22 to 25 are diagrams showing an example of the overall operation when the user interacts with the system to operate the MFP in the voice operation system. FIG. 22 shows the flow of the operation at the time of startup, and FIGS. 23 to 25 show the flow of the interactive operation after the startup. When interacting with the system to operate, session management of the interaction is required. The session management of the dialogue will be described later. Here, as an example, an operation in the case where a user opens a color image on both sides vertically through the smart speaker 50 and gives an instruction to copy two copies at two places on the staple is shown. In this example, the number of copies (=2 copies) is an essential parameter, but the number of copies is not limited, and a plurality of parameters such as monochrome, color, or paper size may be included in the essential parameters.

First, after the user activates the smart speaker 50 (operation processing program), for example, the user inputs the activation word into the smart speaker 50 by voice (step S1′). Here, the user can start the desired voice assistant program by speaking the activation word for activating the voice assistant program. The smart speaker 50 (communication control unit 212) transmits the voice data of the activation word to the cloud 100 (operation voice conversion unit 310) (step S2′).

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the transmission data from the smart speaker 50, and the operation voice conversion unit 310 (text conversion unit 312) converts the voice data into text, that is, converts it into text data. Process (step S3').

The operation voice converting unit 310 (interpretation unit 313) requests the dictionary information from the voice assistant unit 320 (providing unit 321) and acquires the dictionary information from the voice assistant unit 320 (providing unit 321) (step S4′).

Further, the operation voice conversion unit 310 (interpretation unit 313) performs text interpretation from the acquired dictionary information (step S5').

Then, the operation voice conversion unit 310 (interpretation unit 313) passes the interpretation result to the management unit 330 (step S6′).

Here, the management unit 330 searches the linking DB (step S71), confirms the connection state (step S72), confirms the application state (step S73), acquires device information (step S74), and the like, as necessary. .. The order of these processes may be changed as appropriate. Further, if each process is performed at different timing, the process here may be omitted.

In the search of the association DB (step S71), the management unit 330 (DB management unit 337) retrieves the MFP1 (ID of the MFP1) corresponding to the acquired device ID (ID of the smart speaker 50) from the association DB 350. Search and get. At this time, when the ID of the MFP 1 associated with the device ID is not obtained by the search, the management unit 330 (notification unit 336) determines that the smart speaker 50 is not associated with the communication target, the operation voice conversion unit 310 ( The user is notified via the output unit 314). For example, the management unit 330 (notification unit 336) generates response information including a response “This device is not associated with a device”. Here, the management unit 330 (notification unit 336) may include a method of associating the device with the communication target in the response. Note that step S71 may be performed at any other timing when the device ID is acquired.

In the connection state confirmation (step S72), the management unit 330 confirms the device state of the communication target (MFP1 in this example). For example, the DB management unit 337 confirms by referring to the device information acquired in advance in the device information DB 360. Alternatively, the device information acquisition unit 334 may acquire and confirm the device information from the MFP 1 that is the communication target. Here, the confirmation of the device state is, for example, confirmation as to whether or not communication with the communication target MFP 1 is possible and whether or not the MFP 1 is usable. At this time, if the connection with the MFP 1 associated with the device ID (the confirmation target MFP 1) is not established, or if the MFP 1 cannot be used because it is running, the management unit 330 (notification unit 336) displays the operation voice. The user is notified via the conversion unit 310 (output unit 314). For example, the management unit 330 (notification unit 336) generates and notifies response information including a response that "the device is offline" or "the device is being prepared". Here, the management unit 330 (notification unit 336) may include the countermeasure method in the response. The device state may be confirmed at any other timing when the action, the parameter, and the device ID are acquired from the operation voice conversion unit 310 (interpretation unit 313).

In the application status confirmation (step S73), the management unit 330 confirms the status of the application that executes the function designated by the user on the communication target MFP1. For example, the DB management unit 337 confirms by referring to the device information acquired in advance in the device information DB 360. Alternatively, the device information acquisition unit 334 may acquire and confirm the device information from the MFP 1 as the communication target. Here, the confirmation of the application state is, for example, confirmation of whether or not the application is installed and whether or not the application is in the executable state. If it is determined that the function instructed to execute is copy, and the application relating to the copy is not installed in the MFP 1 associated with the device ID, or the application cannot be used because it is running, the management unit 330 (notification unit 336) notifies the user via the operation voice conversion unit 310 (output unit 314). For example, the management unit 330 (notification unit 336) generates and notifies the response information including the response that "the application is not installed" or "the application is currently unavailable". Here, the management unit 330 (notification unit 336) may include the countermeasure method in the response. The state of the application may be confirmed at any other timing when the action, the parameter, and the device ID are acquired from the operation voice conversion unit 310 (interpretation unit 313).

In the device information acquisition (step S74), the management unit 330 acquires the device information of the communication target (MFP 1 in this example). For example, the DB management unit 337 acquires previously acquired device information in the device information DB 360. Alternatively, the device information acquisition unit 334 may acquire the device information from the communication target MFP 1. The device status acquired here is used, for example, in the communication target MFP 1 when determining whether or not the job type and the job condition instructed by the user can be executed.

When these processes are completed at an arbitrary timing after the start-up, the management unit 330 (execution determination unit 335) makes the essential parameter shortage determination (step S75). In the essential parameter shortage determination, the management unit 330 (execution determination unit 335) determines whether or not all conditions necessary for job execution are complete based on the action and parameter of the interpretation result.

If all the job types and required setting conditions have been specified when instructing to start the voice assistant program, the steps of “input feedback” described below are omitted and the MFP 1 is instructed to execute the operation. May be.

At this stage, the user is not instructed to specify a plurality of actions and parameters of the MFP 1 other than the instruction to start by voice. Therefore, the management unit 330 (execution determination unit 335) specifies the essential parameters. Judge that it does not meet. It is determined that the required parameters are not satisfied even if there is an omission of an indispensable condition when instructing to start the voice assistant program. Therefore, the management unit 330 (notification unit 336) creates response information, and transmits the response information to the smart speaker 50 via the operation voice conversion unit 310 (output unit 314) (steps S76 and S77).

The management unit 330 (DB management unit 337) manages the session of communication with the smart speaker 50 with the management DB 340. When transmitting the response information to the smart speaker 50, the management unit 330 (notification unit 336) can transmit the state information indicating that the session is continuing. Here, the state information is information indicating that the session is continuing. Although description is omitted as appropriate in the procedure in the latter stage, when the cloud 100 makes an inquiry to the smart speaker 50, the cloud 100 transmits the status information to the smart speaker 50.

The response information can include text data, voice data, and image data as contents to be inquired to the user. Here, as an example, the audio data “Do you want to copy? Do you want to scan?” is transmitted. As a result, the smart speaker 50 (feedback unit 213) gives a voice feedback "Do you want to copy? Scan?" (step S78).

Note that the content of the feedback is not limited to this as long as it is a message prompting the user to input the job type or the job setting condition. Further, the feedback to the user may be performed by displaying text or an image on the touch panel as well as the voice output. In that case, text data, image data (display information), etc. are transmitted to the smart speaker 50.

After the step S78, when the user utters "copy" (similarly when "uttered "copy" when instructing to start the voice assistant program), the process proceeds as follows. The voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-1). The smart speaker 50 (communication control unit 212) transmits this "copy" voice data to the cloud 100 (step S2-1). At this time, the smart speaker 50 (communication control unit 212) transmits the device ID identifying the smart speaker 50 to the cloud 100.

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and thereafter, as in steps S3′ to S5′, the operation voice conversion unit 310 performs processing up to text interpretation. I (step S3-1 to step S5-1), and the interpretation result is passed to the management unit 330 (step S6-1). Here, the action of "Copy_Execute" corresponding to "copy" is passed as the interpretation result.

Then, the management unit 330 (execution determination unit 335) again performs the essential parameter shortage determination (step S75-1). In the case of this example, the user speaks only "copy", and the set value such as the number of copies of the essential parameter is unknown.

Therefore, the cloud 100 inquires of the smart speaker 50 about the missing parameters. Specifically, since the setting value is insufficient at this stage, the management unit 330 (notifying unit 336) generates response information including “Please input the setting value”, and the operation voice converting unit 310 (output unit 314). ), the voice data of "Please input the setting value" is transmitted to the smart speaker 50 (step S75-1 to step S77-1). Then, the smart speaker 50 (feedback unit 213) outputs a voice message "Please input the setting value" (step S78-1). In this case as well, in addition to the voice output, a text display such as "Please input the setting value" may be displayed on the touch panel 27.

Next, since there was insufficient input feedback, the user utters, for example, "double-sided, 2-copy, staple". The voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-2). The smart speaker 50 (communication control unit 212) transmits this “double-sided, two-copy, staple” audio data to the cloud 100 (step S2-2). At this time, the smart speaker 50 (communication control unit 212) transmits the device ID identifying the smart speaker 50 to the cloud 100.

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and thereafter, as in steps S3′ to S5′, the operation voice conversion unit 310 performs processing up to text interpretation. I (step S3-2 to step S5-2), and the interpretation result is passed to the management unit 330 (step S6-2).

Here, the operation voice conversion unit 310 (interpretation unit 313) generates a parameter with “Parameter: print surface=double-sided, number of copies=two copies, post-processing=staple” as an interpretation result, and the interpretation result is sent to the management unit 330. hand over.

Then, the management unit 330 (execution determination unit 335) again performs the essential parameter shortage determination (step S75-2).

Specifically, the management unit 330 (DB management unit 337) stores the interpretation result of the previous utterance in the management DB 340 or the like, and the management unit 330 (DB management unit 337) stores the interpretation result of the previous utterance and this time. The actions and parameters are completed by integrating the utterance interpretation results. The management unit 330 (execution determination unit 335) again performs the essential parameter shortage determination based on the integrated interpretation result.

In addition, although the case where the management unit 330 integrates the interpretation result of the previous utterance and the interpretation result of the current utterance has been described above, the present invention is not limited to this. That is, the operation voice conversion unit 310 combines the action and the parameter included in the previous utterance with “Copy_Execute”, and the parameter “Parameter: print side=both sides, number of copies=2 copies, post-processing=staple”. The included interpretation result may be passed to the management unit 330.

In the case of this example, the user has not pronounced the stapling setting, and the staple position has insufficient essential parameters. Therefore, the management unit 330 (execution determination unit 335) determines that the essential parameters are insufficient even at this stage (step S75-2).

Therefore, the cloud 100 further inquires of the smart speaker 50 about the missing parameters. At this time, the management unit 330 can search for an image image for specifying the position of the staple and send the searched image to the smart speaker 50. The procedure for inquiring parameters is the same. In other words, the management unit 330 (notification unit 336) generates feedback information including the response “please specify the position of the staple” and the searched image image, and outputs the feedback information via the operation voice conversion unit 310 (output unit 314). The audio data "Please specify the position of the staple" is transmitted to the smart speaker 50 (step S75-2 to step S77-2). Then, the smart speaker 50 (feedback unit 213) outputs a voice message "Please specify the position of the staple" (step S78-2). In this case also, in addition to the voice output, a text display such as "Please specify the position of the staple" may be displayed on the touch panel 27. At this time, an image image can be displayed.

Next, since there is insufficient input feedback, the user utters, for example, "at one place above" or touches the screen of the smart speaker 50 to specify the staple position. The voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-2). The smart speaker 50 (communication control unit 212) transmits this "at one place" voice data to the cloud 100 (step S2-2). At this time, the smart speaker 50 (communication control unit 212) transmits the device ID identifying the smart speaker 50 to the cloud 100.

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and thereafter, as in steps S3′ to S5′, the operation voice conversion unit 310 performs processing up to text interpretation. I (step S3-2 to step S5-2), and the interpretation result is passed to the management unit 330 (step S6-2).

Here, the operation voice conversion unit 310 (interpretation unit 313) generates a parameter with “staple position=1 upper position” as an interpretation result, and passes the interpretation result to the management unit 330.

Then, the management unit 330 (execution determination unit 335) again performs the essential parameter shortage determination (step S75-3).

Specifically, the management unit 330 (DB management unit 337) integrates the interpretation result of the previous utterance and the interpretation result of the current utterance to complete the action and the parameter. That is, in this example, the action of “Copy_Execute” and the parameter of “Parameter: print surface=double-sided, number of copies=two copies, post-processing=staple, staple position=1 upper position” are completed. The management unit 330 (execution determination unit 335) again performs the essential parameter shortage determination based on the integrated interpretation result. In the case of this example, the user utters “upper one place”, and the lack of essential parameters for the copy job is resolved.

In this example, subsequently, the management unit 330 (notification unit 336) responds with information such as "copy two copies on both sides. Staple at one place. Are you sure?" to provide input confirmation feedback. To the smart speaker 50 via the operation voice conversion unit 310 (output unit 314), and the voice data of "2 copies on both sides. Staple at 1 place. Are you sure?" is sent to the smart speaker 50. (Step S75-3 to step S77-3). Then, the smart speaker 50 (feedback unit 213) outputs a voice saying "Copy two copies on both sides. Staple at one place. Are you sure?" (step S78-3). In this case as well, in addition to the voice output, a text display such as “Copy 2 copies on both sides. Staple at 1 place. Are you sure?” may be displayed on the touch panel 27. Here, instead of outputting the text data and voice data included in the response information, the output information is generated by combining the text data stored in the storage unit of the smart speaker 50 based on the information included in the response information. May be.

Then, in response to the input confirmation feedback, the user speaks to the smart speaker 50 to change the setting value or start copying. When the user utters the contents to change the set value, the smart speaker 50 transmits voice data to the cloud 100 to change the uttered set value, the cloud 100 changes the set value, and the smart speaker changes. The fact that the setting value has been changed is feedbacked by voice in 50. The voice feedback confirms whether or not the copying can be started with the changed setting values, such as "Copy with XX setting. Are you sure?".

After that, when the user speaks the content of changing the set value, this procedure is repeated. Therefore, after the voice output saying "Copy 2 copies on both sides. Staple at 1 place. Are you sure?" Only once), the procedure is repeated.

If the user responds with "yes" to instruct to start copying, the n-th procedure shown in FIG. 25 is performed. That is, the voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-n). The smart speaker 50 (communication control unit 212) transmits this "yes" voice data to the cloud 100 (step S2-n). At this time, the smart speaker 50 (communication control unit 212) transmits the device ID identifying the smart speaker 50 to the cloud 100.

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and thereafter, as in steps S3′ to S5′, the operation voice conversion unit 310 performs processing up to text interpretation. I (step S3-n to step S5-n), and the interpretation result is passed to the management unit 330 (step S6-n).

Here, when the operation voice conversion unit 310 (interpretation unit 313) recognizes the copy start instruction, it passes the interpretation result to the management unit 330, and the management unit 330 (execution determination unit 335) determines that the final confirmation is OK ( Step S75-n).

As a result, the management unit 330 (interpretation result conversion unit 332) converts the interpretation result into a job command for the MFP 1 (step S76). Then, the management unit 330 (execution instruction unit 333) transmits the converted execution instruction information to the MFP 1 (step S8). Thus, the MFP 1 can be copy-controlled by the voice input operation.

FIG. 26 is a front view showing a screen displayed on the display unit of the smart speaker 50. As shown in FIG. 26, the screen displayed on the display unit of the smart speaker 50 is the same as the screen displayed on the mobile terminal device 2 shown in FIG.

The process of utterance and feedback to the smart speaker 50 is the same as the process shown in the first embodiment. Schematically, the smart speaker 50 outputs the content uttered by the user and the response information received from the cloud service device 60 (operation voice conversion program). The response information is information including at least one of text data, voice data, and image data.

Note that, in FIG. 26, the comment displayed in a balloon from the right side of the screen of the touch panel 27 of the smart speaker 50 is a comment indicating the content that the user uttered to the smart speaker 50. Further, in FIG. 26, the comment displayed in a balloon from the left side of the screen of the touch panel 27 of the smart speaker 50 is a comment indicating the content of voice feedback from the cloud service device 60 in response to the user's utterance. That is, when the smart speaker 50 receives the feedback information from the cloud service device 60, the smart speaker 50 provides the feedback to the user by voice output and also the feedback by the screen display. However, the feedback of the audio output may be omitted.

Explaining with reference to FIGS. 22 to 25, the comment “Do you want to copy? Do you want to scan?” is displayed on the screen of the touch panel 27 of the smart speaker 50 together with the voice feedback in step S78.

The operation voice processing program of the smart speaker 50 may generate text to be displayed based on the response information from the cloud service device 60 or may display text data stored in advance in the ROM 23 of the smart speaker 50. Is also good. Further, the text data and voice data included in the response information may be displayed as they are.

The operation voice processing program of the smart speaker 50 receives the comment of “copy”, which is obtained by converting the voice data into text data by the cloud service device 60 (operation voice conversion program) as response information, and receives the touch panel 27 of the smart speaker 50. Can be displayed on the screen.

The cloud service device 60 (operation voice conversion program) can transmit the response information at any timing. For example, the cloud service device 60 (operation voice conversion program) may generate response information of “copy” at the timing of conversion to text data and transmit the response information to the smart speaker 50 (in this case, “copy”). Only displayed).

Further, the cloud service device 60 (management program) also generates the response information of “copy” at the timing of generating the response information of “please enter the setting value”, and the smart speaker is generated through the operation voice conversion program. It may be transmitted to the device 50 (in this case, "copy" and "please input the set value" are displayed on the touch panel 27 of the smart speaker 50 almost at the same time.

Note that the operation voice conversion program may transmit information necessary for generating response information of “copy” when transmitting the intent “Copy_Execute” to the management program as the interpretation result.

In addition, the response information is created by the operation voice conversion program, and when the management program sends the response information “Please input the setting value” to the smart speaker 50 via the operation voice conversion program, the response information is also “copy” together. The response information “” may be transmitted to the smart speaker 50.

The operation voice processing program of the smart speaker 50 displays the comment "Please input the setting value" on the screen of the touch panel 27 of the smart speaker 50 together with the voice feedback in step S78-1 of FIG. That is, the comment is displayed based on the response information received from the cloud service device 60 (management program).

The operation voice processing program of the smart speaker 50 receives the comment “two-sided, two copies, staple”, which the cloud service device 60 (operation voice conversion program) converts voice data into text data and displays it on the screen. be able to. The display method is the same as that when "copy" is spoken.

The operation voice processing program of the smart speaker 50 selects each of a plurality of image images on the screen of the touch panel 27 of the smart speaker 50 with the comment of "Please specify the position of the staple" and the voice feedback of step S26 of FIG. Display as possible. For example, when the shortage setting item is the stapling position, the mobile terminal device 2 has four images in total, which show the setting items of “upper one place”, “oblique”, “two left places”, and “upper two places”. Each of the images is independently selectable.

As described above, the smart speaker 50 causes the comment to be displayed on the screen of the touch panel 27 of the smart speaker 50 based on the text data previously stored in the smart speaker 50, the text data received from the cloud service device 60, or the response information.

Here, a case where the specific example described in the first embodiment is applied to this embodiment will be described.

The cloud service device 60 (management program) makes an inquiry to the smart speaker 50 when there is an insufficient setting item (eg, staple position). The smart speaker 50 gives a feedback to the user in a voice/comment display. At this time, feedback to the user is performed by the following procedure.

The cloud service device 60 (management program) sets all the settings that can be selected as the insufficient setting items (for example, when the insufficient setting item is the staple position, “upper one place”, “diagonal”, “two left places”, “upper two”). Etc.) from within the cloud service device 60. Here, the selectable setting can be determined based on, for example, dictionary information registered in advance in the management program. This makes it possible to specify only the setting items that can be interpreted by the management program.

Next, the cloud service device 60 (management program) searches for an image image corresponding to the specified setting (from a storage unit such as the HDD 34 in the cloud service device 60 or from an external server that can communicate with the cloud service device 60). .. For example, when the shortage setting item is the staple position, the staple position is used as a keyword for the search. Here, there are four items that can be set as the staple position: "upper one place", "oblique", "left two places", and "upper two places". In this case, the image image corresponding to each of the four setting items can be searched.

It should be noted that one image in which the above four setting items are collected may be searched for as the staple position.

Next, the cloud service device 60 (management program) transmits the searched image image to the smart speaker 50 via the operation voice conversion program. The smart speaker 50 displays the received image on the screen as described later.

As shown in FIG. 26, when the user speaks “double-sided, dual-copy, staple”, it is converted into text and interpreted by the operation voice conversion program. At this time, the cloud service device 60 (management program) determines whether the required parameters are satisfied. The essential parameter is, for example, the number of copies. However, when a predetermined job condition is designated by the user, the item of the essential parameter can be changed. For example, when the user instructs stapling, the staple position can be set as an essential parameter item in addition to the number of copies.

Here, when the staple position is set as an essential parameter, the cloud service device 60 (management program) determines that the essential parameter is not satisfied, and inquires of the smart speaker 50 about the insufficient setting item. At this time, as shown in FIG. 26, by displaying one or more images showing items that can be set as the staple position on the touch panel 27 of the smart speaker 50, the user can be prompted to input the insufficient setting items. .. As shown in FIG. 26, an image may be displayed in addition to the comment, or the comment may be omitted and only the image may be displayed. Further, images may be displayed for all the settable items, or only representative items may be displayed as images.

In FIG. 26, as the items that can be set as the staple position, an image showing each of the four items of “upper one place”, “oblique”, “two left places”, and “upper two places” is displayed. In addition, the name of each setting item (“upper one place”, “diagonal”, “left two places”, “upper two places”) is displayed in text on each image. The displayed name corresponds to dictionary information registered in advance in the voice assistant program. This allows the user to specify a desired staple position by speaking the text displayed with the image.

Although the staple position has been described as an example of the job condition for displaying an image in the above description, the present invention is not limited to this. For example, images can be displayed for arbitrary printing conditions such as aggregation, bookbinding, punch positions, dithering settings, and the like. Further, not only the conditions for copying and printing, but also the setting conditions for other functions such as scanning and FAX can be displayed as an image.

As described above, according to the present embodiment, for example, “calling” is difficult, and by giving an option about settings that are not suitable for verbal instructions, setting instructions such as “staple position setting” that is difficult to call Can be facilitated. Further, the user can empirically learn the “calling” of the setting, and the user can instruct the “calling” from the next time.

In the present embodiment, the cloud service device 60 (management program) searches for an image image corresponding to the specified setting, but the present invention is not limited to this.

For example, the smart speaker 50 may search for an image image corresponding to the specified setting. In this case, the cloud service device 60 (management program) inquires of the smart speaker 50 about the insufficient setting item. The cloud service device 60 (management program) transmits the response information “Response: Please select the position of the staple”. At this time, the cloud service device 60 (management program) may transmit response information including an instruction for requesting the smart speaker 50 to display an image. In addition, when the smart speaker 50 receives a predetermined response (for example, in response to receiving the text data “Please select the staple position”), the smart speaker 50 determines to display the image. You may.

When an inquiry is received, the smart speaker 50 identifies from the smart speaker 50 all settings that can be selected as insufficient setting items (“upper one place”, “diagonal”, “left two places”, “upper two places”). Then, the image image corresponding to the specified setting is searched (from the storage unit such as the ROM 23 of the smart speaker 50 or the external server or the MFP 1 accessible by the smart speaker 50), and the searched image image is displayed. The method of specifying the selectable items and the method of searching the image image are the same as the method of searching the image image corresponding to the setting specified by the cloud service device 60 (management program).

The smart speaker 50 can prompt the user to input the insufficient setting item by displaying the image image on the screen. Here, the input method of the insufficient setting items can be instructed by voice, but the method may be performed by a method other than voice, for example, by touching the screen.

In this case, the image images are displayed on the touch panel 27 of the smart speaker 50 in a selectable manner, and when the selection is received by touching the screen, the setting information corresponding to the selected image is transmitted to the cloud service device 60. May be.

The smart speaker 50 displays a plurality of image images on the screen of the touch panel 27 so that they can be selected. For example, when the shortage setting item is the staple position, a total of four image images indicating “one place”, “diagonal”, “two places on the left”, and “two places on the top” are set independently. And selectably display.

When any one of the image images is selected, the smart speaker 50 converts the setting value corresponding to the selected image image (or the information specifying which image image is selected) into the operation voice conversion of the cloud service device 60. Send to the program.

Lastly, each of the above-described embodiments is presented as an example, and is not intended to limit the scope of the present invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention.

Such respective embodiments and modifications of the respective embodiments are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

In the above embodiment, the image forming apparatus of the present invention is described as an example in which the image forming apparatus is applied to a multifunction machine having at least two functions of a copy function, a printer function, a scanner function, and a facsimile function. The present invention can be applied to any image forming apparatus such as a printer, a scanner device, and a facsimile device.

1 External Device 2, 50 Information Processing Device 51, 211 Acquisition Unit 52, 212 Output Unit 55, 213 Notification Unit 62 Speech Recognition Unit 72 Interpretation Unit

JP, 2014-203024, A

Claims (8)

In an information processing system including an information processing device and an external device,
An acquisition unit for acquiring audio information including a setting instruction for operating the external device,
A voice recognition unit for recognizing the voice information,
A notification unit for notifying operation information based on the recognition result of the voice information by the voice recognition unit on the screen of the information processing device,
An interpreting unit that interprets a setting that is insufficient in the operation information,
An output unit that outputs the operation information to the external device,
Equipped with
The informing unit informs a screen of the information processing device of options related to a setting that is insufficient in the operation information,
An information processing system characterized by the above. The notification unit displays an image showing items that can be set as the options,
The information processing system according to claim 1, wherein: The notification unit displays a name indicating an item that can be set as the option,
The information processing system according to claim 1, wherein: The notification unit displays an image and a name indicating an item that can be set as the option,
The information processing system according to claim 1, wherein: Of the items that can be set as the options, the notification unit displays an image only for typical items, and displays a name for other items that can be set.
The information processing system according to claim 1, wherein: The notifying unit is configured to selectably display an image showing items that can be set as the options,
The interpretation unit interprets a setting that is insufficient in the operation information including the settable item related to the selected image,
The information processing system according to claim 1, wherein: A computer that controls the information processing device
An acquisition unit that acquires audio information including a setting instruction for operating an external device,
A notification unit that notifies operation information based on the recognition result of the voice information by the voice recognition unit that recognizes the voice information on the screen of the information processing device,
An output unit that outputs the operation information to the external device,
Function as
The informing unit informs a screen of the information processing device of options related to a setting that is insufficient in the operation information,
A program characterized by that. An information processing method in an information processing system including an information processing device and an external device,
An acquisition step of acquiring voice information including a setting instruction for operating the external device;
A voice recognition step of recognizing the voice information,
A notification step of notifying operation information based on the recognition result of the voice information by the voice recognition step on the screen of the information processing device,
An interpreting step of interpreting a setting that is insufficient in the operation information,
An output step of outputting the operation information to the external device,
Including,
The informing step informs the options related to the setting lacking in the operation information on the screen of the information processing device,
An information processing method characterized by the above.

Images