JP7151428B2 - Information processing system, program and information processing method - Google Patents

Description

The present invention relates to an information processing system, a program, and an information processing method.

2. Description of the Related Art Conventionally, there is known an operation method of giving instructions to an image forming apparatus such as a multifunction peripheral (MFP) by voice. For example, Patent Document 1 discloses an image forming apparatus that can be operated by voice.

However, in the case of adopting an interactive operation method for giving instructions to an external device by voice, there is a problem that the more complicated the setting is, the more time it takes to execute a job.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an information processing system, a program, and an information processing method that enable intuitive feedback for setting confirmation and job setting. .

In order to solve the above-described problems and achieve the object, the present invention provides an information processing system including an information processing device and an external device, an acquisition unit that acquires voice information including setting instructions for operating the external device. a voice recognition unit for recognizing the voice information; a reporting unit for reporting operation information based on the result of recognition of the voice information by the voice recognition unit on the screen of the information processing device; and an output unit for outputting to, the reporting unit displays a finished image showing a finished image based on the setting related to the operation information on the screen of the information processing device, and there is a predetermined operation on the finished image. In this case, the corresponding setting is stored together with the finished image .

According to the present invention, by displaying a finished image showing a finished image based on settings, it is possible to intuitively perform feedback for setting confirmation and job setting.

FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. FIG. 2 is a hardware configuration diagram of the MFP. FIG. 3 is a hardware configuration diagram of the mobile terminal device. FIG. 4 is a hardware configuration diagram of the speech recognition server device. FIG. 5 is a hardware configuration diagram of the AI assistant server device. FIG. 6 is a functional block diagram of the mobile terminal device. FIG. 7 is a functional block diagram of the speech recognition server device. FIG. 8 is a functional block diagram of the AI assistant server device. FIG. 9 is a sequence diagram showing the overall operation flow of voice manipulation in the voice manipulation system. FIG. 10 is a diagram showing an example of entity information used to interpret the user's input speech. FIG. 11 is a diagram showing entity information registered based on uttered phrases. FIG. 12 is a diagram showing the flow of interactive input operations. FIG. 13 is a diagram showing an example of screen display when the process shown in FIG. 12 is executed. FIG. 14 is a sequence diagram showing the flow of the first half of the interactive input operation. FIG. 15 is a sequence diagram showing the latter half of the interactive input operation. FIG. 16 is a diagram exemplifying how stamps are stored. FIG. 17 is a diagram showing a modification of the stamp. FIG. 18 is a system configuration diagram of a voice operation system according to the second embodiment. FIG. 19 is a hardware configuration diagram of a smart speaker. FIG. 20 is a hardware configuration diagram of a cloud service device. FIG. 21 is a schematic explanatory diagram showing the overall functions of the cloud. FIG. 22 is a diagram illustrating an example of a functional block configuration of a smart speaker. FIG. 23 is a diagram showing an example of the configuration of each function of the cloud service. FIG. 24 is a sequence diagram showing the flow of operations at startup. FIG. 25 is a sequence diagram showing the flow of interactive operations after activation. FIG. 26 is a sequence diagram showing the flow of interactive operations after activation. FIG. 27 is a sequence diagram showing the flow of interactive operations after activation. FIG. 28 is a diagram showing an example of screen display.

Exemplary embodiments of an information processing system, a program, and an information processing method will be described in detail below with reference to the accompanying drawings.

(First embodiment)
(System configuration)
FIG. 1 is a system configuration diagram of a voice operation system according to the first embodiment. As shown in FIG. 1, the voice operation system of the first embodiment, which is an information processing system, includes a multifunction peripheral (MFP) 1, which is an example of an external device, a mobile terminal device such as a smartphone or a tablet terminal. 2 (an example of an information processing device), a voice recognition server device 3 and an AI (artificial intelligence) assistant server device 4 are formed by interconnecting them via a predetermined network 5 such as a LAN (Local Area Network). ing. However, the external device is not limited to the multifunction machine, and may be various electronic devices including office equipment such as an electronic blackboard and a projector.

The mobile terminal device 2 accepts voice input from the user for operating the MFP 1 by voice. Also, the received operation is fed back to the user by voice or screen display. The mobile terminal device 2 also relays data communication (communication of text data to be described later) between the voice recognition server device 3 and the AI assistant server device 4 . The speech recognition server device 3 analyzes the speech data received from the mobile terminal device 2 and converts it into text data. Note that the voice recognition server device 3 corresponds to a first server device. The AI assistant server device 4 analyzes the text data, converts it into the pre-registered user's intention (job execution command of the MFP 1 ), and transmits it to the portable terminal device 2 .

Note that the AI assistant server device 4 corresponds to a second server device. MFP 1 executes a job execution command transmitted from portable terminal device 2 . Communication between the mobile terminal device 2 and the MFP 1 may be wireless communication or wired communication. In other words, the mobile terminal device 2 may be an operation terminal fixedly connected to the MFP 1 .

Also, in this example, two server devices, the voice recognition server device 3 and the AI assistant server device 4, are provided, but each of the server devices 3 and 4 may be physically one server device. . Alternatively, each of the server devices 3 and 4 may be further implemented by a plurality of server devices.

(MFP hardware configuration)
FIG. 2 is a hardware configuration diagram of the MFP 1 provided in the voice operation system. The MFP 1 has multiple functions such as a printer function and a scanner function. That is, the MFP 1 has a controller 19, a communication section 15, an operation section 16, a scanner engine 17 and a printer engine 18, as shown in FIG.

The controller 19 has a CPU 10 , an ASIC (Application Specific Integrated Circuit) 11 , a memory 12 , a HDD (Hard Disk Drive) 13 and a timer 14 . The CPU 10 to timer 14 are interconnected so as to be communicable via a bus line.

The communication unit 15 is connected to the network 5, and acquires a job execution command, such as a scan instruction or a print instruction, input by voice using the portable terminal device 2, as described later.

The operation unit 16 is a so-called touch panel in which a liquid crystal display (LCD) and a touch sensor are integrally formed. When the operator uses the operation unit 16 to issue a command to execute a desired operation, the operator touches an operation button (software key) displayed on the operation unit 16 to specify the desired operation.

The scanner engine 17 controls the scanner unit to optically read the document. The printer engine 18 controls the image writing unit to print an image on transfer paper or the like. The CPU 10 comprehensively controls the image forming apparatus. The ASIC 11 is a so-called large-scale integrated circuit (LSI: Large-Scale Integration), and performs various image processing necessary for images processed by the scanner engine 17 and the printer engine 18 . The scanner engine 17 and printer engine 18, which are means for executing a job execution command acquired from the mobile terminal device 2, correspond to functional units.

The memory 12 stores various applications executed by the CPU 10 and various data used when executing the applications. The HDD 13 stores image data, various programs, font data, various files, and the like. An SSD (Solid State Drive) may be provided instead of the HDD 13 or together with the HDD 13 .

(Hardware configuration of mobile terminal)
FIG. 3 is a hardware configuration diagram of the mobile terminal device 2 provided in the voice operation system. The mobile terminal device 2 is formed by connecting a CPU 21, a RAM 22, a nonvolatile ROM 23, an interface section (I/F section) 24, and a communication section 25 to each other via a bus line 26 as shown in FIG. there is The RAM 22 stores an address book that stores e-mail addresses of users to whom e-mails, scanned images, etc. are sent. The RAM 22 also stores files and the like that are image data to be printed.

The ROM 23 stores an operation voice processing program. The CPU 21 enables the voice input operation of the MFP 1 by executing this operation voice processing program.

A touch panel 27 , a speaker section 28 and a microphone section 29 are connected to the I/F section 24 . The microphone unit 29 collects (obtains) the input voice of a job execution command to the MFP 1 in addition to the call voice. The input speech is transmitted to the speech recognition server device 3 via the communication unit 25 and converted into text data.

(Hardware configuration of speech recognition server device)
FIG. 4 is a hardware configuration diagram of the voice recognition server device 3 provided in the voice operation system. As shown in FIG. 4, the speech recognition server device 3 connects a CPU 31, a RAM 32, a ROM 33, a HDD (Hard Disk Drive) 34, an interface section (I/F section) 35, and a communication section 36 to each other via a bus line 37. formed by connecting to A display unit 38 and an operation unit 39 are connected to the I/F unit 35 . The HDD 34 stores an operation voice conversion program for converting voice data into text data. By executing the operation voice conversion program, the CPU 31 converts the voice data transmitted from the mobile terminal device 2 into text data and sends the text data back to the mobile terminal device 2 .

(Hardware configuration of AI assistant server device)
FIG. 5 is a hardware configuration diagram of the AI assistant server device 4 provided in the voice operation system. The AI assistant server device 4 is formed by interconnecting a CPU 41, a RAM 42, a ROM 43, an HDD 44, an interface section (I/F section) 45, and a communication section 46 via a bus line 47, as shown in FIG. ing. A display unit 48 and an operation unit 49 are connected to the I/F unit 45 . The HDD 44 stores an operation interpretation program for interpreting a job designated by the user. By executing this operation interpretation program, the CPU 41 interprets the job instructed by the user from the text data generated (converted) by the speech recognition server device 3 . This interpretation result is transmitted to the mobile terminal device 2 . The mobile terminal converts the interpreted result into a job command and supplies it to the MFP1. As a result, the MFP 1 can be operated by voice input via the mobile terminal device 2 .

(mobile terminal functions)
FIG. 6 is a functional block diagram of the mobile terminal device 2 provided in the voice operation system. By executing the operation voice processing program stored in the ROM 23, the CPU 21 of the mobile terminal device 2 executes an acquisition unit 51, a communication control unit 52, a feedback unit 55, a processing capacity acquisition unit 56, and an execution program as shown in FIG. It functions as a determination unit 57 and a search unit 58 .

Acquisition unit 51 is an example of an acquisition unit, and acquires a user's instruction voice for voice operation of MFP 1 collected via microphone unit 29 . The communication control unit 52 is an example of an output unit, and is between the mobile terminal device 2 and the MFP 1, between the mobile terminal device 2 and the voice recognition server device 3, and between the mobile terminal device 2 and the AI assistant server device 4. each control the communication between The interpretation result conversion unit 53 converts the interpretation result of the text data of the instruction voice of the user in the AI assistant server device 4 into a job execution command for the MFP 1 . Execution instructing unit 54 transmits a job execution instruction to MFP 1 to instruct execution of the job.

The feedback unit 55 is an example of a notification unit, and in order to realize an interactive voice input operation, for example, feeds back voice or screen display prompting input to compensate for missing data, or voice or screen display for confirming input. feedback, etc. The processing capacity acquisition unit 56 acquires the processing capacity, such as the maximum number of pixels that can be processed, from the MFP 1 . Execution determination unit 57 determines whether or not MFP 1 can execute the job specified by the user by comparing the capability of MFP 1 with the job specified by the user. The search unit 58 searches a memory such as the RAM 22 for a destination, a file, or the like specified by the user's voice.

In this example, the acquiring unit 51 to the searching unit 58 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 51 to the search unit 58 may be realized by the operation voice processing program alone, may be executed by another program to execute a part of the processing, or may be indirectly obtained by using another program. Processing may be executed. For example, acquisition of information such as the processing capacity of MFP 1 is executed by another program, and the processing capacity acquiring unit 56 acquires the information acquired by the other program, thereby indirectly acquiring information possessed by MFP 1 . can.

(Function of voice recognition server device)
FIG. 7 is a functional block diagram of the voice recognition server device 3 provided in the voice operation system. The CPU 31 of the speech recognition server device 3 functions as an acquisition unit 61, a text conversion unit 62, and a communication control unit 63 as shown in FIG. 7 by executing the operation speech conversion program stored in the HDD . The acquisition unit 61 acquires voice data input by the user, which is transmitted from the mobile terminal device 2 . The text conversion unit 62 is an example of a speech recognition unit, and converts voice data input by the user into text data. The communication control unit 63 performs communication control of the communication unit 36 so as to receive voice data input by the user, transmit text data to the mobile terminal device 2, and the like.

In this example, the acquisition unit 61 to the communication control unit 63 are realized by software, but some or all of them may be realized by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 61 to the communication control unit 63 may be realized by the operation voice conversion program alone, by causing another program to execute part of the processing, or indirectly by using another program. You can let the .

(Function of AI assistant server device)
FIG. 8 is a functional block diagram of the AI assistant server device 4 provided in the voice operation system. The CPU 41 of the AI assistant server device 4 functions as an acquisition unit 71, an interpretation unit 72, and a communication control unit 73 as shown in FIG. 8 by executing the operation interpretation program stored in the HDD 44. FIG. The acquisition unit 71 acquires text data of voice data input by the user, which is transmitted from the mobile terminal device 2 . The interpretation unit 72 interprets an operation instruction from the user based on the text data. The communication control unit 73 controls communication of the communication unit 46 so as to transmit interpretation results to the mobile terminal device 2 of the user and receive text data of voice data input by the user.

In this example, the acquisition unit 71 to the communication control unit 73 are implemented by software, but some or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, the functions realized by the acquisition unit 71 to the communication control unit 73 may be realized by the operation interpretation program alone, or may be executed indirectly by using another program to execute a part of the processing by another program. Processing may be executed.

In addition, the operation voice processing program, the operation voice conversion program and the operation interpretation program are recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) as files in an installable format or an executable format. may be provided as Alternatively, it may be provided by being recorded on a computer-readable recording medium such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), or a semiconductor memory. Moreover, it may be provided in a form of being installed via a network such as the Internet, or may be provided by being incorporated in advance in a ROM or the like in the device.

(Overall operation of voice input operation)
Next, the overall voice input operation in the voice operation system according to the embodiment will be described. FIG. 9 is a sequence diagram showing the overall operation flow of voice manipulation in the voice manipulation system. The example in FIG. 9 is an example in which the double-sided copy function of the MFP 1 is operated by voice input via the mobile terminal device 2 . In this case, the user activates the operation voice processing program of the mobile terminal device 2 and utters, for example, "copy on both sides". The user's voice is collected by the microphone unit 29 of the mobile terminal device 2 and acquired by the acquisition unit 51 (step S1). The communication control section 52 of the mobile terminal device 2 transmits the voice data of "copy on both sides" to the voice recognition server device 3, and controls the communication section 25 to request text conversion (step S2).

The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "copy on both sides" into text data. Then, the communication control section 63 controls the communication section 36 to transmit the converted text data to the mobile terminal device 2 (step S3). The communication control unit 52 of the mobile terminal device 2 transmits the text data "copy on both sides" to the AI assistant server device 4 (step S4).

In this example, the interpreting unit 72 of the AI assistant server device 4 interprets the action requested to the MFP 1 to be "copy (Action: Copy_Execute)", and interprets that "the printing surface is double-sided (printing surface = double-sided )” (step S5). In this manner, the interpretation unit 72 generates an interpretation result indicating the type (action) and content (parameters) of the job specified by the user, based on the text data. This interpretation result is transmitted to the portable terminal device 2 via the communication section 46 by the communication control section 63 of the AI assistant server device 4 (step S6).

Interpretation result conversion unit 53 of mobile terminal device 2 converts the interpretation result received from AI assistant server device 4 into a job command for MFP 1 (step S7). Table 1 below shows an example of an interpretation result and a job command converted from the interpretation result. Note that the interpretation result conversion unit 53 may store information corresponding to Table 1 in the storage unit (ROM 23) of the mobile terminal device 2 and refer to it in order to convert the interpretation result into a job command.

In the example of Table 1, "COPY_EXECUTE", "SCAN_EXECUTE", "PRINT_EXECUTE" and "FAX_EXECUTE" are shown as examples of actions. Also, "printing surface", "number of copies", etc. are shown as examples of parameters. The parameters include all parameters that can be specified as job setting values.

The interpretation result conversion unit 53 of the portable terminal device 2 converts the interpretation result of "COPY_EXECUTE" into a job command of the MFP 1 "execute copy job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "SCAN_EXECUTE" into a job command of MFP 1, "execute scan job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "PRINT_EXECUTE" into a job command of MFP 1 "execute print job". Similarly, interpretation result conversion unit 53 converts the interpretation result of "FAX_EXECUTE" into a job command of MFP 1, "execute FAX job".

Further, when the interpretation result includes the parameter of "printing surface", the interpretation result conversion unit 53 of the portable terminal device 2 forms a job command for the MFP 1 to "change the setting value of the printing surface". Similarly, when the interpretation result includes a parameter of "number of copies", the interpretation result conversion unit 53 forms a job command for the MFP 1 to "change the set value of the number of copies".

That is, the interpretation result conversion unit 53 of the portable terminal device 2 determines the type of job to be executed by the MFP 1 based on the information included in the "Action" of the interpretation result, and converts the value included in the "Parameter" to the job setting value. Then, the interpreted result is converted into a job instruction.

The communication control unit 52 of the mobile terminal device 2 controls the communication unit 25 so as to transmit the job command thus formed to the MFP 1 (step S8). In the case of this example, a job command of "execute copy job (print side=both sides)" is transmitted to MFP1. As a result, MFP 1 executes double-sided printing.

(Details of the interpretation operation in the AI assistant server device)
The AI storage unit 40 of the HDD 44 of the AI assistant server device 4 stores AI assistant service information for interpreting a job instructed by a user's voice input. This AI assistant service information includes entity information, action information, and intent information. Entity information is information that associates job parameters with natural language. Multiple synonyms can be registered for one parameter. The action information is information indicating the type of job. The intent information is information that associates a user's uttered phrase (natural language) with entity information, and a user's uttered phrase (natural language) with action information, respectively. The intent information enables correct interpretation even if the utterance order or nuance of the parameters changes slightly. Also, the indentation information makes it possible to generate a response text (interpretation result) based on the input content.

FIG. 10 is a diagram showing an example of entity information used to interpret the user's input speech. FIG. 10 shows entity information corresponding to Print Color. In FIG. 10, the characters "Print Color" indicate the entity name. In FIG. 10, characters such as “auto_color”, “monochrome”, “color”, etc. in the left column indicate parameter names. In FIG. 10, characters such as “auto_color”, “monochrome, black and white”, “color, full color”, etc. in the right column indicate synonyms.

As can be seen from FIG. 10, as entity information, parameters and synonyms are associated and stored. By registering synonyms along with parameters, for example, when instructing to copy in monochrome, even if you say "Please copy by black and white" or "Please copy by monochrome", you can set parameters can be made possible.

FIG. 11 is a diagram showing entity information registered based on uttered phrases. FIG. 11(a) shows an example of a user's uttered phrase, FIG. 11(b) shows an action name, and FIG. 11(c) shows entity information. As shown in FIGS. 11A to 11C, the user's utterance is dragged by operating the operation unit 49 on the screen displayed on the display unit 48 provided in the AI assistant server device 4. . Alternatively, the user's utterance is dragged by operating the operation unit of the device on the screen displayed on the display unit of the device that accessed the AI assistant server device 4 via the network.

This makes it possible to select entity information to be associated. Also, if a value (VALUE) is set in the selected entity information, the parameters entered in the response are changed. For example, when "Please copy by black and white" is uttered and the value is "$printColor", "printColor=monochrome" is returned as the return value. On the other hand, if the value is "$printColor.original", "printColor=black and white" is returned as the return value. Here, if the value is "$printColor.original", the content of the user's utterance can be returned as it is as a parameter of the response.

(interactive behavior)
Next, the voice operation system of the embodiment realizes an interactive system in which the system responds based on the contents of input from the user. In the voice operation system according to the embodiment, in addition to responding with fixed sentences necessary for dialogue, two types of responses, "input insufficient feedback" and "input confirmation feedback", are given as responses specific to the operation of the MFP 1. , realizes an interactive MFP operation system.

"Insufficient input feedback" is a response that is output when the information required to execute a job is not complete. This is output when the user's input result cannot be heard, or when a required parameter is missing. In other words, there is no need to provide insufficient input feedback for parameters other than essential parameters, even if they are not indicated. In addition to the parameters, processing for confirming a function to be used, such as a copy function or a scan function, may also be included.

For example, the functions and parameters to be confirmed by the user may be changed according to the type of external device with which the mobile terminal device 2 is in communication connection. In this case, the processing capacity acquisition unit 56 acquires information indicating the type and function of the external device at a predetermined timing after communication with the external device is established, and based on the acquired information, determines the functions and parameters to be confirmed by the user. For example, the feedback unit 55 may decide. For example, when the type of the external device is MFP1, the user can confirm the functions of copy, print, scan, FAX, etc. that MPF1 has. As for the functions, the user may be asked which function to use.

"Input confirmation feedback" is a response that is output when the information necessary for executing a job is complete. That is, input confirmation feedback is provided when indicated for all required parameters. Also, the input confirmation feedback is provided to prompt the user to select whether to execute the job with the current set values or to change the set values. In addition, in order to confirm whether or not to execute the job with the current setting values, all parameters specified by the user (regardless of whether they are required parameters or parameters other than required parameters) can be output by voice. can be verified.

FIG. 12 is a diagram showing the flow of interactive input operations. An example of interactive operation between the system and the user, including each of these feedbacks, is shown in FIG. The example of FIG. 12 is an example of manipulating the MFP 1 so as to copy color images on both sides (upper/lower opening) and staple them at the upper two positions. Also, in this example, the staple position is an essential parameter. Note that the essential parameter is not limited to the staple position, and may include a plurality of parameters such as monochrome, color, or paper size.

FIG. 13 is a diagram showing an example of screen display when the processing shown in FIG. 12 is executed. In other words, the mobile terminal device 2 displays the content of the user's speech (recognition result) and the content (operation information) fed back from the AI assistant server device 4 on the screen of the touch panel 27 . Note that the comment balloon-displayed from the right side of the screen of the touch panel 27 of the mobile terminal device 2 in FIG. 13, the comments and images displayed in balloons from the left side of the screen of the touch panel 27 of the mobile terminal device 2 are comments or images (stamps) indicating the content of feedback from the AI assistant server device 4 in response to the user's utterance. is. In other words, when the mobile terminal device 2 receives feedback from the AI assistant server device 4 , the mobile terminal device 2 outputs the feedback to the user through voice output, and at the same time, also provides the feedback through the screen display of the touch panel 27 . However, feedback of audio output may be omitted.

It should be noted that which parameters among the parameters are essential parameters can be stored in advance in the storage unit of the AI assistant server device 4 . Further, the user may be able to appropriately change which parameter is the essential parameter by operating the operation unit 49 or by accessing the AI assistant server device 4 via the network.

In the example of FIG. 12, the utterances indicated by hatching are user utterances, the utterances without slanting lines are the system utterances, and the hatched conversations are the screen of the mobile terminal device 2 or the system utterances (voice). First, when the system outputs a voice saying "Do you want to copy? Do you want to scan?", the user pronounces "copy" to instruct the use of the copy function. On the system side, in order to request input of the setting value of "copy" specified by the user, the portable terminal device 2 outputs a voice message "Please input the setting value."

In response to this, it is assumed that the user utters "color, double-sided, open top and bottom, two places on top of staple". In this example, the staple position is an essential parameter as described above. Therefore, the system side determines that the input is not insufficient, displays an image (stamp) indicating a finished image on the mobile terminal device 2, and prompts an instruction to start copying. This is the "input confirmation feedback" that is output when the information necessary to execute the job is complete.

Then, when the user replies "OK", the system responds "Job will be executed" and executes the job instructed by the user.

(Interactive operation flow)
14 and 15 are sequence diagrams showing the flow of such interactive operations. The sequence diagram of FIG. 14 shows the flow of the first half of the interactive operation, and the sequence diagram of FIG. 15 shows the flow of the second half of the interactive operation.

First, when the user activates the operation voice processing program of the mobile terminal device 2 (step S11), the feedback unit 55 provides voice and screen display feedback of "Do you want to copy? Do you want to scan?" (step S12).

The portable terminal device 2 displays the comment "Do you want to copy? Do you want to scan?" That is, the mobile terminal device 2 displays the text data stored in advance in the ROM 23 of the mobile terminal device 2 or the like.

When the user pronounces "copy" (step S13), the communication control unit 52 of the mobile terminal device 2 transmits voice data of "copy" to the voice recognition server device 3 and requests text conversion. (Step S14). The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "copy" into text data, and transmits the text data to the portable terminal device 2 (step S15).

The mobile terminal device 2 displays the comment "copy" on the screen of the touch panel 27 at the timing when the mobile terminal device 2 receives the text data from the voice recognition server device 3 in step S15. At this time, the mobile terminal device 2 may give voice feedback of "copy" or may omit it.

The acquisition unit 51 of the mobile terminal device 2 acquires this text data. Further, the communication control unit 52 of the mobile terminal device 2 transmits the acquired text data to the AI assistant server device 4 (step S16). The interpretation unit 72 of the AI assistant server device 4 interprets the action and parameters based on the user's uttered phrase indicated by the received text data, as described with reference to FIGS. 10 and 11 . In this example, the user pronounces only "copy", so the stapling position and the like are unknown (insufficient input).

Therefore, the interpretation unit 72 forms an interpretation result by adding a response of "Please input a setting value" to the action of "Copy_Parameter_Setting" (step S17). The communication control unit 73 of the AI assistant server device 4 transmits this interpretation result to the mobile terminal device 2 (step S18). The feedback unit 55 of the mobile terminal device 2 outputs a voice message “Please input the setting value” through the speaker unit 28, and also outputs a text message “Please input the setting value” to the touch panel 27. Display is performed (step S19: input shortage feedback).

Next, since there is insufficient input feedback, the user utters, for example, "color, double-sided, top and bottom open, two places on staple" (step S20). The communication control unit 52 of the mobile terminal device 2 transmits voice data of "color, double-sided, top and bottom opening, two staples on top" to the voice recognition server device 3, and requests text conversion (step S21). . The text conversion unit 62 of the speech recognition server device 3 converts the speech data of "color, double-sided, top and bottom opening, staple top 2 places" into text data, and transmits the text data to the portable terminal device 2 (step S22).

The acquisition unit 51 of the mobile terminal device 2 acquires this text data. Further, the communication control unit 52 of the mobile terminal device 2 transmits the acquired text data to the AI assistant server device 4 (step S23). The interpretation unit 72 of the AI assistant server device 4 interprets the action and parameters based on the user's uttered phrase indicated by the received text data (step S24). In this example, the user pronounces "copy" and "color, double-sided, open top and bottom, two places on top of staple", so the lack of essential parameters for the copy job is resolved. For this reason, the interpreting unit 72 adds the parameters "color/monochrome = color", "printing side = double-sided", "opening direction = top and bottom opening", and "stapling position = top two places" to the action "Copy_Confirm". to form an interpretation result with the addition of The communication control unit 73 of the AI assistant server device 4 transmits this interpretation result to the mobile terminal device 2 (step S25).

The feedback unit 55 of the mobile terminal device 2 resolves the shortage of essential parameters and is ready to start copying. 13, an image (stamp) is generated based on the analysis result as shown in FIG. 13 (step S27).

If it is determined that the required parameters are insufficient, the user can be urged to set the missing required parameters by further displaying a comment and outputting a voice.

The feedback unit 55 of the mobile terminal device 2 refers to "Parameter" when the "Action" of the feedback analysis result is "Copy_confirm". Then, the feedback unit 55 of the mobile terminal device 2 searches for an image (stamp) indicating a finished image that matches the value of “Parameter”, and causes the touch panel 27 to display the searched image (stamp).

Here, if a plurality of setting values are set in "Parameter", an image (stamp) that satisfies all the setting values is searched. For example, the ROM 23 of the mobile terminal device 2 stores images (stamps) in association with setting values.

Here, FIG. 16 is a diagram exemplifying a stamp storage mode. For example, as shown in FIG. 16, images (stamps) can be stored as table data. If there is no image (stamp) that satisfies all set values, the closest image (stamp) may be displayed.

Note that the table data shown in FIG. 16 may be stored not in the mobile terminal device 2 but in an external device that the mobile terminal device 2 can access. For example, it may be stored in the MFP 1 with which the mobile terminal device 2 is communicating, or may be stored in a server connected via the network 5 . In this case, the operation voice processing program can access the MFP 1 or the server, transmit the setting values included in the analysis result, and obtain an image (stamp) that satisfies the setting values as a response from the MFP 1 or the server.

In the above description, a case where the mobile terminal device 2 searches for an image (stamp) based on feedback from the AI assistant server device 4 has been described. You can search for In this case, an image ( stamp) to the mobile terminal device 2 . The mobile terminal device 2 displays the received image (stamp) on the touch panel 27 .

At this time, the AI assistant server device 4 searches for an image (stamp) showing a finished image that matches the value of "Parameter" included in the analysis result. The AI assistant server device 4 can search and acquire an image (stamp) indicating an image by querying the HDD 44 of the AI assistant server device 4 or a server accessible by the AI assistant server device 4 .

Here, FIG. 17 is a diagram showing a modification of the stamp. For example, when the analysis result fed back from the AI assistant server device 4 is "Action: Copy_confirm" and "Parameter: Print surface = double-sided, number of copies = 2", the image (stamp) of FIG. can be displayed as In the example shown in FIG. 17, the number "2" indicating the number of copies is displayed together with the finish image.

In addition to displaying an image (stamp) indicating the finished image, the mobile terminal device 2 may provide voice feedback such as "I will copy two copies on both sides. Are you sure?", or the voice feedback may be omitted. You can Also, in addition to displaying the finished image, a comment "I will copy two copies on both sides. Are you sure?" may be displayed, or the comment display may be omitted.

An image (stamp) indicating a finished image can be displayed in a selectable manner on the touch panel 27 of the mobile terminal device 2 . For example, the mobile terminal device 2 can store in the ROM 23 of the mobile terminal device 2 comments and images that were displayed when jobs were executed in the past. As a result, when the operation voice processing program is activated, the comments and images that were displayed when the job was executed in the past can be displayed as shown in FIG.

Here, when the user selects an image fed back during execution of a past job by touching the touch panel 27 of the mobile terminal device 2, the mobile terminal device 2 (operation voice processing program) performs settings corresponding to the image. The value can be reflected as the setting value of the current job.

Further, when an image is selected, the portable terminal device 2 (operation voice processing program) displays the selected image again (like the display at 17:00) as shown in FIG. The MFP 1 is instructed to execute a job using a setting value associated with an image as "Parameter". For example, the interpretation result of "Action: Copy_execute", "Parameter: color/monochrome = color, print surface = double-sided, opening direction = top and bottom opening, staple = top two places" is received from the AI assistant server device 4. can be processed. In this case, the mobile terminal device 2 (operation voice processing program) can execute the job without using the voice recognition server device 3 and the AI assistant server device 4 .

In this case, the mobile terminal device 2 carries an image (stamp) indicating a finished image and a setting value corresponding to the image (stamp) (that is, the "Parameter" value fed back from the AI assistant server device 4). It is stored in association with the ROM 23 of the terminal device 2 .

The mobile terminal device 2 (operation voice processing program) stores an image (stamp) indicating a finished image so that it can be used when executing a job from the next time onward. That is, the mobile terminal device 2 (operation voice processing program) instructs to save the image (stamp) displayed on the touch panel 27, which indicates the finished image.

For example, when the user continues to touch the image (stamp) indicating the finished image for a predetermined time (long press), the mobile terminal device 2 (operation voice processing program) selects whether or not to save the image. Display the acceptance screen. When the user instructs to save an image, the mobile terminal device 2 (operation voice processing program) stores the image in the ROM 23 of the mobile terminal device 2 . At this time, the mobile terminal device 2 (operation voice processing program) associates the image (stamp) with the setting value corresponding to the image (stamp) and stores them in the ROM 23 .

The images stored in this way can be called up by a user's instruction. For example, the portable terminal device 2 (operation voice processing program) displays a list of pre-stored images when the user touches the icon I1 shown in the lower left of FIG. When the user designates a desired image from the list, the mobile terminal device 2 (operation voice processing program) displays the image on the touch panel 27 as shown in FIG. 13 (like the display at 17:00). . Thereby, the setting values corresponding to the image can be reflected as the current job setting values.

Returning to FIG. 14, next, the portable terminal device 2 performs the above-described input confirmation feedback such as displaying a finished image (stamp) (step S27). In response to this input confirmation feedback, the user performs voice input instructing to change the setting value or start copying.

Which parameter among the plurality of parameters is the essential parameter can be stored in advance in the storage unit of the AI assist server device 4 . In this case, the interpretation unit 72 determines whether or not the parameters acquired from the mobile terminal device 2 set all the required parameters based on the information of the required parameters stored in the storage unit. If the settings have not been made, the user can be prompted via the mobile terminal device 2 to set the essential parameters.

In this way, the operation voice processing program transmits comments to the mobile terminal device 2 based on the text data stored in advance in the mobile terminal device 2, the text data received from the voice recognition server device 3, and the Response received from the AI assistant server device 4. It is displayed on the screen of the touch panel 27 of the device 2 .

FIG. 15 is a sequence diagram showing the flow of operations when steps S35 to S42 in the sequence diagram of FIG. 15 are voice instructions to change the set values. In FIG. 15, when the user makes an utterance to change the setting value (step S35), the text conversion unit 62 of the speech recognition server device 3 generates text data of the changed setting value, to the AI assistant server device 4 via (steps S36 to S38). AI assistant server device 4 generates an interpretation result indicating the changed setting value based on the user's uttered phrase indicated by the received text data (step S39), and transmits it to portable terminal device 2 (step S40). .

The feedback unit 55 of the mobile terminal device 2 generates a feedback text based on the response included in the interpretation result (step S41), for example, "Monochrome, 2 copies, double-sided copy. Are you sure?" By performing the above-described input confirmation feedback, it is confirmed whether or not copying can be started with the changed set values (step S42).

Steps S43 to S50 in the sequence diagram of FIG. 15 are the operation flow of each section when the start of copying is instructed. That is, when the user responds "yes" to the input confirmation feedback (step S43), the text is converted to text and sent to the AI assistant server device 4 via the mobile terminal device 2 (steps S44 to S46). When the AI assistant server device 4 recognizes the copy start instruction based on the received text data, the interpretation result is obtained by adding the parameters of "print side=both sides" and "number of copies=1 copy" to the action of "Copy_Execute". is formed and transmitted to the mobile terminal device 2 (steps S47 and S48).

Interpretation result conversion unit 53 of portable terminal device 2 converts the interpretation result into a job command for MFP 1 (step S49), and transmits the job command to MFP 1 (step S50). As a result, the MFP 1 can be copy-controlled by the voice input operation.

After the mobile terminal device 2 transmits the job command to the MFP 1, when the MFP 1 is in the connection copy mode ON and the stapling mode ON, the mobile terminal device 2 as the master device sends the job command to the MFP 1 as the slave device. , to request the connection status of peripherals such as finishers. When the peripheral device is not connected to the MFP 1, the touch panel 27 of the portable terminal device 2 displays a message indicating that the linked copy cannot be performed.

(Example of information fed back from AI assistant server device 4)
Table 2 below shows an example of interpretation results fed back from the AI assistant server device 4 to the mobile terminal device 2 .

As shown in Table 2, for example, actions such as "Copy_Parameter_Setting" for prompting input of job setting values, "Copy_Confirm" for prompting confirmation of job setting values, and "Copy_Execute" for notifying start of job execution. is included in the interpretation result and fed back to the mobile terminal device 2 .

The feedback unit 55 can determine feedback to the user according to actions, parameters, and responses included in the interpretation result. The feedback unit 55 may store information corresponding to Table 2 in the storage unit of the mobile terminal device 2 and refer to it in order to determine the content to be fed back. In Table 2, the case of copying was explained as an example, but in the case of printing, scanning, and facsimile, as in Table 2, the action is "Parameter_Setting" for prompting the input of job setting values, and confirming the job setting values. "Confirm" may be used to prompt.

In addition, parameters such as the set value of the printing surface such as double-sided or single-sided, or the number of copies are included in the interpretation result and fed back to the portable terminal device 2 . Furthermore, if the required parameters are missing, a message prompting the input of the missing parameters is included in the interpretation result as a response and fed back to the mobile terminal device 2 .

As described above, according to the present embodiment, by displaying a finished image showing a finished image based on settings, it is possible to intuitively perform feedback for setting confirmation and job setting.

In the description of the above embodiment, the speech recognition server device 3 generates text data corresponding to the user's utterance, and the AI assistant server device 4 interprets the operation intended by the user based on the text data. did. However, the mobile terminal device 2 may be provided with such a speech recognition function and an interpretation function, and the mobile terminal device 2 may interpret the intended operation from the user's utterance. This eliminates the need for the voice recognition server device 3 and the AI assistant server device 4, thereby simplifying the system configuration.

(Second embodiment)
Next, a second embodiment will be described.

The second embodiment differs from the first embodiment in that a smart speaker is applied instead of the mobile terminal device 2 . Hereinafter, in the description of the second embodiment, the description of the same portions as those of the first embodiment will be omitted, and the portions different from those of the first embodiment will be described.

FIG. 18 is a system configuration diagram of a voice operation system according to the second embodiment. As shown in FIG. 18, the voice operation system of the second embodiment employs a smart speaker 50 (an example of an information processing device) in place of the mobile terminal device 2 described in FIG. A smart speaker, also called an AI speaker, is a speaker with an AI assistant function that supports interactive voice operations.

The voice operation system connects an MFP 1, which is an example of an external device, a smart speaker 50 (an example of an information processing device), and a cloud service device 60, via a predetermined network 5 such as a LAN (Local Area Network). It is formed by However, the external device is not limited to the multifunction machine, and may be various electronic devices including office equipment such as an electronic blackboard and a projector.

Smart speaker 50 accepts voice input from the user to operate MFP 1 by voice. Smart speaker 50 is installed close to MFP 1 . Also, the smart speaker 50 and the MFP 1 have a one-to-one correspondence. Therefore, the smart speaker 50 basically provides functions for the user who is operating in front of the MFP 1 . However, this is not the only option, and the smart speaker 50 may correspond to multiple MPFs 1 and other electronic devices.

The cloud service device 60 may physically be one server device, or may be realized by a plurality of server devices. The cloud service device 60 is a control device installed with an operation voice conversion program for converting voice data into text data and interpreting user's intentions. Cloud service device 60 is a control device in which a management program for managing MFP 1 is installed. Therefore, the cloud service device 60 exhibits functions similar to those of the speech recognition server device 3 and the AI assistant server device 4 of the first embodiment.

The operation voice conversion program creates/registers an operation voice dictionary and operations for the MFP1. The management program links accounts/devices of the smart speaker 50 and MFP 1 and manages the entire system.

(Hardware configuration of smart speaker 50)
FIG. 19 is a hardware configuration diagram of the smart speaker 50 provided in the voice operation system. As shown in FIG. 19, the smart speaker 50 includes a CPU 21, a RAM 22, a ROM 23, an interface section (I/F section) 24 and a communication section 25, and a bus line 26, similarly to the portable terminal device 2 described in FIG. are formed by connecting to each other through

The ROM 23 stores an operation voice processing program. The CPU 21 enables the voice input operation of the MFP 1 by executing this operation voice processing program.

A touch panel 27 , a speaker section 28 and a microphone section 29 are connected to the I/F section 24 . The microphone unit 29 collects (obtains) the input voice of a job execution command to the MFP 1 in addition to the call voice. The input speech is transmitted to the cloud service device 60 via the communication unit 25 and converted into text data.

(Hardware Configuration of Cloud Service Device 60)
FIG. 20 is a hardware configuration diagram of the cloud service device 60 provided in the voice operation system. Note that in FIG. 20, the cloud service device 60 is physically composed of one server device. As shown in FIG. 20, the cloud service device 60 includes a CPU 31, a RAM 32, a ROM 33, a HDD (Hard Disk Drive) 34, an interface section (I/F section) 35, as in the speech recognition server device 3 described in FIG. and a communication unit 36 are connected to each other via a bus line 37 . A display unit 38 and an operation unit 39 are connected to the I/F unit 35 . The HDD 34 stores a voice dictionary for operations on the MFP 1 and an operation voice conversion program for creating/registering operations. The HDD 34 also stores a management program that links accounts/devices of the smart speaker 50 and the MFP 1 and manages the entire system. The CPU 31 enables the MFP 1 to be operated based on the voice data transmitted from the mobile terminal device 2 by executing the operation voice conversion program and the management program.

(Overall functional configuration)
FIG. 21 is a schematic explanatory diagram of the overall functions. FIG. 21 shows main functions for providing cloud services. Details of the main functions and functional descriptions of the smart speaker 50 shown in FIG. 21 will be described later with reference to FIGS. 22-23.

Functions of the cloud 100 are implemented by one cloud service device 60 or a plurality of cloud service devices 60 . These functions are appropriately set in one or a plurality of cloud service devices 60, and one cloud service device 60 or a plurality of cloud service devices 60 may be provided.

The CPU 31 of the cloud service device 60 functions as an operation voice conversion unit 310 by reading the operation voice conversion program from the HDD 34 into the RAM 32 and executing it. The operation voice conversion unit 310 has a function of converting voice data into text data. Furthermore, the operation voice conversion unit 310 has a function of determining whether or not the text data matches predefined dictionary information. Further, the operation voice conversion unit 310 has a function of converting the text data into parameters indicating variables such as an action indicating the user's intention and job conditions when there is a match.

Further, the CPU 31 of the cloud service device 60 functions as the voice assistant unit 320 by reading the voice assistant program of the HDD 34 to the RAM 32 and executing it. Voice assistant unit 320 has a function of holding dictionary information.

In addition, the CPU 31 of the cloud service device 60 functions as a management unit 330 by reading the management program of the HDD 34 to the RAM 32 and executing it. The management unit 330 has a function of converting the action and parameters into a job execution instruction in a format interpretable by the MFP 1 and transmitting the instruction to the registered MFP 1 .

As described above, the cloud 100 provides the cloud service 300 by at least the functions of the operation voice conversion unit 310 , the voice assistant unit 320 and the management unit 330 .

The cloud service 300 stores various types of information in the DB based on communication with the MFP 1 and information processing devices. As an example, the management unit 330 manages various information using the management DB 340, the linking DB 350, the device information DB 360, and the like.

The management DB 340 is a database that stores data related to content provided by the cloud service 300, such as text data, image data, and audio data.

The linking DB 350 is a database that stores external devices linked to the information processing apparatus. In this example, the linking DB 350 associates and stores the device ID of the smart speaker 50 used as the information processing device and the ID of the external device (MFP 1 in this example) associated with the smart speaker 50 . Note that the smart speaker 50 and an external device may be associated one-to-one, or the smart speaker 50 and a plurality of external devices may be associated. In other words, the type and number of external devices associated with the device ID are not limited. Also, the method of linking the external device and the smart speaker 50 is not limited to the above method. In other words, a configuration may be employed in which information identifying a user, such as a user account or a user ID, is linked to an external device. In this case, information specifying the smart speaker 50, such as the device ID, which is transmitted from the smart speaker 50 to the cloud, and information specifying the user are stored in a linking DB or the like of the cloud 100, and the management unit 330 The configuration may be such that the external device is specified based on information specifying the user associated with the device ID. Alternatively, information identifying the user may be transmitted from the smart speaker 50 instead of the device ID. Further, instead of the information for identifying the user, information for identifying an organization such as a department or company, or information for identifying a location such as a room or building may be linked with an external device. In this case, one or more smart speakers 50 and one or more external devices may be linked.

The device information DB 360 is a database that associates and stores the ID of each external device including the MFP 1 and the respective device information.

(Function of smart speaker 50)
FIG. 22 is a diagram showing an example of the functional block configuration of the smart speaker 50. As shown in FIG. The CPU 21 of the smart speaker 50 functions as an acquisition unit 211, a communication control unit 212, and a feedback unit 213 as shown in FIG. 22 by executing the operation processing program stored in the ROM 23. FIG.

Acquisition unit 211 acquires a user's instruction voice for voice operation of MFP 1, which is collected through microphone unit 29 (see FIG. 3). Note that the acquisition unit 211 may acquire the user's operation via the touch panel 27 (see FIG. 3), a physical switch (not shown), or the like. The communication control unit 212 controls communication with the cloud 100 . The communication control unit 212 communicates with the cloud 100 , transmits information acquired by the acquisition unit 211 to the cloud 100 , and acquires text data, image data, and audio data from the cloud 100 . When transmitting the information acquired by the acquisition unit 211 to the cloud 100 , the communication control unit 212 may also transmit a device ID that identifies the smart speaker 50 .

The feedback unit 213 feeds back to the user side, for example, a voice prompting an input to compensate for missing data, a voice confirming the input, and the like, in order to realize an interactive voice input operation. Further, the feedback unit 213 may give feedback to the user in the form of text or images by controlling the display of the touch panel 27 .

In this example, the acquisition unit 211 to the feedback unit 213 are implemented by software, but part or all of them may be implemented by hardware such as an IC (Integrated Circuit). Further, each function of the acquisition unit 211 to the feedback unit 213 may be realized by the operation processing program alone, or may be executed by another program to execute part of the processing, or may be indirectly processed by using another program. may be executed.

(Details of cloud service functions)
FIG. 23 is a diagram showing an example of the configuration of each function of the cloud service. The operation voice conversion unit 310 includes functions such as an acquisition unit 311, a text conversion unit 312, an interpretation unit 313, and an output unit 314, as shown in FIG. The acquisition unit 311 acquires audio data transmitted from the smart speaker 50 (audio data input by the user). The acquisition unit 311 may also acquire data indicating operations performed by the user on the touch panel 27 of the smart speaker 50, physical switches (including buttons and the like), and the like. The text conversion unit 312 includes an STT (Speech To Text) that converts voice data (user's voice data input by the smart speaker 50) into text data. The interpretation unit 313 interprets the content of the user's instruction based on the text data converted by the text conversion unit 312 . Specifically, the interpreting unit 313 confirms whether the words included in the text data converted by the text converting unit 312 match the dictionary information provided by the voice assistant unit 320, and confirms whether or not they match the dictionary information provided by the voice assistant unit 320. In this case, the action is converted into an action indicating the type of job and a parameter indicating variables such as job conditions. The interpretation unit 313 then transmits the action and the parameter to the management unit 330 together with, for example, a device ID that identifies the smart speaker 50 from which the audio data is obtained. The output unit 314 includes TTS (Text To Speech) for synthesizing text data with voice data. The output unit 314 controls the communication of the communication unit 36 (see FIG. 4), and transmits data such as text data, voice data, and image data to the smart speaker 50 .

In this example, the acquisition unit 311 to the output unit 314 are implemented by software, but part or all of them may be implemented by hardware such as an IC (Integrated Circuit). In addition, each function realized by the acquisition unit 311 to the output unit 314 may be realized by the operation voice conversion program alone, by causing another program to execute part of the processing, or indirectly by using another program. may be allowed to perform the processing. Also, part or all of the functions of the interpretation unit 313 of the operation voice conversion program may be executed by the voice assistant program. In this case, for example, checking whether words contained in the text data match the dictionary information, and if they match, conversion to parameters indicating variables such as actions indicating user's intentions and job conditions, etc. Voice assistant unit 320 performs this. The interpreter 313 only needs to obtain the actions and parameters from the voice assistant 320 .

The voice assistant unit 320 includes the functions of the providing unit 321 as shown in FIG. The providing unit 321 manages dictionary information that predefines the relationship between text data, actions, and parameters, and provides the operation voice converting unit 310 with the dictionary information. Note that the voice assistant unit 320 may receive text data from the operation voice conversion unit 310 and interpret the user's operation instructions from the text data. For example, the voice assistant unit 320 acquires text data from the interpretation unit 313, checks whether words included in the text data match the dictionary information, and if they match, the text data is treated as an action. Convert to parameters. The actions and parameters are then provided to the interpreter 313 .

In this example, the voice assistant unit 320 (including the providing unit 321) is implemented by software, but part or all of it may be implemented by hardware such as an IC (Integrated Circuit). In addition, the functions of the providing unit 321 and the like may be realized by the voice assistant program alone, may be caused to execute part of the processing by another program, or may be indirectly executed by using another program. good.

As shown in FIG. 23, the management unit 330 includes an acquisition unit 331, an interpretation result conversion unit 332, an execution instruction unit 333, a device information acquisition unit 334, an execution determination unit 335, a notification unit 336, and a DB management unit. 337 and other functions.

Acquisition unit 331 acquires an interpretation result from interpretation unit 313 .

The interpretation result conversion unit 332 converts the interpretation result of the action and parameters converted by the operation voice conversion unit 310 into a job execution command that can be interpreted by the MFP 1 .

Execution instruction unit 333 instructs execution of a job by transmitting a job execution instruction to MFP 1 . Specifically, the execution instructing unit 333 acquires the device ID of the smart speaker 50 given the voice instruction by the user, along with the action and the parameter. The execution instruction unit 333 searches the MFP 1 corresponding to the obtained device ID from the linking DB 350 (see FIG. 21), and transmits a job execution instruction to the MFP 1 obtained by the search.

The device information acquisition unit 334 acquires device information from each registered external device (MFP 1 in this example). For example, the device information acquisition unit 334 acquires information indicating processing capacity such as the maximum number of pixels that can be processed. In addition, the device information acquiring unit 334 obtains the connection status indicating whether or not communication connection is established with the MFP 1, the power status indicating whether the MFP 1 is powered on/off or in sleep mode, and the presence or absence of an error. and error type, remaining amount of consumables such as paper and toner, user login status, authorization information indicating the functions the logged-in user is permitted to use, etc. do.

Note that when the device information acquisition unit 334 acquires device information such as the processing capability from a plurality of MFPs 1, the device information DB 360 (see FIG. 21) associates each external device with information such as an ID that identifies each external device. Manage device information.

The execution determination unit 335 compares the processing capability of the MFP 1 with the job specified by the user (that is, the actions and parameters generated by the operation voice conversion unit 310). Determine whether it is executable or not. If it is determined that the job specified by the user can be executed, it transmits a job execution command to MFP1. Note that when it is determined that the operation cannot be performed, the notification unit 336 may feed back an error message or the like as response information to the smart speaker 50 via the operation voice conversion unit 310 .

The notification unit 336 transmits text data, voice data, image data, etc. to the operation voice conversion unit 310 as a response to the user's job execution instruction. Further, when the parameters indicating the job conditions for executing the job are insufficient, feedback is provided to the smart speaker 50 via the operation voice conversion unit 310 to prompt the user to further instruct the parameters. Here, parameter information may be sent as information necessary to confirm missing parameters, or text data, voice data, and image data may be sent as information necessary to prompt the user to specify parameters. You may send.

The DB management unit 337 manages a management DB 340 , a linking DB 350 and a device information DB 360 . Specifically, it sets various tables, and registers, searches, deletes, and updates data in various tables. For example, the DB management unit 337 associates the device ID of the smart speaker 50 with the ID of the MFP 1 based on information and instructions input to the client device of the MFP 1, the smart speaker 50, or the cloud service apparatus 60. Register in DB 350 for use. The linking DB 350 holds information linking the device ID of the smart speaker 50 and the ID of the MFP 1 as table data or the like.

(Interactive operation flow)
24 to 27 are diagrams showing an example of the overall operation when the user interacts with the voice operation system to operate the MFP. FIG. 24 shows the flow of operations at startup, and FIGS. 25 to 27 show the flows of interactive operations after startup. Interacting with the system to perform operations requires session management of the interaction. Conversation session management will be described later. Here, as an example, an operation is shown in which the user instructs an operation to open a color image on both sides vertically via the smart speaker 50 and to make two copies at two positions on the staple. In this example, the number of copies (=2 copies) is an essential parameter, but it is not limited to the number of copies, and a plurality of parameters such as monochrome, color, paper size, etc. may be included in the essential parameters.

First, after the smart speaker 50 (operation processing program) is activated by the user, for example, the user inputs an activation word into the smart speaker 50 by voice (step S1′). Here, the desired voice assistant program can be started by the user uttering an activation word for starting the voice assistant program. The smart speaker 50 (communication control unit 212) transmits voice data of the activation word to the cloud 100 (operation voice conversion unit 310) (step S2').

In the cloud 100, an operation voice conversion unit 310 (acquisition unit 311) acquires transmission data from the smart speaker 50, and an operation voice conversion unit 310 (text conversion unit 312) converts the voice data into text, that is, converts it into text data. process (step S3').

The operation voice converting unit 310 (interpreting unit 313) requests dictionary information from the voice assistant unit 320 (providing unit 321) and acquires the dictionary information from the voice assistant unit 320 (providing unit 321) (step S4').

Further, the operation voice conversion unit 310 (interpretation unit 313) interprets the text from the acquired dictionary information (step S5').

Then, the operation voice conversion section 310 (interpretation section 313) passes the interpretation result to the management section 330 (step S6').

Here, the management unit 330 searches the linking DB (step S71), checks the connection status (step S72), checks the application status (step S73), acquires device information (step S74), etc., as necessary. . Note that the order of these processes may be changed as appropriate. Also, if each process is performed at a different timing, the process here may be omitted.

In the linking DB search (step S71), the management unit 330 (DB management unit 337) retrieves the MFP 1 (MFP 1 ID) corresponding to the acquired device ID (smart speaker 50 ID) from the linking DB 350. Search and retrieve. At this time, if the ID of the MFP 1 associated with the device ID is not obtained by searching, the management unit 330 (notification unit 336) notifies the operation voice conversion unit 310 ( The user is notified via the output unit 314). For example, the management unit 330 (notification unit 336) generates response information including a response that "this device is not associated with equipment". Here, the management unit 330 (notification unit 336) may include in the response a method for associating the device with the communication target. Note that step S71 may be performed at any timing other than when the device ID is acquired.

In connection state confirmation (step S72), the management unit 330 confirms the device state of the communication target (MFP 1 in this example). For example, the DB management unit 337 checks by referring to device information acquired in advance in the device information DB 360 . Alternatively, the device information acquisition unit 334 may acquire and confirm the device information from the MFP 1 to be communicated with. Here, confirmation of the device state is, for example, confirmation of whether or not communication with the MFP 1 to be communicated is possible and whether or not the MFP 1 is usable. At this time, if the connection with the MFP 1 (confirmation target MFP 1) associated with the device ID has not been established, or if the MFP 1 cannot be used because it is being activated, the management unit 330 (notification unit 336) outputs an operation voice. The user is notified via the conversion unit 310 (output unit 314). For example, the management unit 330 (notification unit 336) generates and notifies response information including a response of "the device is offline" or "the device is being prepared". Here, the management unit 330 (notification unit 336) may include the countermeasure method in the response. Note that the confirmation of the device status may be performed at any other time when the action, parameter, and device ID are acquired from the operation voice conversion unit 310 (interpretation unit 313).

In application status confirmation (step S73), management unit 330 confirms the status of an application that executes a function specified by the user in MFP 1 to be communicated with. For example, the DB management unit 337 checks by referring to device information acquired in advance in the device information DB 360 . Alternatively, the device information acquisition unit 334 may acquire and confirm the device information from the MFP 1 to be communicated with. Here, confirmation of the application state is, for example, confirmation of whether or not the application is installed and whether or not the application is in an executable state. Assuming that the function instructed to be executed is copying, if an application related to copying is not installed in the MFP 1 associated with the device ID, or if the application cannot be used because it is running, the management unit 330 (notification unit 336) notifies the user via operation voice conversion unit 310 (output unit 314). For example, the management unit 330 (notification unit 336) generates and notifies response information including a response of "the application is not installed" or "the application is currently unavailable". Here, the management unit 330 (notification unit 336) may include the countermeasure method in the response. Note that confirmation of the status of the application may be performed at any other timing when the action, parameter, and device ID are acquired from the operation voice conversion unit 310 (interpretation unit 313).

In device information acquisition (step S74), the management unit 330 acquires device information of a communication target (MFP 1 in this example). For example, the DB management unit 337 acquires previously acquired device information from the device information DB 360 . Alternatively, the device information acquisition unit 334 may acquire device information from the MFP 1 to be communicated with. The device status acquired here is used, for example, in the communication target MFP 1 when determining whether or not the job type and job conditions specified by the user can be executed.

If these processes are completed at an arbitrary timing after activation, the management unit 330 (execution determination unit 335) determines that essential parameters are insufficient (step S75). The management unit 330 (execution determination unit 335) determines whether or not all the conditions necessary for job execution are met based on the action and parameters of the interpretation result in the essential parameter shortage determination.

Note that if all the job types and essential setting conditions have been specified when instructing to start the voice assistant program, the "input feedback" step shown below is omitted and the MFP 1 is instructed to execute. may

At this stage, the user has not received the designation of a plurality of actions and parameters that the MFP 1 has, except that the activation is instructed by voice. determine that it is not satisfied. Even if there is an omission of an instruction of essential conditions when instructing activation of the voice assistant program, it is determined that the essential parameters are not satisfied. Therefore, the management unit 330 (notification unit 336) creates response information and transmits the response information to the smart speaker 50 via the operation voice conversion unit 310 (output unit 314) (steps S76 and S77).

Note that the management unit 330 (DB management unit 337 ) manages the communication session with the smart speaker 50 in the management DB 340 . When the management unit 330 (notification unit 336) transmits response information to the smart speaker 50, it can transmit state information indicating that the session is continuing. Here, the state information is information indicating that the session is continuing. Although descriptions are omitted as appropriate in the subsequent procedures, when the cloud 100 makes an inquiry to the smart speaker 50 , it transmits to the smart speaker 50 including state information.

The response information can include text data, voice data, and image data as contents of inquiry to the user. Here, as an example, voice data of "Do you want to copy? Do you want to scan?" is transmitted. As a result, the smart speaker 50 (feedback unit 213) provides audio feedback such as "Do you want to copy? Do you want to scan?" (step S78).

Note that the content of the feedback is not limited to this as long as it is a message prompting the user to enter the type of job or setting conditions for the job. Further, the feedback to the user may be provided by displaying text or images on the touch panel as well as voice output. In that case, text data, image data (display information), and the like are transmitted to the smart speaker 50 .

If the user utters "copy" after step S78 (the same applies if the user utters "copy" when instructing to start the voice assistant program), the process proceeds as follows. The voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-1). The smart speaker 50 (communication control unit 212) transmits this “copy” voice data to the cloud 100 (step S2-1). At this time, the smart speaker 50 (communication control unit 212 ) transmits the device ID that identifies the smart speaker 50 to the cloud 100 .

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and then the operation voice conversion unit 310 performs processing up to text interpretation in the same manner as in steps S3′ to S5′. Then (steps S3-1 to S5-1), the interpretation result is passed to the management unit 330 (step S6-1). Here, an action of "Copy_Execute" corresponding to "copy" is passed as an interpretation result.

Then, the management unit 330 (execution determination unit 335) again determines the shortage of essential parameters (step S75-1). In this example, the user only utters "copy", and the set values such as the number of copies of the essential parameters are unknown.

Therefore, the cloud 100 queries the smart speaker 50 for the missing parameters. Specifically, since the set values are insufficient at this stage, the management unit 330 (notification unit 336) generates response information including "Please enter the set values", and the operation voice conversion unit 310 (output unit 314 ) to the smart speaker 50 (step S75-1 to step S77-1). Then, the smart speaker 50 (feedback unit 213) outputs a voice message "Please input the setting value" (step S78-1). Also in this case, in addition to voice output, the touch panel 27 may display a text message such as "Please enter a setting value."

Next, since there is insufficient input feedback, the user utters, for example, "at two places on color double-sided top and bottom opening staples". The voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-2). The smart speaker 50 (communication control unit 212) transmits the voice data of "at two places on the color double-sided top and bottom opening staples" to the cloud 100 (step S2-2). At this time, the smart speaker 50 (communication control unit 212 ) transmits the device ID that identifies the smart speaker 50 to the cloud 100 .

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and then the operation voice conversion unit 310 performs processing up to text interpretation in the same manner as in steps S3′ to S5′. Then (steps S3-2 to S5-2), the interpretation result is passed to the management unit 330 (step S6-2).

Here, the operation voice conversion unit 310 (interpretation unit 313) generates as an interpretation result a parameter with "Parameters: color/monochrome = color, printing surface = double-sided, opening direction = top and bottom opening, staple = top two places", The interpretation result is passed to the management unit 330 .

Specifically, the management unit 330 (DB management unit 337) completes the action and parameters by integrating the interpretation result of the previous utterance and the interpretation result of the current utterance. That is, in this example, the action "Copy_Execute" and the parameters "Parameters: color/monochrome = color, print surface = double-sided, opening direction = top and bottom opening, staple = top two places" are completed. The management unit 330 (execution determination unit 335) again determines the lack of essential parameters based on the integrated interpretation result. In this example, the user's utterance of "two copies" resolves the shortage of essential parameters for the copy job.

In this example, the management unit 330 (notification unit 336) subsequently sends the response information "color, double-sided, top and bottom opening, and two stapled top copies. Are you sure?" to provide input confirmation feedback. , and transmits voice data to the smart speaker 50 via the operation voice conversion unit 310 (output unit 314) "Color, double-sided, open top/bottom, copy in 2 places on top of staple. Are you sure?" (Steps S75-3 to S77-3). Then, the smart speaker 50 (feedback section 213) outputs a voice saying "I will copy two copies on both sides. Are you sure?" (step S78-3). Also in this case, in addition to voice output, the touch panel 27 may display text such as "color, double-sided, top and bottom opening, and two stapled top copies. Are you sure?" Here, instead of outputting the text data and voice data included in the response information, the output information is generated by combining the text data stored in the storage unit of the smart speaker 50 based on the information included in the response information. may

After that, the user speaks to the terminal to change the setting value or start copying in response to this input confirmation feedback. When the user utters a content to change the setting value, voice data is transmitted from the smart speaker 50 to the cloud 100 regarding the change in the uttered setting value, and the setting value is changed in the cloud 100, and the smart speaker At 50, voice feedback is provided about the fact that the setting value has been changed. The voice feedback is, for example, "Copy with XX settings. Are you sure?"

After that, if the user speaks to change the setting value, this procedure is repeated. Therefore, after the voice output of "color, double-sided, top-bottom opening, two-position copy on staple. Are you sure?" is output, the number of times (k times) the procedure is repeated.

If the user gives an instruction to start copying by answering "yes" or the like, the n-th procedure shown in FIG. 27 is performed. That is, the voice uttered by the user is acquired as voice data by the smart speaker 50 (acquisition unit 211) (step S1-n). The smart speaker 50 (communication control unit 212) transmits this "yes" voice data to the cloud 100 (step S2-n). At this time, the smart speaker 50 (communication control unit 212 ) transmits the device ID that identifies the smart speaker 50 to the cloud 100 .

In the cloud 100, the operation voice conversion unit 310 (acquisition unit 311) acquires the voice data, and then the operation voice conversion unit 310 performs processing up to text interpretation in the same manner as in steps S3′ to S5′. Then (steps S3-n to S5-n), the interpretation result is passed to the management unit 330 (step S6-n).

Here, when operation voice conversion unit 310 (interpretation unit 313) recognizes the copy start instruction, operation voice conversion unit 310 (interpretation unit 313) passes the interpretation result to management unit 330, and management unit 330 (execution determination unit 335) determines that the final confirmation is OK ( step S75-n).

As a result, the management unit 330 (interpretation result conversion unit 332) converts the interpretation result into a job command for the MFP 1 (step S76). The management unit 330 (execution instruction unit 333) then transmits the converted execution instruction information to the MFP 1 (step S8). As a result, the MFP 1 can be copy-controlled by the voice input operation.

FIG. 28 is a front view showing a screen displayed on the display unit of smart speaker 50. As shown in FIG. As shown in FIG. 28, the screen displayed on the display unit of the smart speaker 50 is the same as the screen displayed on the mobile terminal device 2 shown in FIG.

Processing of speech and feedback to the smart speaker 50 is the same as the processing shown in the first embodiment. Schematically, the smart speaker 50 outputs the content uttered by the user and the response information received from the cloud service device 60 (operation voice conversion program). The response information is information including at least one of text data, voice data, and image data.

Note that the comment balloon-displayed from the right side of the screen of the touch panel 27 of the smart speaker 50 in FIG. Also, in FIG. 28 , the comment balloon-displayed from the left side of the screen of the touch panel 27 of the smart speaker 50 indicates the contents of voice feedback from the cloud service device 60 in response to the user's utterance. It is a comment or an image (stamp) indicating the content of feedback to the user's utterance. In other words, when the smart speaker 50 receives the feedback information from the cloud service device 60, the smart speaker 50 outputs the feedback to the user through voice output, and at the same time, also provides the feedback through the screen display. However, feedback of audio output may be omitted.

24 to 27, the comment "Do you want to copy? Do you want to scan?"

Note that the operation voice processing program of the smart speaker 50 generates text to be displayed based on the response information from the cloud service device 60, and displays text data pre-stored in the ROM 23 of the smart speaker 50. Also good. Also, the text data and voice data included in the response information may be displayed as they are.

The operation voice processing program of the smart speaker 50 receives the comment "copy" as response information, which is converted from voice data to text data by the cloud service device 60 (operation voice conversion program), and displays the response information on the touch panel 27 of the smart speaker 50. can be displayed on the screen.

Note that the cloud service device 60 (operation voice conversion program) can transmit the response information at any timing. For example, the cloud service device 60 (operation voice conversion program) may generate response information of “copy” at the timing of conversion into text data and transmit it to the smart speaker 50 (in this case, “copy” is displayed).

In addition, the cloud service device 60 (management program) also generates the response information of "copy" at the timing of generating the response information of "Please enter the setting value", 50 (In this case, "Copy" and "Please enter the setting value" are displayed on the touch panel 27 of the smart speaker 50 almost simultaneously.

Note that the operation voice conversion program may transmit information necessary for generating response information of "copy" when transmitting the intent of "Copy_Execute" to the management program as an interpretation result.

In addition, the response information is created by the operation voice conversion program, and when the management program transmits the response information of "Please enter the setting value" to the smart speaker 50 via the operation voice conversion program, "copy ” may be transmitted to the smart speaker 50 .

The operation voice processing program of the smart speaker 50 displays the comment "Please enter the setting value" on the screen of the touch panel 27 of the smart speaker 50 along with voice feedback in step S78-1 of FIGS. That is, the comment is displayed based on the response information received from the cloud service device 60 (management program).

The operation voice processing program of the smart speaker 50 receives the comment "color, double-sided, top and bottom opening, two places on the staple", which is converted from voice data into text data by the cloud service device 60 (operation voice conversion program). can be displayed on the screen. The display method is the same as when "copy" is spoken.

At the same time, if the operation voice processing program of the smart speaker 50 determines that the input is not insufficient, it displays an image (stamp) indicating a finished image on the touch panel 27 and prompts an instruction to start copying.

In this way, the smart speaker 50 displays comments on the screen of the touch panel 27 of the smart speaker 50 based on text data stored in advance in the smart speaker 50, text data received from the cloud service device 60, or response information.

Here, a case where the specific example described in the first embodiment is applied to the present embodiment will be described.

The cloud service device 60 (management program) can transmit, as response information, text data or voice data for voice output feedback, and text data or image data (stamp) for display output feedback. Also, the response information may include information indicating the job type and job setting conditions, such as intent and parameters. In this case, the cloud service device 60 (operation voice conversion program) can include, as response information, intents and parameters that are analysis results obtained from the operation voice conversion program.

At this time, the operation voice processing program of the smart speaker 50 displays the finished image based on the analysis result as an image (stamp) as shown in FIG. can do.

The operation voice processing program of the smart speaker 50 refers to the "parameter" when the "intent" of the analysis result is "Copy_execute". Then, the operation voice processing program of the smart speaker 50 searches for an image (stamp) indicating a finished image that matches the value of the “parameter” and causes the touch panel 27 to display the searched image (stamp). Here, if a plurality of setting values are set in the "parameter", an image (stamp) that satisfies all the setting values is searched. For example, the ROM 23 of the smart speaker 50 stores images (stamps) in association with setting values. For example, it can be stored as table data as shown in FIG. If there is no image (stamp) that satisfies all set values, the closest image (stamp) may be displayed.

Note that the table data shown in FIG. 16 may be stored not in the smart speaker 50 but in an external device that the smart speaker 50 can access. For example, it may be stored in a server connected via network 5 . In this case, the operation voice processing program of the smart speaker 50 can access the server, transmit the setting value included in the analysis result, and acquire an image (stamp) that satisfies the setting value as a response from the server. The stamp is not limited to this and may be stored in the HDD 34 of the cloud service device 60 .

Although the case where the smart speaker 50 searches for images (stamps) based on feedback from the cloud service device 60 has been described above, the present invention is not limited to this, and images (stamps) may be searched for by the cloud service device 60. good. In this case, the response information including an image (stamp) indicating the finished image is transmitted to the smart speaker 50 . Here, the cloud service device 60 (operation voice conversion program) may retrieve images (stamps) and transmit them to the smart speaker 50, but other programs (for example, operation voice conversion programs) may retrieve and transmit images (stamps). Also good.

The smart speaker 50 causes the touch panel 27 to display the received image. At this time, the cloud service device 60 (management program) searches for an image (stamp) showing a finished image that matches the value of the "parameter" included in the analysis result obtained from the operation voice conversion program. The cloud service device 60 (management program) can search and acquire an image (stamp) representing an image by querying the HDD 44 of the cloud service device 60 or a server accessible by the cloud service device 60.

Further, when the analysis result is "intent: Copy_confirm" and "parameters: print surface = double-sided, number of copies = 2", an image (stamp) showing a finished image can be displayed as shown in FIG. In the example shown in FIG. 17, the number "2" indicating the number of copies is displayed together with the finish image.

In addition to displaying an image (stamp) indicating a finished image, the smart speaker 50 may provide voice feedback such as "I will make two copies of both sides. Are you sure?" can be Also, in addition to displaying the finished image, a comment "I will copy two copies on both sides. Are you sure?" may be displayed, or the comment display may be omitted.

An image (stamp) indicating a finished image can be displayed in a selectable manner on the touch panel 27 of the smart speaker 50 . For example, the smart speaker 2 can store in the ROM 23 of the smart speaker 2 comments and images that were displayed when jobs were executed in the past.

Alternatively, the smart speaker 2 may store in the cloud service device 60 comments and images that were displayed when jobs were executed in the past. When the operation voice processing program is started, or when the program of the cloud service device 60 is called, the operation voice conversion program or the management program sends the stored information to the smart speaker at a predetermined timing via the operation voice conversion program. You can send it to 2.

As a result, when the operation voice processing program is activated, the comments and images that were displayed when the job was executed in the past can be displayed as shown in FIG.

Here, when the user selects an image fed back during execution of a past job by touching the touch panel 27 of the smart speaker 50, the mobile terminal device 2 (operation voice processing program) performs settings corresponding to the image. The value can be reflected as the setting value of the current job.

Further, when an image is selected, the mobile terminal device 2 (operation voice processing program) displays the selected image again (like the display at 17:00) as shown in FIG. The MFP 1 is instructed via the cloud service device 60 to execute a job based on the job type and set values associated with the cloud service device 60 .

As a result, the cloud service device 60 (operation voice conversion program) specifies "intent: Copy_execute", "parameters: color/monochrome = color, print surface = double-sided, opening direction = top and bottom opening, staple = top two places". Interpretation results can be sent to the management program.

That is, the smart speaker 50 transmits information on the job type and job setting values linked to the image to the cloud service device 60 (operation voice conversion program), and the cloud service device 60 (operation voice conversion program) Based on the acquired job type and set values, an interpretation result is generated and sent to the management program.

The management program transmits a job execution command to MFP 1 based on the interpretation result. Here, the information about the job type and job setting values may be the intent and parameters included in the response information, or may be text data included in the response information. In the case of intents and parameters, the operation speech conversion program sends the acquired intents and parameters to the management program without the need to convert them into text and generate interpretation results. In the case of text data, the operation voice conversion program does not convert the data into text, but only generates an interpretation result, and transmits the generated interpretation result to the management program.

In this case, the smart speaker 50 displays an image (stamp) indicating a finished image, and setting values corresponding to the image (stamp) (that is, the values of the "intent" and "parameter" fed back from the cloud service device 60, or text data included in the response information) is associated with the ROM 23 of the smart speaker 50 and stored.

The smart speaker 50 (operation voice processing program) stores an image (stamp) indicating a finished image in the smart speaker 50 so that it can be used when executing a job from the next time onward. In other words, the smart speaker 50 (operation voice processing program) instructs to save an image (stamp) indicating a finished image displayed on the touch panel 27 of the smart speaker 50 .

For example, when the user continues to touch an image (stamp) indicating a finished image for a predetermined time (long press), the smart speaker 50 (operation voice processing program) accepts a selection as to whether to save the image. display the screen. When the user instructs to save an image, the smart speaker 50 (operation voice processing program) stores the image in the ROM 23 of the smart speaker 50 . At this time, the smart speaker 50 (operation voice processing program) associates the image (stamp) with the setting value corresponding to the image (stamp) and stores them in the ROM 23 .

The images stored in this way can be called up by a user's instruction. For example, the smart speaker 50 (operation voice processing program) displays a list of pre-stored images when the user touches the icon I1 shown in the lower left of FIG. When the user designates a desired image from the list, the smart speaker 50 (operation voice processing program) displays the image on the touch panel 27 as shown in FIG. 28 (like the display at 17:00). Thereby, the setting values corresponding to the image can be reflected as the current job setting values.

As described above, according to the present embodiment, by displaying a finished image showing a finished image based on settings, it is possible to intuitively perform feedback for setting confirmation and job setting.

In the above description, an image (stamp) and an action and parameters associated with the image are stored on the smart speaker 50 side. ), and the intent and parameters associated with the image. Thus, when a user selects an image fed back when executing a past job by touching the touch panel 27, the setting value corresponding to the image can be reflected as the setting value of the current job.

When an image (stamp) is selected on the screen of the touch panel 27 of the smart speaker 50, the smart speaker 50 notifies the cloud service device 60 of which image has been selected. For example, ID information of the image may be notified to the cloud service device 60 .

Based on the information indicating which image has been selected, the operation speech conversion program of the cloud service device 60 stores the intent and parameters (or text data) associated with the image in a storage unit such as the HDD 44 in the cloud service device 60. read from In other words, images, intents, and parameters are linked and managed in the storage unit of the cloud service device 60 . Thereby, the operation voice conversion program can transmit the intent and parameters associated with the image to the management program.

Finally, each embodiment described above is presented as an example and is not intended to limit the scope of the present invention. This novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention.

Such each embodiment and modifications of each embodiment are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

In the above embodiment, an example in which the image forming apparatus of the present invention is applied to a multifunctional machine having at least two functions out of a copy function, a printer function, a scanner function and a facsimile function will be described. , printers, scanners, facsimile machines, and other image forming apparatuses.

2, 50 information processing device 1 external device 51, 211 acquisition unit 52, 212 output unit 55, 213 notification unit 62 speech recognition unit

JP 2014-203024 A

Claims (4)

In an information processing system including an information processing device and an external device,
an acquisition unit that acquires voice information including setting instructions for operating the external device;
a voice recognition unit that recognizes the voice information;
a notification unit that notifies operation information based on the result of recognition of the voice information by the voice recognition unit on a screen of the information processing device;
an output unit that outputs the operation information to the external device;
with
The notification unit displays on the screen of the information processing apparatus a finished image indicating a finished image based on the settings related to the operation information, and when a predetermined operation is performed on the finished image, the corresponding settings are displayed together with the finished image. remember ,
An information processing system characterized by: The notification unit displays a list of the stored finished images on a screen of the information processing device.
The information processing system according to claim 1 , characterized by: A computer that controls an information processing device,
an acquisition unit that acquires voice information including setting instructions for operating an external device;
a notification unit that notifies, on a screen of the information processing device, operation information based on a result of recognition of the voice information by a voice recognition unit that recognizes the voice information;
an output unit that outputs the operation information to the external device;
function as
The notification unit displays on the screen of the information processing apparatus a finished image indicating a finished image based on the settings related to the operation information, and when a predetermined operation is performed on the finished image, the corresponding settings are displayed together with the finished image. remember ,
A program characterized by An information processing method in an information processing system including an information processing device and an external device,
an acquisition step of acquiring audio information including setting instructions for operating the external device;
a voice recognition step of recognizing the voice information;
a notification step of notifying operation information based on the result of recognition of the voice information by the voice recognition step on the screen of the information processing device;
an output step of outputting the operation information to the external device;
including
The notifying step displays a finished image showing a finished image based on the settings related to the operation information on the screen of the information processing device , and when a predetermined operation is performed on the finished image, the corresponding settings are displayed together with the finished image. remember ,
An information processing method characterized by:

Images