JP2019049901A - Information processor and program - Google Patents

Abstract

To allow a user to visually confirm that an instruction has been given to a target to which the instruction is supposed to be given.SOLUTION: If a permission to use an apparatus is given to a dialogue partner in an interface for a user to have a dialogue with the dialogue partner, the display of the partner is switched to the display of the apparatus.SELECTED DRAWING: Figure 36

Description

  The present invention relates to an information processing apparatus and program.

  Patent Document 1 discloses a technology for establishing communication with one or more bots and interacting with a device through the one or more bots.

Japanese Patent Application Publication No. 2009-522687

  An object of the present invention is to enable a user to visually confirm whether an instruction is given to an object for which the instruction is given.

  The invention according to claim 1 is a control means for switching the display of the other party in the response to the display of the device when the user gives an instruction to use the device in the interface for interacting with the other party. An information processing apparatus having

  The invention according to claim 2 is the information processing apparatus according to claim 1, wherein the control means switches the other party in the response to the device while switching the display.

  The invention according to claim 3 is characterized in that the control means switches an image associated with the other party in response to an image associated with the device as the switching of display. 2 is the information processing apparatus described in 2;

  The invention according to claim 4 is the information processing apparatus according to claim 3, wherein the image associated with the device is an image representing a state of the device.

  The invention according to claim 5 is the information processing apparatus according to claim 4, wherein the image representing the state of the device is an image representing a state in which the device is executing processing. .

  The invention according to a sixth aspect is the information processing apparatus according to the third aspect, wherein the image associated with the device is an image associated with a portion of the device that is performing processing. .

  In the invention according to claim 7, when the user gives an instruction to use the portion of the device in the interface to the interaction partner, the control means displays the display of the portion other than the device in response. The information processing apparatus according to claim 1, characterized in that:

  The invention according to claim 8 is characterized in that the control means switches an image linked to the other party in the response to an image linked to the part of the device as the display switching. It is an information processor of a statement.

  In the invention according to claim 9, the control means responds when the user gives an instruction to use the plurality of devices to execute the cooperation function using the plurality of devices in the interface. The information processing apparatus according to claim 1, wherein the display of the conversation partner is switched to the display of the plurality of devices.

  The invention according to claim 10 is characterized in that the control means switches an image linked to the conversation partner in response to an image of the plurality of devices as switching of display. It is an information processing apparatus.

  The invention according to claim 11 is a control means for switching the display of the conversation partner in response to the display of the function when the user gives an instruction to use the function on the interface for interacting with the conversation partner. An information processing apparatus having

  The invention according to claim 12 is the information processing apparatus according to claim 11, wherein the control means switches the other party in the response to the function while switching the display.

  The invention according to claim 13 is characterized in that, as the switching of the display, the control means switches an image linked to the conversation partner in response to an image linked to the function. 12 is the information processing apparatus according to 12.

  The invention according to claim 14 is the information processing apparatus according to claim 13, wherein the image associated with the function is an image representing the state of the function.

  The invention according to claim 15 is the information processing apparatus according to claim 14, wherein the image representing the state of the function is an image representing a state in which the function is executing processing. .

  In the invention according to claim 16, the control means responds when the user gives an instruction to use the plurality of functions in order to execute the cooperation function using the plurality of functions in the interface. The information processing apparatus according to claim 11, wherein the display of the conversation partner is switched to the display of the plurality of functions.

  The invention according to claim 17 is characterized in that the control means switches an image linked to the conversation partner in response to an image of the plurality of functions as switching of display. It is an information processing apparatus.

  The invention according to claim 18 is that, when the user gives an instruction to use the device and function in the interface for allowing the user to interact with the other person, the display of the other person in response is displayed of the device and the function. It is an information processor which has a control means switched to a display.

  The invention according to claim 19 is a control means for switching the display of the other party in response to the display of the file when the user gives an instruction to use the file in the interface for interacting with the other party. An information processing apparatus having

  The invention according to claim 20 is a control means for switching the display of the conversation partner in response to the display of the software when the user gives an instruction to use the software partner in the interface for interacting with the conversation partner. An information processing apparatus having

  The invention according to claim 21 is an interface for a user to interact with a conversation partner, wherein an instruction to use at least two of a device, a function, a file and software is given to the conversation partner in response. It is an information processing apparatus which has a control means which changes a display of the said dialog other party to a display of the said combination.

  The invention according to claim 22 is that, when the computer gives an instruction to use a device in the interface for the user to interact with the other party, the display of the other party in response is displayed on the device. It is a program made to function as a control means to switch.

  According to the first, second, and third aspects of the invention, the user can visually confirm whether or not an instruction is given to the device.

  According to the invention described in claim 2, the user can interact with the device.

  According to the fourth aspect of the present invention, the user can visually confirm the state of the device.

  According to the fifth aspect of the present invention, the user can visually confirm whether the device is performing processing.

  According to the sixth aspect of the present invention, the user can visually confirm the portion where the processing is being performed.

  According to the seventh and seventh aspects of the present invention, the user can visually confirm whether or not an instruction is given to a part of the device.

  According to the inventions of claims 9 and 10, the user can visually confirm whether or not instructions are given to a plurality of devices.

  According to the invention described in claims 11 and 13, the user can visually confirm whether or not an instruction is given to the function.

  According to the invention of claim 12, the user can interact with the function.

  According to the invention of claim 14, the user can visually confirm the state of the function.

  According to the fifteenth aspect of the present invention, the user can visually confirm whether the function is executing a process.

  According to the inventions of claims 16 and 17, the user can visually confirm whether or not an instruction is given to a plurality of functions.

  According to the eighteenth aspect of the present invention, the user can visually confirm whether an instruction has been given to the device and the function.

  According to the nineteenth aspect of the present invention, the user can visually confirm whether or not an instruction is given to the file.

  According to the invention set forth in claim 20, the user can visually confirm whether or not an instruction is given to the software.

  According to the invention described in claim 21, the user can visually confirm whether or not an instruction is given to the combination.

It is a block diagram showing composition of a device system concerning a 1st embodiment of the present invention. It is a block diagram which shows the structure of a terminal device. FIG. 1 is a block diagram showing the configuration of an image forming apparatus. It is a block diagram showing composition of a server. FIG. 1 is a schematic view showing an appearance of an image forming apparatus. It is a figure which shows an apparatus function management table. It is a figure which shows a cooperation function management table. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a sequence diagram which shows operation | movement of an apparatus system. It is a figure which shows a screen. It is a figure which shows a synthetic | combination image. It is a figure which shows the cooperation function management table which concerns on the modification 3. FIG. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows the apparatus function management table which concerns on the modification 4. FIG. It is a figure which shows an apparatus image. It is a figure which shows an apparatus image. It is a figure which shows a screen. It is a figure which shows a screen. FIG. 18 is a diagram illustrating another example of the device function management table according to the fourth modification. It is a figure which shows the cooperation function management table which concerns on the modification 4. FIG. It is a figure which shows a screen. It is a figure which shows the cooperation function management table which concerns on 2nd Embodiment. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows the cooperation function management table which concerns on 3rd Embodiment. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows the cooperation function management table which concerns on 4th Embodiment. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows the image associated with the other party of interaction. It is a figure which shows the image associated with the other party of interaction. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen.

First Embodiment
An apparatus system as an information processing system according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 shows an example of the device system according to the first embodiment.

  The device system according to the first embodiment includes, as an example, a terminal device 10, a plurality of devices (for example, devices 12 and 14), and a server 16 as an example of an external device. In the example shown in FIG. 1, the terminal device 10, the devices 12 and 14, and the server 16 have a function of communicating with each other via a communication path N such as a network. Of course, the terminal device 10, the devices 12, 14 and the server 16 may communicate with other devices via different communication paths without using the communication path N. In the example shown in FIG. 1, two devices (devices 12 and 14) are included in the device system, but three or more devices may be included in the device system. Also, a plurality of terminal devices 10 and a plurality of servers 16 may be included in the device system. The server 16 may not be included in the device system.

  The terminal device 10 is a device such as a personal computer (PC), a tablet PC, a smartphone, a mobile phone, and the like, and has a function of transmitting and receiving data to and from another device. The terminal device 10 may also be a wearable terminal (watch-type terminal, wristband-type terminal, eyeglass-type terminal, ring-type terminal, contact lens-type terminal, implantable terminal, wearable terminal, etc.). The terminal device 10 functions as, for example, a user interface unit (UI unit) when using the device.

  The devices 12 and 14 are devices having functions, for example, an image forming device having an image forming function, a PC, a tablet PC, a smartphone, a mobile phone, a robot (humanoid robot, non-human animal robot, other than them). Robots, etc.), projectors, display devices such as liquid crystal displays, recording devices, reproducing devices, imaging devices such as cameras, refrigerators, rice cookers, microwave ovens, coffee makers, vacuum cleaners, washing machines, air conditioners, lighting devices, watches, Devices such as surveillance cameras, automobiles, motorcycles, aircrafts (for example, unmanned aerial vehicles (so-called drones)), game machines, various sensing devices (for example, temperature sensors, humidity sensors, voltage sensors, current sensors, etc.). The devices 12 and 14 may be devices (for example, an image forming apparatus or a PC) that outputs an output to the user, or devices (for example, a sensing device) that do not output an output to the user. It is also good. Also, all the devices among the plurality of devices that execute the cooperation function described later may be devices that output the output to the user, or some devices output the output to the user Devices that do not output the output to the user, or all the devices may not output the output to the user. The category of the concept of the devices 12 and 14 may include general devices. For example, information devices, video devices, audio devices, and other devices may be included in the category of devices according to the present embodiment. The devices 12 and 14 have a function of transmitting and receiving data to and from other devices.

  The server 16 is a device that manages data, manages user information, and manages the devices 12 and 14. Also, the server 16 has a function of transmitting and receiving data to and from other devices.

  In the terminal device 10, a conversation partner having a function of interacting with the user is used. The dialogue partner has a function of, for example, receiving a user's speech and analyzing the contents of the speech to create a reply such as an answer to the speech and provide the user with the reply. For example, the user's speech is performed by character input, voice input, image input and the like, and the reply such as an answer is performed by character output, voice output, image output and the like. The interaction partner is realized, for example, by executing a program, and the program is installed in, for example, the terminal device 10. Of course, the program related to the interaction partner may be installed in an external device such as the server 16 and the function of the interaction partner may be provided from the external device such as the server 16 to the terminal device 10.

  The above-mentioned dialogue partner is constituted by automatic response AI realized by artificial intelligence (AI), for example. The automatic response AI has a function of analyzing the content of the user's speech and providing the user with a reply such as an answer to the content of the speech. The automatic response AI may be a so-called chatbot (automatic conversation program utilizing artificial intelligence). The automatic response AI may have a learning function by artificial intelligence, and may have an ability to make a judgment close to a person by the learning function. Also, deep learning of neural network type may be used, reinforcement learning etc. that partially strengthens the learning field may be used, and others, genetic algorithm, cluster analysis, self-organizing map, ensemble Learning, etc. may be used. Of course, other artificial intelligence related techniques may be used.

  By using the automatic response AI, while the user interacts with the automatic response AI, a notification of a function possessed by each device, a notification of a collaboration function that can be executed using a plurality of functions, and the like are performed. Note that the process of specifying the function or cooperation function of each device may be performed by the terminal device 10 or may be performed by an external device such as the server 16 or the like. In the following, it is assumed that the above-described identification process is performed by the terminal device 10.

  Hereinafter, the configuration of the terminal device 10 will be described in detail with reference to FIG.

  The communication unit 18 is a communication interface, and has a function of transmitting data to another device and a function of receiving data from the other device. The communication unit 18 may be a communication interface having a wireless communication function or may be a communication interface having a wired communication function. The communication unit 18 supports, for example, one or more types of communication methods, and may communicate with the communication partner according to a communication method suitable for the communication partner (that is, a communication method compatible with the communication partner). . The communication method is, for example, infrared communication, visible light communication, Wi-Fi (registered trademark) communication, proximity wireless communication (for example, NFC (Near Field Communication)), or the like. As proximity wireless communication, Felica (registered trademark), Bluetooth (registered trademark) (registered trademark), RFID (Radio Frequency Identifier), etc. are used. Of course, another type of wireless communication may be used as close proximity wireless communication. For example, the communication unit 18 may switch the communication scheme or the frequency band according to the other party of communication, or may switch the communication scheme or the frequency band according to the surrounding environment.

  The UI unit 20 is a user interface unit, and includes a display unit and an operation unit. The display unit is, for example, a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel or a keyboard. Of course, a user interface (for example, a touch-type display, a device that electronically displays a keyboard and the like on the display, and the like) that doubles as a display unit and an operation unit may be used. In addition, the UI unit 20 may include a sound collection unit such as a microphone and an audio generation unit such as a speaker. In this case, information may be input to the terminal device 10 by voice input, or information may be emitted by voice.

  The display unit of the UI unit 20 includes, for example, an image captured by a camera, an image to be associated with a device to be used (for example, a device used alone or a device to be cooperated), An image with a string, etc. may be displayed. The image associated with the device may be an image (still image or moving image) representing the device captured by a camera, or an image (for example, an icon) schematically representing the device. Data of an image schematically represented may be created, for example, by the control unit 26 of the terminal device 10, may be stored in advance in the storage unit 24 of the terminal device 10, or is stored in the server 16 16 may be provided to the terminal device 10, or may be stored in another device and provided to the terminal device 10 from the other device. The image associated with the function is, for example, an image such as an icon representing the function.

  The camera 22 as a photographing unit generates image data (for example, still image data and moving image data) by photographing an object to be photographed. The image is displayed on the display unit of the UI unit 20, for example. The image displayed on the display unit may be manipulated by the user. Note that image data captured by an external camera may be sent to the terminal device 10 via the communication path, and the image may be displayed on the display unit of the UI unit 20. Also in this case, the image may be manipulated by the user.

  The storage unit 24 is a storage device such as a hard disk or a memory (for example, an SSD). The storage unit 24 includes, for example, device function management information, cooperation function management information, various data, various programs (for example, OS (Operating System), programs (software) for realizing automatic response AI, various applications Program (application software) etc., information indicating the address of each device (device address information), information indicating the address of the server 16 (server address information), information on the identified device, information on the identified device to be cooperated Information on the function of the identified device, information on the cooperation function, and the like are stored. Of course, they may be stored in separate storage devices, or may be stored in one storage device. The program for realizing the automatic response AI is stored in an external device such as the server 16 and the program itself is executed by the external device to provide the terminal device 10 with the function of the automatic response AI from the external device. May be In this case, the program for realizing the automatic response AI may not be stored in the storage unit 24.

  The device function management information and the linked function management information will be described below.

  The device function management information is information for managing the function of each device, and, for example, device identification information (device identification information) for identifying the device and function information indicating the function of the device. It is information indicating the correspondence. The device identification information includes, for example, a device ID, a device name, information indicating a device type, device type number, information for managing a device (for example, asset management number), and information indicating a position where the device is installed. (Position information of the device), device image associated with the device, address information of the device, and the like. The device image is, for example, an appearance image representing the device. The appearance image may be an image representing the outside of the device (for example, the housing of the device), or an image representing a state in which the inside can be seen from the outside (for example, the internal structure) by opening the housing. It may be an image representing a state in which the device is covered by a packing sheet or the like. The device image may be an image (such as an image representing the outside of the device or an image representing the inside of the device) generated by photographing the device, or an image (for example, an icon or the like) simulating the device. . The function information is, for example, a function ID or a function name. For example, when the device 12 is an image forming apparatus, and the image forming apparatus has a print function, a scan function, and a copy function, the device identification information of the image forming apparatus is, for example, function information indicating the print function. , Function information indicating a scan function, and function information indicating a copy function are associated with each other. The function possessed by each device is identified (identified) by referring to the device function management information.

  The devices managed by the device function management information are, for example, devices (for example, devices 12 and 14) included in the device system. Of course, devices not included in the device system may be managed by the device function management information. For example, the terminal device 10 may acquire information related to a new device (information including device identification information and function information) not included in the device system, and newly register the information in the device function management information. The information on the device may be acquired by using, for example, the Internet or may be input by a manager or the like. In addition, the terminal device 10 may update the device function management information at an arbitrary timing, periodically, or at a timing designated by a manager or the like. As a result, there may be a case where function information indicating a function that the device did not have before the update but that the device had after the update is registered in the device function management information. Similarly, there may be cases where function information indicating functions that the device had before the update but that the device did not have after the update is deleted from the device function management information or registered as unusable information. is there. The information for update may be acquired, for example, by using the Internet or the like, or may be input by a manager or the like. Of course, the external device such as the server 16 acquires information on the device and generates the device function management information without the terminal device 10 itself acquiring the information on the device and executing generation and update of the device function management information. Or you may perform an update. In this case, the device function management information generated or updated by the external device is transmitted from the external device to the terminal device 10 and stored in the storage unit 24.

  The cooperation function management information is information for managing a cooperation function executed by causing a plurality of functions to cooperate. By linking a plurality of functions, one or more linking functions are executed. The cooperation function may be, for example, a function that can be executed by linking a plurality of functions of one device (for example, device 12), or a plurality of functions of a plurality of devices (for example, devices 12 and 14) It may be an executable function by linking the Moreover, the terminal device 10 which has issued the operation instruction may also be used as a device to be cooperated, and the function of the terminal device 10 may be used as part of the cooperation function.

  The cooperation function may be a function executed without using a device as hardware. For example, the linkage function may be a function executed by linking a plurality of software. Of course, the cooperation function may be a function executed by linking the function of the device as hardware and the function realized by the software.

  The cooperation function management information is, for example, information indicating correspondence between a combination of function information indicating each function used for the cooperation function and the cooperation function information indicating the cooperation function. The collaboration function information is, for example, a collaboration function ID or a collaboration function name. When a single function is updated, the linked function management information is also updated along with the update. As a result, cooperation functions by a plurality of functions that could not cooperate with each other before the update become available after the update, and conversely, usable cooperation functions before the update become unavailable after the update It may be possible. Linked function information indicating the linked function that has become available after the update is registered in the linked function management information, and linked function information that indicates the linked function that can not be used after the update is deleted from the linked function management information, or , Registered as unusable information. The update of the cooperation function management information may be performed by the terminal device 10 or may be performed by an external device such as the server 16 or the like. When the update of the cooperation function management information is executed by the external device, the updated cooperation function management information is transmitted from the external device to the terminal device 10 and stored in the storage unit 24.

  In the case of linking a plurality of devices, for example, the linking function management information is information for managing a linking function using a plurality of functions possessed by a plurality of devices, and in order to identify each device used for the linking function It is information which shows matching with combination of device discernment information of, and cooperation function information. Also, a device having a plurality of output functions (a device that outputs an output to the user and having a plurality of output functions) and a sensing device that does not have an output function (an output to the user) When linking with a device that does not output data, the linkage function management information may include information indicating a combination of a specific sensing device and a specific output function among the plurality of output functions. . In this way, among the plurality of output functions, an output function to be used in combination with a specific sensing device is identified. As described above, when the device function management information is updated, the cooperation function management information is also updated along with the update. As a result, the cooperation function by a plurality of devices which could not cooperate with each other before the update becomes available after the update, and conversely, the usable cooperation functions before the update are not usable after the update It may be possible.

  The linkage function may be a function that can be executed by linking a plurality of different functions, or a function that can be executed by linking the same function. The cooperation function may be a function that can not be used before cooperation. The function that can not be used before the cooperation may be a function that can be used by using the same function among the functions possessed by the cooperation target device, or a function that can be used by combining different functions. May be For example, by linking an apparatus (printer) having a print function and an apparatus (scanner) having a scan function, a copy function as a cooperation function is realized. That is, the copy function is realized by linking the print function and the scan function. In this case, the copy function as the cooperation function is associated with the combination of the print function and the scan function. The cooperation function management information includes, for example, cooperation function information indicating a copy function as a cooperation function, device identification information for identifying an apparatus having a print function, and device identification information for identifying an apparatus having a scan function. The combinations are associated with each other.

  The category of the concept of the cooperation function may include a combination function in which a new function can be executed by linking a plurality of functions or a plurality of devices. For example, an extended display function as a combined function may be realized by combining a plurality of displays. As another example, a combination of a television and a recorder may realize a recording function as a combination function. The recording function may be a function of recording an image displayed on the television. Also, by combining a plurality of cameras, a shooting area expansion function as a combination function may be realized. The extended function is, for example, a function of connecting the shooting areas of the respective cameras and shooting. In addition, by combining the telephone with a translator or translation software, a translation call function (a function for translating a conversation via a telephone) may be realized as a combination function. As described above, in the category of the concept of cooperative functions, functions that can be realized by linking the same type of device and function with each other, and functions that can be achieved by linking different types of device and function with each other Is included.

  The control unit 26 controls the operation of each unit of the terminal device 10. The control unit 26 performs, for example, control of communication by the communication unit 18, control of display of information on the display unit of the UI unit 20, and the like. Further, the control unit 26 includes a specifying unit 28.

  The specifying unit 28 has a function of the device specified by the user, and a function of specifying a cooperative function that can be executed using a plurality of devices specified by the user. The identifying unit 28 is realized by, for example, an automatic response AI. Of course, the specification unit 28 may be realized by another program or the like instead of the automatic response AI.

  For example, the specifying unit 28 receives device identification information for identifying a device designated by a user, and indicates a function associated with the device identification information in the device function management information stored in the storage unit 24. Identify feature information. Thereby, the function that the device has is identified (identified).

  The control unit 26 controls notification (guidance) of information related to the function specified by the specifying unit 28 (for example, function information, explanatory information of the function, etc.). The control unit 26 may display information on the function on the display unit of the UI unit 20 as the notification, or may emit information on the function as voice information by using a speaker or the like. Further, the control unit 26 may control the execution of the function. The designation of the device by the user may be performed on the screen of the display unit of the UI unit 20 or may be performed by voice. For example, when an image linked to a device is displayed on the screen of the UI unit 20, the user may specify the device by specifying the image on the screen. As another example, the user may specify the device by voice regardless of whether or not the image linked to the device is displayed on the screen of the UI unit 20. For example, when the user utters a device name by voice, the voice is collected by a sound collection unit such as a microphone, and voice information indicating the device name is input to the terminal device 10. The identifying unit 28 determines that the device having the device name is designated by the user, and identifies the function. Thus, the notification of the operation and the function for specifying the device may be performed by display on the screen or may be performed by voice. The display on the screen may be combined with the sound. For example, the device may be specified by the user specifying an image associated with the device on the screen, and information indicating the function may be displayed on the screen. As another example, the device may be specified by the user specifying an image associated with the device on the screen, and information indicating the function may be emitted by voice. As yet another example, the device name may be input to the terminal device 10 as voice information, whereby the device may be designated, and information indicating the function may be displayed on the screen. As yet another example, the device name may be input to the terminal device 10 as voice information to designate the device, and information indicating the function may be issued by voice.

  The function identification processing may be performed by the server 16. In this case, device identification information is transmitted from the terminal device 10 to the server 16, and the specification unit 28 provided in the server 16 specifies function information indicating a function associated with the device identification information. When the specific process is performed by the server 16, device function management information is stored in the server 16. The information on the function may be transmitted from the server 16 to the terminal device 10 and displayed on the display unit of the UI unit 20, or may be emitted as audio information.

  Further, the specifying unit 28 receives device identification information for identifying each device to be cooperated, and the cooperation function associated with the combination of each device identification information in the cooperation function management information stored in the storage unit 24 Identify linkage function information that indicates As a result, the cooperation function that can be executed is identified (identified) by causing the functions of the respective devices to be cooperated to cooperate. The control unit 26 controls notification (guidance) of information (for example, cooperation function information, explanation information of the cooperation function, and the like) related to the specified cooperation function. The control unit 26 may cause the display unit of the UI unit 20 to display information related to the cooperation function, or may emit the information as audio information. In the same manner as described above, designation of each device to be cooperated may be performed on the screen or may be performed by voice.

  The specifying process of the cooperation function may be performed by the server 16. In this case, a plurality of pieces of device identification information are transmitted from the terminal device 10 to the server 16, and the cooperation function information indicating cooperation functions associated with the plurality of pieces of device identification information by the specifying unit 28 provided in the server 16 is It is identified. Note that, when the specific process is performed by the server 16, cooperative function management information is stored in the server 16. The information related to the cooperation function may be transmitted from the server 16 to the terminal device 10 and displayed on the display unit of the UI unit 20, or may be emitted as audio information.

  For example, when one device is specified, the function of the device is specified, and when a plurality of devices are specified, an executable cooperative function is specified using the plurality of devices.

  Further, the specifying unit 28 receives function information indicating each function used for the cooperation function, and specifies cooperation function information indicating a cooperation function associated with a combination of each function information in the cooperation function management information. Good. As a result, by linking the respective functions to be linked, an executable collaboration function is identified (identified). This specifying process is also performed by the server 16, and the result of the process may be displayed on the display unit of the UI unit 20, or may be issued as audio information.

  For example, the device is designated by the user sending the device information to the automatic response AI. When information on a plurality of devices of the user is sent to the automatic response AI, the identifying unit 28 identifies the plurality of devices as devices to be cooperated, and identifies an executable cooperation function using the plurality of devices. . In addition, when the user sends the information of one device to the automatic response AI, the specification unit 28 identifies the device and specifies the function of the device. The device information is device identification information for identifying the device, and is, for example, a device image associated with the device, character information indicating the device identification information, voice information, or the like. For example, when the user sends a device image associated with a device to the automatic response AI, or when the user inputs a character string indicating the device identification information of the device to the terminal device 10 and gives the automatic response AI, When the device identification information of the device is given by voice to the automatic response AI, the identification unit 28 (the identification unit 28 realized by the automatic response AI) identifies the device.

  In addition, the function (the single function and the cooperation function which an apparatus has) which can be used by the user may be managed for every user. This management may be performed by the terminal device 10 or may be performed by an external device such as the server 16. The functions available to the user are, for example, functions provided free to the user, functions provided for the user for a fee and purchased by the user, and the like. For each user, available function information (for example, function purchase history information) indicating functions available to the user may be created and managed. When the management of the available functions is performed by the terminal device 10, the available function information is stored in the storage unit 24, and when the management is performed by an external device such as the server 16, the available function information is stored in the external device Be done. Of course, since there are functions that can be used free of charge, additional update functions, and functions that are specially managed by the administrator, it is not necessary to judge the availability of the functions depending on the purchase status. The purchase process of the function is performed by an external device such as the server 16, for example.

  The available function management information is information for managing functions available to each user, and for example, user identification information for identifying the user and function information indicating functions available to the user (cooperation function It is the information which shows correspondence with (it may include information). As described above, the functions available to the user are, for example, a function provided free to the user, a function purchased by the user, etc., and may be a single function or a cooperative function. It is also good. The user identification information is, for example, user account information such as a user ID and a name. By referring to the available function management information, the functions available to each user are identified (identified). The available function management information may be updated, for example, each time the function is provided to the user (e.g., each time the function is provided to the user free of charge or paid).

  When the functions available to the user are managed, the specification unit 28 receives user identification information for identifying the user, and usable function management information (for example, information stored in the terminal device 10 or the server 16) Function information indicating each function associated with the user identification information). Thereby, the functions available to the user are identified (identified). The control unit 26 may cause the display unit of the UI unit 20 to display information regarding the specified function, or may emit the information as audio information. For example, the identification unit 28 receives device identification information and user identification information, identifies function information indicating a function associated with the device identification information in the device function management information, and further, specifies the function information in the usable function management information. The function information indicating the function associated with the user identification information is specified. As a result, the functions possessed by the device identified by the device identification information that are usable by the user identified by the user identification information are identified.

  The functions available to the user may be specified by an external device such as the server 16. In this case, user identification information is transmitted from the terminal device 10 to the server 16, and the specification unit 28 provided in the server 16 specifies function information indicating each function associated with the user identification information. When the specific process is performed by the server 16, the available function information is stored in the server 16. Information on each function available to the user may be transmitted from the server 16 to the terminal device 10 and displayed on the display unit of the UI unit 20 or may be emitted as audio information.

  Hereinafter, the configuration of the device 12 will be described in detail with reference to FIG. As an example, the device 12 is an image forming apparatus. The device 12 may be hereinafter referred to as an image forming apparatus 12. The configuration of the image forming apparatus 12 is shown in FIG.

  The communication unit 30 is a communication interface, and has a function of transmitting data to another device, and a function of receiving data from the other device. The communication unit 30 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function. The communication unit 30 supports, for example, one or more types of communication methods, and may communicate with the communication partner according to a communication method suitable for the communication partner (that is, a communication method compatible with the communication partner). . The communication method is, for example, infrared communication, visible light communication, Wi-Fi communication, proximity wireless communication, or the like. For example, the communication unit 30 may switch the communication method or the frequency band according to the other party of communication, or switch the communication method or the frequency band according to the surrounding environment.

  The image forming unit 32 has an image forming function. Specifically, the image forming unit 32 has at least one of a scan function, a print function, a copy function, and a facsimile function. By executing the scan function, the document is read and scan data (image data) is generated. By executing the print function, an image is printed on a recording medium such as paper. By executing the copy function, the original is read and printed on the recording medium. By executing the facsimile function, image data is transmitted by facsimile or received by facsimile. In addition, a cooperation function that uses a plurality of functions may be executed. For example, a scan transfer function combining the scan function and the transmission function (transfer function) may be performed. By executing the scan transfer function, a document is read to generate scan data (image data), and the scan data is transmitted to a transmission destination (for example, an external device such as the terminal device 10). Of course, this cooperation function is only an example, and another cooperation function may be executed.

  The storage unit 34 is a storage device such as a hard disk or a memory (for example, an SSD). The storage unit 34 indicates, for example, information indicating a command for image formation (for example, job information etc.), image data to be printed, scan data generated by executing a scan function, and addresses of other devices. Device address information, information (terminal address information) indicating the address of the terminal device 10, server address information of the server 16, various control data, various programs, and the like are stored. Of course, they may be stored in separate storage devices, or may be stored in one storage device.

  The UI unit 36 is a user interface unit, and includes a display unit and an operation unit. The display unit is, for example, a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel or a keyboard. Of course, a user interface (for example, a touch-type display, a device that electronically displays a keyboard and the like on the display, and the like) that doubles as a display unit and an operation unit may be used. The image forming apparatus 12 may not have the UI unit 36, or may have a hardware user interface unit (hardware UI unit) as hardware without including the display unit. The hardware UI unit is, for example, a hardware key specialized for numeric input (for example, a ten key), a hardware key specialized for direction indication (for example, a direction indication key), or the like.

  The control unit 38 controls the operation of each unit of the image forming apparatus 12.

  Hereinafter, the configuration of the server 16 will be described in detail with reference to FIG. The configuration of the server 16 is shown in FIG.

  The communication unit 40 is a communication interface, and has a function of transmitting data to another device, and a function of receiving data from the other device. The communication unit 40 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

  The storage unit 42 is a storage device such as a hard disk or a memory (for example, an SSD or the like). The storage unit 42 stores, for example, various data, various programs, terminal address information of the terminal device 10, device address information of each device, server address information of the server 16, and the like. Of course, they may be stored in separate storage devices, or may be stored in one storage device.

  When the server 16 manages the functions and linked functions of each device, the above-described device function management information and linked function management information are stored in the storage unit 42, and in the server 16, the device function management information and linked function management information are stored. It may be updated. In this case, when the terminal device 10 performs the function specifying process, the device function management information and the cooperation function management information (the information after the update when the information is updated) is transmitted from the server 16 to the terminal device 10 . The transmission may be performed periodically or at a designated timing. Thereby, the information stored in the terminal device 10 is updated. Of course, the specifying unit 28 of the terminal device 10 may specify the function or cooperation function of the device by referring to the device function management information or the cooperation function management information stored in the server 16. Further, the server 16 may execute a process of specifying a function.

  The control unit 44 controls the operation of each unit of the server 16. The control unit 44 controls, for example, communication by the communication unit 40.

  The control unit 44 may execute a process of purchasing a function and manage the history of the purchase. For example, when a paid function is purchased by a user, the control unit 44 may apply a charging process to the user.

  In addition, the control unit 44 may execute functions related to image processing, such as a character recognition function, a translation function, an image processing function, and an image forming function. Of course, the control unit 44 may execute a function related to processing other than image processing. By executing the character recognition function, characters in the image are recognized and character data indicating the characters is generated. By executing the translation function, characters in the image are translated into characters represented in a specific language, and character data indicating the translated characters is generated. The image is processed by executing the image processing function. The control unit 44 receives, for example, scan data generated by executing the scan function from the image forming apparatus 12, and performs image processing such as a character recognition function, a translation function, an image processing function, etc. on the scan data. You may perform functions related to The control unit 44 may receive image data from the terminal device 10 and execute each function on the image data. The character data and the image data generated by the control unit 44 are transmitted from the server 16 to the terminal device 10, for example. The server 16 may be used as an external device, and the cooperation function may be a function that uses functions of a plurality of devices including the server 16.

  As described above, the device function management information may be stored in the storage unit 42 of the server 16. In this case, the device function management information may not be stored in the storage unit 24 of the terminal device 10. Similarly, the cooperation function management information may be stored in the storage unit 42 of the server 16. In this case, the cooperation function management information may not be stored in the storage unit 24 of the terminal device 10. The control unit 44 of the server 16 includes the above-described identification unit 28. The control unit 44 may identify the device based on the device identification information, identify the function possessed by the device, or can execute using a plurality of functions. The linkage function may be specified. In this case, the terminal device 10 may not have the specifying unit 28.

  When the available function management information is created, the available function management information may be stored in the storage unit 42 of the server 16. In this case, the available function management information may not be stored in the storage unit 24 of the terminal device 10. The control unit 26 of the terminal device 10 may manage the purchase history of the function by the user. In this case, the control unit 44 of the server 16 may not have the management function.

  Furthermore, as another example, device function management information and linked function management information may be stored in the device such as the devices 12 and 14 or even if the device such as the devices 12 and 14 has the specifying unit 28 Good. That is, the processing by the identification unit 28 may be performed in the terminal device 10, in the devices such as the devices 12, 14 or in the server 16, or in another device. It may be

  In the first embodiment, as an example, device identification information is acquired and devices are identified by applying an augmented reality (AR) technology (augmented reality technology). For example, by applying the AR technology, the device identification information of the device to be used alone is acquired and the device is identified, and the device identification information of the device to be cooperated is acquired and the device to be cooperated is identified Be done. AR technology known as AR technology is used. For example, a marker type AR technology using a marker such as a two-dimensional barcode, a markerless type AR technology using an image recognition technology, a position information AR technology using position information, or the like is used. Of course, device identification information may be acquired and devices may be identified without applying AR technology. For example, if the device is a device connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device. Furthermore, when a device or terminal device having various wireless communication functions such as infrared communication, visible light communication, Wi-Fi, or Bluetooth is used, the ID of the device is acquired by using the wireless communication function, and the device is identified. It is also good.

  The process of acquiring device identification information will be described in detail below with reference to FIG. The case of acquiring device identification information of the image forming apparatus 12 will be described as an example. FIG. 5 schematically shows the appearance of the image forming apparatus 12. Here, processing for acquiring device identification information by applying the marker type AR technology will be described. The casing of the image forming apparatus 12 is provided with a marker 46 such as a two-dimensional bar code. The marker 46 is information in which device identification information of the image forming apparatus 12 is coded. The user activates the camera 22 of the terminal device 10, and the camera 22 captures an image of the marker 46 provided in the image forming apparatus 12 to be used. Thereby, image data representing the marker 46 is generated. The control unit 26 of the terminal device 10 extracts device identification information by applying the decoding process to the marker image represented in the image data. As a result, the image forming apparatus 12 to be used (the image forming apparatus 12 having the photographed marker 46) is identified. The specifying unit 28 of the terminal device 10 specifies, in the device function management information, function information indicating a function associated with the extracted device identification information. Thus, the function of the image forming apparatus 12 to be used is identified (identified).

  The control unit 44 of the server 16 may extract the device identification information by applying the decoding process to the image data representing the marker 46. In this case, the image data is transmitted from the terminal device 10 to the server 16, and the server 16 applies decoding processing to the image data. When the specifying unit 28 is provided in the server 16, the function associated with the device identification information may be specified in the server 16.

  The marker 46 may include function information indicating the function of the image forming apparatus 12 in a coded form. In this case, by applying the decoding process to the image data representing the marker 46, the device identification information of the image forming apparatus 12 is extracted, and the function information indicating the function of the image forming apparatus 12 is also extracted. Thereby, the image forming apparatus 12 is identified (identified), and the function of the image forming apparatus 12 is identified (identified). This decoding process may be performed by the terminal device 10 or may be performed by the server 16.

  Further, by capturing the markers of each of a plurality of devices, device identification information of each device is acquired.

  When the device identification information is acquired by applying the markerless AR technology, for example, the user captures all or part of the appearance of the device to be used (for example, the image forming device 12) by the camera 22 of the terminal device 10. Of course, it is useful to capture and obtain information for identifying a device, such as the name of a device to be used (for example, a product name), a model number, and an asset management number from the appearance. By photographing, appearance image data representing all or part of the appearance of the device to be used is generated. The control unit 26 of the terminal device 10 identifies the device to be used based on the appearance image data. For example, in the storage unit 24 of the terminal device 10, appearance image association information indicating correspondence between appearance image data representing all or a part of the appearance of the device and device identification information of the device for each device. It is memorized. For example, the control unit 26 compares the appearance image data with the appearance image data included in the appearance image association information, and specifies the device identification information of the device to be used based on the comparison result. For example, the control unit 26 extracts the feature of the appearance of the device to be used from the appearance image data, and in the appearance image data group included in the appearance image association information, an appearance showing a feature identical or similar to the feature of the appearance. The image data is identified, and the device identification information associated with the appearance image data is identified. Thereby, the device to be used (the device photographed by the camera 22) is identified. As another example, if the device name (for example, a product name) or a model number is captured and appearance image data representing the name or model number is generated, the name or model number represented in the appearance image data is used. The device to be used may be identified. The specifying unit 28 of the terminal device 10 specifies function information indicating each function associated with the specified device identification information in the device function management information. Thus, the function of the device to be used (for example, the image forming apparatus 12) is specified.

  The control unit 44 of the server 16 compares appearance image data representing all or part of the appearance of the device to be used (for example, the image forming apparatus 12) with each appearance image data included in the appearance image association information. The device identification information of the device to be used may be specified based on the comparison result. The appearance image correspondence information is stored in the storage unit 42 of the server 16. In this case, the control unit 44 of the server 16 specifies the device identification information of the device to be used by referring to the appearance image correspondence information.

  The device identification information of each device is acquired by capturing all or part of the appearance of each of the plurality of devices.

  When acquiring device identification information by applying position information AR technology, for example, position information indicating a position where an apparatus (for example, the image forming apparatus 12) is installed is available by using a GPS (Global Positioning System) function. It is acquired. For example, each device has a GPS function, and acquires device position information indicating the position of the device itself. The terminal device 10 outputs information indicating an acquisition request for device position information to a device to be used, and receives device position information of the device from the device as a response to the acquisition request. The control unit 26 of the terminal device 10 identifies the device to be used based on the device position information. For example, the storage unit 24 of the terminal device 10 stores, for each device, position association information indicating an association between device position information indicating the position at which the device is installed and device identification information of the device. ing. The control unit 26 specifies the device identification information associated with the device location information in the location association information. Thereby, the device to be used is identified (identified). The specifying unit 28 of the terminal device 10 specifies function information indicating each function associated with the specified device identification information in the device function management information. Thereby, the function of the device to be used (for example, the image forming apparatus 12) is identified (identified).

  The control unit 44 of the server 16 may specify, in the position association information, device identification information associated with the position information of the device to be used. The position association information is stored in the storage unit 42 of the server 16. In this case, the control unit 44 of the server 16 specifies the device identification information of the device to be used by referring to the position association information.

  Device location information of each of a plurality of devices is acquired, and device identification information of each device is specified based on each device location information.

  The devices may be identified using a plurality of identification techniques. For example, the device may be identified using a plurality of technologies selected from marker type AR technology, markerless type AR technology, and position information AR technology. Other identification techniques may also be used to identify devices when one identification technique fails to identify the device. For example, if the device can not be identified by the marker type AR technology or the markerless type AR technology, the device may be identified using the position information AR technology.

  The device system according to the first embodiment will be described in more detail below.

  The device function management information will be described in detail with reference to FIG. FIG. 6 shows an example of the device function management table as the device function management information. In the device function management table, as an example, a device ID, information indicating a device name (for example, type of device), information (function information) indicating a function of the device, and an image ID are associated with each other. There is. The device ID and the device name correspond to an example of device identification information. The image ID is an example of image identification information for identifying a device image associated with a device. The device function management table may not include the image ID. For example, the device whose device ID is “B” is a multifunction peripheral (image forming apparatus having a plurality of image forming functions) and has functions such as a print function and a scan function. The device is associated with an image ID for identifying a device image associated with the device. For example, data of a device image linked to a device may be stored in the terminal device 10, may be stored in the server 16, or may be stored in another device.

  For example, by applying the AR technology, a device ID for identifying a device to be used is acquired. The specifying unit 28 of the terminal device 10 specifies the device name, the function, and the image ID associated with the device ID by referring to the device function management table. Thus, the device to be used is identified. Information indicating a device name or a device image may be displayed on the display unit of the UI unit 20. Of course, the image itself taken by the camera 22 may be displayed on the display unit of the UI unit 20. In addition, when a device image (for example, an image captured by the camera 22 or an image schematically representing the device) associated with the device is designated by the user in the UI unit 20, information on the function of the device (for example, , Function information, function explanation information, etc.) may be displayed on the display unit of the UI unit 20.

  Hereinafter, the linked function management information will be described in detail with reference to FIG. FIG. 7 shows an example of the cooperation function management table as cooperation function management information. In the cooperation function management table, as an example, a combination of device IDs, information indicating the device name of cooperation target (for example, type of each device), and information indicating cooperation function (cooperation function information) are mutually associated. ing. For example, the device with the device ID “A” is a PC (personal computer), and the device with the device ID “B” is a multifunction peripheral. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” and a “print function” are realized as linked functions. The “scan transfer function” is a function of transferring image data generated by scanning by the multifunction peripheral (B) to the PC (A). The “print function” is a function of transmitting data (for example, image data and document data) stored in the PC (A) to the multifunction peripheral (B) and printing the data in the multifunction peripheral (B). Note that an executable cooperation function may be defined by using three or more devices. In this case, a combination of three or more devices is associated with the cooperation function, and the association is registered in the cooperation function management table.

  Hereinafter, the operation of the terminal device 10 will be described in detail by giving a specific example.

  An example of the screen displayed on the terminal device 10 is shown in FIG. The control unit 26 of the terminal device 10 causes the display unit of the UI unit 20 to display the screen 48 and causes the screen 48 to display various types of information. The screen 48 is an interface (interaction response screen) for the user to interact with the automatic response AI. On the screen 48, information (for example, a character string, an image, and the like) input to the terminal device 10 by the user and information (for example, a character string, an image, and the like) indicating the contents of the automatic response AI are displayed. The interaction between the user and the automatic response AI may be a so-called chat type interaction (i.e., a real time type interaction, in other words, a format in which the user and the automatic response AI proceed while the dialog is speaking). For example, when a program (for example, a chat program) for interacting with the automatic response AI is executed, a screen 48 (interaction response screen) for the dialog is displayed on the display unit of the UI unit 20.

  The screen 48 is provided with a display area for each dialogue person, and the speech contents of each dialogue person may be displayed in the display area for oneself. For example, the display area 48a is an area where the contents of the automatic response AI are displayed, and the display area 48b is an area where the contents of the user's speech (information input to the terminal device 10 by the user) are displayed. In the example shown in FIG. 8, a display area 48a for the automatic response AI is provided on the left side of the screen 48, and a display area 48b for the user is provided on the right side. Of course, this display example is merely an example, and the arrangement relationship of the display areas may be reversed, each display area may be provided in the vertical direction, or all or part of the display areas may overlap. The latest utterance content may be displayed in a specific area (for example, the center area) of the screen 48, and past utterance content may be displayed around it. In addition, each display area may have the same size as each other, or may have different sizes from one another, or may be enlarged or reduced according to the amount of one speech content or the number of times of speech May be Control of the setting and the size of the display area is performed by the control unit 26.

  In the example shown in FIG. 8, the interlocutor is only one user and one automatic response AI, but multiple users and multiple automatic responses AI may participate in the interaction. In this case, a display area for each communicator may be provided on the screen 48. For example, in the case where two users and one automatic response AI participate in the dialogue, the screen 48 is provided with three display areas, and the content of each dialogue person's speech is displayed in his own display area. For example, when communication is performed between the plurality of terminal devices 10 and the dialog via the screen 48 is performed among the plurality of users, the speech contents of the plurality of users are displayed on the screen 48. The same is true when multiple automatic responses AI participate in the dialogue. A program related to a plurality of automatic responses AI may be installed in the same terminal device 10, and the plurality of automatic responses AI may be stored in the same terminal device 10. A program related to each of the plurality of automatic responses AI May be installed on separate terminal devices 10, and each automatic response AI may be stored on separate terminal devices 10. For example, the message content of the automatic response AI stored in the terminal device 10 which is the own device is displayed on the screen 48, and the message content of the automatic response AI stored in another terminal device 10 is further displayed. Ru. The automatic response AI stored in another terminal 10 participates in the dialogue, for example via the communication path N.

  For example, when the chat program is activated, the control unit 26 of the terminal device 10 causes the display unit of the UI unit 20 to display the screen 48. In the display area 48a for the automatic response AI, an image 50 linked to the automatic response AI is displayed, and in the display area 48b for the user, an image 52 linked to the user is displayed. The image 50 is an image for identifying the automatic response AI, and the image 52 is an image for identifying the user. In place of the image 50 or together with the image 50, a character string indicating the automatic response AI (a character string for identifying the automatic response AI) may be displayed. Similarly, instead of the image 52 or together with the image 52, a character string (for example, a character string for identifying the user such as a name, a user ID, a handle name, a nickname, etc.) indicating the user may be displayed.

  Thereafter, an interaction is performed between the user and the automatic response AI. In the dialog, the user's speech content may be input to the terminal device 10 by the user operating the UI unit 20 and inputting information such as a character string or an image, or may be input by voice. In addition, the user's statement content may or may not be displayed on the screen 48 as information such as a character string or an image. Similarly, the statement content of the automatic response AI may be displayed on the screen 48 as information such as a character string or an image, or may be emitted as a voice without being displayed or displayed. In the following, as an example, it will be described that the contents of each dialogue person's speech are displayed on the screen 48, but the contents of the speech may be spoken.

  When the dialog advances on the screen 48, the screen 48 scrolls, the past statement contents which can not be displayed on the screen 48 disappear from the screen 48, and the latest statement contents and the latest statement contents are displayed on the screen 48 Be done.

  When the chat program is started, the automatic response AI issues a general question (a question asking the user's request) such as "How did it happen?" 48 (not shown in FIG. 8). Of course, the contents of the automatic response AI may be uttered as voice instead of or together with the text. Since the content of the message is the content of the automatic response AI, the control unit 26 displays the content of the message in the display area 48a as the content of the message linked to the image 50 of the automatic response AI. The same is true for the following dialogues.

  Next, when the user requests the cooperation function by saying "I want to execute the cooperation function" or the like, a character string indicating the request is displayed on the screen 48 as the user's speech content (shown in FIG. 8). Absent). For example, the user may input information indicating the content of the message to the terminal device 10 by operating the screen 48 and inputting a character string, or information on the content of the message may be input to the terminal device 10 by voice. May be Since the content of the message is the content of the user's message, the control unit 26 displays the content of the message in the display area 48b as the content of the message linked to the image 52 of the user. The same is true for the following dialogues.

  The automatic response AI identifies the user's request by analyzing the user's speech content, and creates an answer such as an answer to the request. The control unit 26 displays a character string indicating the reply in the display area 48a as the speech content 54 of the automatic response AI. For example, the statement content 54 such as "which device do you want to use?" Is displayed in the display area 48a as the statement content of the automatic response AI. The automatic response AI may search information using the Internet or the like, analyze the search result, or may acquire information on the devices 12 and 14 and analyze the information. The automatic response AI may reflect the analysis result in the reply. The automatic response AI may manage the usage history of various applications by the user, the operation history of the terminal device 10 by the user, the usage history of the devices 12 and 14 by the user, and the like. Also, the automatic response AI may obtain information from another automatic response AI by communicating with the automatic response AI stored in another device. The automatic response AI may reflect the information managed by itself or the acquired information in the reply.

  When the user designates a device in response to the above inquiry, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the device as a device to be cooperated. For example, as shown in FIG. 8, the user designates the device image 56 linked to the device, and arranges the device image 56 at a position where the user's speech content is displayed, whereby the control unit 26 displays the device image Send 56 to the automatic response AI as the user's message content. In response to this operation, the identification unit 28 identifies the device associated with the device image 56 as the device to be cooperated. In the example shown in FIG. 8, the operation of causing the user to display the device image 56 in the display area 48 b corresponds to the action of the user emitting the content of the statement. The user may operate the UI unit 20 to input a character string indicating the device to the terminal device 10 as the user's speech content. In this case, the identification unit 28 identifies the device indicated by the character string as the device to be cooperated. In addition, information on the device to be cooperated may be input to the terminal device 10 by voice as the content of the user's speech. Note that the user may specify the type of device without specifying each device. In this case, the type of the device is identified as the type of the device to be cooperated. For example, the user may designate the device itself as the multifunction machine B or may designate the type as the multifunction machine.

  The device image 56 is an image associated with the multifunction peripheral B (image forming apparatus) as a device. The multifunction device B is a device identified by the identification unit 28 by using the above AR technology (marker type AR technology, markerless type AR technology, position information AR technology) or other identification technology. Of course, the device image 56 may not be an image associated with the multifunction peripheral B itself, but an image associated with a type of multifunction device.

  For example, the control unit 26 displays the stock area 58 in the display area 48b for the user, and displays the device image in the stock area 58. The stock area 58 is an area in which information on one or more candidates for cooperation is displayed. In the example illustrated in FIG. 8, an apparatus image is displayed as an example of the candidate information. For example, the control unit 26 causes the stock area 58 to display an apparatus image associated with the already identified apparatus. In the example shown in FIG. 8, in the stock area 58, device images 56, 60, 62 are displayed. The device image 60 is an image associated with a PC (A) as a device. The device image 62 is an image associated with the projector C as a device. When the device is identified in this way, the control unit 26 displays the device image associated with the device in the stock area 58. Of course, the control unit 26 may display, in the stock area 58, an apparatus image associated with the type of the device without being associated with the device itself. For example, the control unit 26 associates the device image 56 attached to the type of the multifunction machine B itself without being tied, the device image 60 attached to the kind PC without being tied to the PC (A) itself, and the projector C itself An apparatus image 62 associated with a type of projector may be displayed in the stock area 58.

  For example, information on a device once identified is stored in the storage unit 24. After that, even if the identification process for the device is not performed again, the device image associated with the device is displayed in the stock area 58. Of course, the information on the identified device may be deleted from the storage unit 24 by the operation of the user. In this case, the device image associated with the device is also deleted. As another example, after the cooperation function is executed using the identified device, or after a predetermined time has elapsed from the time when the device was identified, the information on the identified device is stored in the storage unit 24. May be deleted from

  The device image 56 is an image (it may be an image having a size at the time of shooting, or an enlarged or reduced image) generated by shooting the MFP B with the camera 22. It may be a schematic image (for example, an icon) linked to the multifunction machine B. Similarly, the device image 60 may be an image generated by photographing the PC (A), or may be a schematic image tied to the PC (A). The same applies to the device image 62.

  When a schematic image is used, the identifying unit 28 identifies a schematic image associated with the identified device by referring to the device function management table shown in FIG. The schematic image is displayed as a device image. The data of the schematic image may be stored in the terminal device 10 or may be stored in an external device such as the server 16.

  In addition, when using the image data generated by photographing the device, the appearance of the current device itself (for example, the appearance in which the scratch, the note, the sticker attached to the device, etc. are reflected) is reflected in the image As a result, it is possible for the user to visually recognize the difference from other similar devices. In addition, since the device image associated with the device itself (for example, an image obtained by photographing, an icon associated with the device itself, etc.) is an image unique to the device, the device image is sent to the automatic response AI. Thus, the automatic response AI can identify the device itself as the device to be cooperated. On the other hand, if the device image is an image not tied to the device itself but tied to the device type, the automatic response AI can identify the type of the device from the device image, but may identify the individual device itself Can not. For example, when the device image 56 is an image tied to the multifunction device B, the automatic response AI can identify the multifunction device B itself from the device image 56 as a device to be cooperated. On the other hand, when the device image 56 is an image associated with the type of the multifunction device, the automatic response AI identifies the type of the multifunction device from the device image 56 as the type of the device to be cooperated. In the following, each device image is described as an image associated with an individual device itself. That is, the device image 56 is an image associated with the multifunction machine B itself, the device image 60 is an image associated with the PC (A) itself, and the device image 62 is an image associated with the projector C itself. .

  Also, when a device is identified, information indicating the name of the device may be displayed on the screen 48. For example, the character string “MFP” may be displayed together with the device image 56.

  In addition, when there are other device images that can not be displayed in the stock area 58, as indicated by the arrow 64, the user scrolls the device image displayed in the stock area 58 to display the other images. The device image of is displayed in the stock area 58.

  The user selects an image (device image 56 in the example shown in FIG. 8) associated with the device to be cooperated from the stock area 58, as indicated by the arrow 66, and moves the image to the outside of the stock area 58. Can be sent to the automatic response AI as the user's speech content. For example, the user moves the device image 56 from the stock area 58 to a position where the user's utterance content is displayed by a drag operation, and performs a drop operation at the display position of the utterance content. By this operation, the control unit 26 sends the device image 56 to the automatic response AI as the user's statement content.

  Further, in the example illustrated in FIG. 8, following the device image 56, the user selects the device image 60 from the stock area 58, and moves the device image 60 to the display position of the user's statement content. By this operation, the device image 60 is sent to the automatic response AI as the user's speech content.

  When the user sends information of a plurality of devices to the automatic response AI on the screen 48, that is, when the information of a plurality of devices is spoken as information of a device to be cooperated, the specifying unit 28 is shown in FIG. In the linked function management table, the linked function associated with the plurality of devices is specified. The control unit 26 displays information on the cooperation function in the display area 48a as the contents of the automatic response AI.

  In the above example, the device images 56 and 60 are sent to the automatic response AI as the information of the device to be cooperated. In this case, the identification unit 28 identifies the MFP B linked to the device image 56 and the PC (A) linked to the device image 60 as the device to be cooperated. Alternatively, the device image 56 may be sent to the automatic response AI first, and then the device image 60 may be sent to the automatic response AI. Alternatively, the device image 60 may be sent to the automatic response AI, and then the device image 56 may be automatically replied. May be sent to AI.

  For example, when the user selects and extracts an image (in the above example, the device images 56 and 60 in the above example) from the stock area 58 by the indicator (for example, the user's finger, pen, stylus, etc.) Is sent to the automatic response AI. For example, the device image is moved from the stock area 58 to the display position of the statement content of the user by the drag and drop operation, whereby the device image is sent to the automatic response AI as the statement content of the user. The control unit 26 detects the touch of the indicator on the screen 48 and detects the movement of the indicator on the screen 48.

  As described above, when the device images 56 and 60 are sent to the automatic response AI and the multifunction peripheral B and the PC (A) are identified as the cooperation target devices, the control unit 26 is identified as the cooperation target devices. The device information is displayed in the display area 48a as the statement content 68 of the automatic response AI. When the user approves the comment content 68 (for example, when the user issues a statement content 70 such as “Yes.”), The cooperation function that can be executed using the PC (A) and the multifunction peripheral B The information is displayed on the screen 48.

  The identifying unit 28 is illustrated in FIG. 7 at the timing when the device images 56 and 60 are sent to the automatic response AI as information of the device to be cooperated, or at the timing when the user approves the message content 68. In the cooperation function management table, the cooperation functions (for example, “scan transfer function” and “print function”) associated with the combination of the PC (A) and the MFP B are specified. As a result, an executable cooperative function is identified using the PC (A) and the multifunction machine B. The “scan transfer function” is a function of transferring image data generated by scanning by the multifunction machine B to the PC (A). The “print function” is a function of transferring image data stored in the PC (A) to the multifunction device B and printing the image data on the multifunction device B. Information on the cooperation function specified in this manner is displayed on the screen 48 as the content of the automatic response AI. In addition, the server 16 may perform identification processing of an apparatus or a cooperation function.

  FIG. 9 shows a screen 48 continued from FIG. When the cooperation function is specified as described above, the control unit 26 of the terminal device 10 displays information on the cooperation function in the display area 48a as the statement content 72 of the automatic response AI.

  When the user designates the cooperation function according to the statement content 74 and gives an execution instruction to the statement content 72, the control unit 26 (for example, the control unit 26 realized by the automatic response AI) receives the execution instruction. And transmits information (execution instruction information) indicating the execution instruction to the device to be cooperated. In addition, the control unit 26 displays information indicating that the execution instruction has been received in the display area 48a as the speech content 76 of the automatic response AI. As another example, a button image for instructing execution of the cooperation function may be displayed on the screen 48, and the user may give an execution instruction of the cooperation function by pressing the button image. When the PC (A) and the multifunction peripheral B are selected as devices to be cooperated, execution instruction information is transmitted to the PC (A) and the multifunction peripheral B. The PC (A) and the multifunction machine B having received the execution instruction information execute the cooperation function (the cooperation function designated by the user) indicated in the execution instruction information. For example, when an execution instruction of “print function” is given as a cooperation function, the PC (A) transmits image data to be printed to the multifunction device B, and the multifunction device B uses the image data as a recording medium Print on top.

  When the user designates the cooperation function according to the statement content 74, the control unit 26 (automatic response AI) displays a screen for controlling the cooperation target device without giving an execution instruction of the cooperation function to the cooperation target device. May be displayed on the UI unit 20. The screen 78 is shown in FIG. The screen 78 is, for example, a screen (a screen for remote control) for controlling the PC (A) and the multifunction peripheral B by the terminal device 10. On the screen 78, for example, a button image for instructing execution of a cooperation function (for example, a button image for instructing execution of a “scan transfer image”, a button image for instructing execution of a “print function”) Is displayed. When the user presses a button image on the screen 78, a linked function linked to the button image is executed. In addition, various parameters may be set on the screen 78.

  Further, FIG. 11 shows another screen for remote control. The screen 78 includes a screen 78 a for controlling the MFP B and a screen 78 b for controlling the PC (A). On the screen 78a, for example, a button image for instructing execution of a single function of the multifunction machine B (for example, a button image for instructing execution of a copy function, a button image for instructing execution of a print function, etc. ) May be displayed. Further, on the screen 78a, setting and the like of various parameters used in the multifunction machine B may be performed. For example, data (files) stored in the PC (A) and storage locations (folders) are displayed on the screen 78 b. Further, various parameters used in the PC (A) may be set on the screen 78b. In addition, on the screens 78a and 78b, setting or execution instruction of a cooperation function (for example, a "scan transfer function" or a "print function" as the cooperation function) may be made possible.

  The device image displayed in the stock area 58 in FIG. 8 may be edited by the user, or the information on the device associated with the device image may be edited by the user. For example, when the user designates an apparatus image displayed in the stock area 58 and gives an editing instruction, the control unit 26 causes the display unit of the UI unit 20 to display an editing screen. For example, when the user long-presses the device image (for example, when the length of time during which the device image is pressed becomes equal to or more than the threshold), the control unit 26 presses the button image for displaying the editing screen. In the case where the user inputs an editing instruction into the terminal device 10 by voice or the like, the editing screen is displayed on the display unit of the UI unit 20. On the editing screen, editing of the device image is possible. For example, it is also possible to cut out a partial image linked to a part of the device from the device image. In addition, it is also possible to edit information on the device associated with the device image.

  In addition, the control unit 26 may display an apparatus image associated with an apparatus for which communication with the terminal apparatus 10 has been established in the stock area 58 or the apparatus image registered in the terminal apparatus 10 may be used as a string. The attached device image may be displayed in the stock area 58.

  When each device is identified by the marker type AR technology or the markerless type AR technology, the respective devices may be separately photographed and identified, or may be photographed together and identified. For example, the MFP B is photographed in a state in which the MFP B is included in the imaging area of the camera 22, and then the PC (A) is imaged in a state in which the PC (A) is included in an imaging area of the camera 22. Then, the MFP B and the PC (A) are sequentially identified. The respective devices to be cooperated are arranged at positions close to each other, and both of the multifunction peripheral B and the PC (A) are not necessarily included together in the imaging region of the camera 22. Although there are cases where it is possible to cope with this by changing the angle of the imaging area, enlarging or reducing the imaging area, etc., there are cases where such an operation can not cope. In this case, each device can be identified by photographing a plurality of times.

  As another example, a device to be cooperated may be preset as a basic cooperative device. For example, it is assumed that the multifunction device B is set in advance as a basic cooperation device. The device identification information of the basic cooperation device may be stored in advance in the storage unit 24 of the terminal device 10. The user may designate the basic cooperation device using the UI unit 20. When the basic cooperation device is set, the user sends the device image attached to the cooperation target device other than the basic cooperation device to the automatic response AI as the message content. As a result, the device to be cooperated is identified (identified), and the cooperation function that can be executed using the basic cooperative device and the device is identified (identified).

  Hereinafter, an operation when executing the cooperation function will be described. For example, when executing the cooperation function, a connection request is transmitted from the terminal device 10 to each device to be cooperated, and the terminal device 10 and each device are connected. Hereinafter, this connection process will be described with reference to FIG. FIG. 12 is a sequence diagram showing the process.

  In the terminal device 10, when the MFP B and PC (A) are designated by the user as the devices to be cooperated (that is, the information of the MFP B and PC (A) is sent to the automatic response AI as the user's interactive content. Terminal device 10) transmits the information indicating the connection request to the cooperation target devices (MFP and PC (A)) (S01). Control of this transmission may be performed by an automatic response AI. For example, when the address information of each device to be cooperated is stored in the server 16, the terminal device 10 acquires the address information of each device from the server 16. When the device identification information includes the address information, the control unit 26 of the terminal device 10 acquires the address information from the device identification information of each device. Acquisition processing of address information may be performed by automatic response AI. Of course, the storage unit 24 of the terminal device 10 may store the address information of each device. The terminal device 10 may acquire the address information of each device by another method. The terminal device 10 transmits information indicating a connection request to each device to be cooperated using the address information of each device to be cooperated (for example, the MFP B and the PC (A)).

  The MFP B and the PC (A) having received the information indicating the connection request determine whether the connection with the terminal device 10 is permitted or not (S02). For example, if the multifunction machine B and the PC (A) correspond to devices for which connection is not permitted, or if the number of devices requesting connection exceeds the upper limit, the connection is not permitted. . When the connection from the terminal device 10 is permitted, the change operation may be prohibited so that the setting information unique to each of the multifunction peripheral B and the PC (A) is not changed from the terminal device 10. For example, the change of the color parameter of the multifunction peripheral B, the setting time when shifting to the power saving, or the like may be prohibited. This improves the security of the device to be linked. As another example, when devices are linked, the change in setting information may be restricted as compared to the case where the devices are used alone without linking. For example, as compared with the case of using the device alone, it may be permitted to change less setting items. Further, browsing of personal information of other users such as operation history may be prohibited. This improves the security of the user's personal information.

  Result information indicating permission or non-permission of connection is transmitted from the multi-functional peripheral B and PC (A) to the terminal device 10 (S03). When the connection to the MFP B and the PC (A) is permitted, communication is established between the terminal device 10 and the MFP B, and further, communication is established between the terminal device 10 and the PC (A). Be done.

  Next, in the terminal device 10, the user instructs execution of the cooperation function (for example, "print function") (S04). In accordance with the instruction, the terminal device 10 transmits information (execution instruction information) indicating an execution instruction of the cooperation function to each device (MFP B and PC (A)) to be cooperated (S05). Control of this transmission may be performed by an automatic response AI. The execution instruction information transmitted to the MFP B includes information (for example, job information) indicating a process to be executed by the MFP B, and the execution instruction information transmitted to the PC (A) includes the PC The information (for example, job information) indicating the process to be executed in A) is included.

  The MFP B and the PC (A) having received the execution instruction information execute the cooperation function (for example, the “print function”) in accordance with the execution instruction information (S06). Communication is established between MFP B and PC (A) when the cooperation function to be executed includes processing in which data transmission / reception is performed between MFP B and PC (A). . For example, the execution instruction information transmitted to the MFP B includes the address information of the PC (A), and the execution instruction information transmitted to the PC (A) includes the address information of the MFP B. There is. Communication is established between the MFP B and the PC (A) using the address information. After the communication is established, the PC (A) transmits the image data to be printed to the multifunction device B, and the multifunction device B prints the image data on a sheet as a recording medium.

  When the execution of the cooperation function ends, information indicating the completion of the execution of the cooperation function is transmitted from the MFP B and the PC (A) to the terminal device 10 (S07). Information indicating that the execution of the cooperation function has been completed is displayed on the UI unit 20 of the terminal device 10 (S08). If the information indicating the completion of execution is not displayed even if the time set in advance has passed since the time of giving the execution instruction, the control unit 26 of the terminal device 10 causes the UI unit 20 to display the information indicating an error, The execution instruction information or the information indicating the connection request may be transmitted to the MFP B and the PC (A) again.

  Next, the user confirms whether or not to cancel the cooperation state of the MFP B and the PC (A) (S09), and a process according to the cancellation is executed (S10). When canceling the cooperation state, the user uses the terminal device 10 to give a cancellation instruction. Thus, the communication between the terminal device 10 and the multifunction peripheral B is released, and the communication between the terminal device 10 and the PC (A) is released. Similarly, the communication between the multifunction peripheral B and the PC (A) is released. If the cooperation state is not released, an execution instruction may be continuously given.

  Moreover, you may increase the apparatus of cooperation object. For example, an image linked to the third device may be sent as an interactive content to the automatic response AI, and three devices including the multifunction machine B and the PC (A) may be linked. Information indicating that the MFP B and the PC (A) have already been identified as devices to be cooperated is stored in the terminal device 10 or the server 16.

  The device identification information indicating the device to be cooperated and the cooperation function information indicating the executed cooperation function may be stored in the terminal device 10, the server 16, or the like. For example, user account information (user identification information) of a user who uses the terminal device 10 is acquired, and the user account information, device identification information indicating a device to be cooperated, and cooperation function information indicating an executed cooperation function The historical information which shows correspondence of, may be created, and may be memorized by terminal unit 10, server 16, etc. The history information may be created by the terminal device 10 or may be created by the server 16. By referring to the history information, it is specified what kind of cooperation function has been executed using what kind of device group.

  Note that the devices to be linked (for example, multifunction machine B and PC (A)) store, as history information, user account information of the user who requested connection, and terminal identification information indicating the terminal device 10 of the connection request source. It is also good. By referring to this history information, the user who has used the device is identified. For example, when identifying a user who has used the device when the device is broken, or when performing a charging process for consumables or the like, the user may be identified using history information. The history information may be stored in the terminal device 10 or the server 16 or may be stored in another device.

  The user account information is stored in advance in, for example, the storage unit 24 of the terminal device 10, and the control unit 26 of the terminal device 10 functions as an example of a user identification unit, and reads user account information of the user from the storage unit 24. , Identify the user who uses the terminal device 10. When user account information of a plurality of users is stored in the storage unit 24, the user uses the terminal device 10 to specify his own user account information. Thereby, the user account information of the user is read, and the user is identified. As another example, the control unit 26 of the terminal device 10 may identify the user by reading user account information of the user who has logged in to the terminal device 10. As still another example, when only one piece of user account information is stored in the same terminal device 10, the control unit 26 of the terminal device 10 may identify the user by reading the user account information. . When the user account is not set and the user account information is not created, the initial setting is performed, whereby the user account information is created.

  When the user is identified, an image (for example, a picture of the user (for example, a face picture etc.) representing the user), an icon that is not an image of the user but tied to the user, etc. It may be displayed as a tied image 52. In addition, user identification information such as the identified user's name, user ID, handle name, and nickname may be displayed on the screen 48 as a character string.

  The usage history of the collaboration function may be managed for each user, and information indicating the collaboration function used in the past by the user indicated by the read user account information may be displayed on the UI unit 20 of the terminal device 10. The information indicating the usage history may be stored in the terminal device 10 or may be stored in the server 16. In addition, information indicating a collaboration function being used may be displayed at a frequency higher than a preset usage frequency. By providing such a shortcut function, the time and effort of the user's operation regarding the cooperation function can be reduced.

  As described above, according to the first embodiment, on the screen 48 for interacting with the automatic response AI, the information on the device to be cooperated is sent as the interactive content of the user to the automatic response AI, thereby the information regarding the collaboration function Is provided. As described above, the information on the cooperation function is provided by the simple method using the interaction with the automatic response AI.

  When three or more device images are sent as the interactive content to the automatic response AI, information on a collaboration function that can be executed using three or more devices is provided.

  In addition, when information of a plurality of devices is sent to the automatic response AI as the user's utterance content within a predetermined time limit, the specifying unit 28 specifies a collaboration function that can be executed using the plurality of devices. If the information of one device is sent to the automatic response AI as the user's utterance content within the time limit, the function (single function) of the device may be specified. In this case, information on the linked function or the single function specified by the specifying unit 28 is displayed on the display unit of the UI unit 20. The start time of the time limit may be, for example, the time when the automatic response AI inquires the user of the device to be cooperated, as in the statement content 54, or the user is the first device after the statement content 54 occurs. It may be the point when the image is sent to the automatic response AI.

  For example, in the example shown in FIG. 8, the user sends only the device image 56 as the speech content to the automatic response AI within the time limit from the time when the automatic response AI inquires the user of the device to be cooperated. In this case, the specifying unit 28 specifies the function (single function) associated with the multifunction peripheral B in the device function management table shown in FIG. The control unit 26 causes the display area 48a to display information about the single function as the content of the automatic response AI. In this case, as in the process shown in FIG. 10, information indicating a connection request is sent to MFP B, and when the connection is permitted, MFP B performs the single function designated by the user. Be done.

  On the other hand, when the user sends the device image 56, 60 as the speech content to the automatic response AI within the time limit from when the automatic response AI inquires the user of the device of the cooperation function as in the speech content 54, The identifying unit 28 identifies the cooperative function associated with the combination of the MFP B and the PC (A) in the cooperative function management table shown in FIG. 7. The control unit 26 causes the display area 48a to display information related to the cooperation function.

  If the start time of the time limit is when the user sends the first device image (e.g., the device image 56) to the automatic response AI, the same process as described above is performed.

  By providing the time limit as described above, it is possible to switch between providing the cooperation function or the single function.

  Further, in the example shown in FIG. 8, when the automatic response AI inquires the user of the device to be cooperated with as in the content 54 of the statement, the control unit 26 when the user photographs the device (for example, the multifunction device B) with the camera 22. The device image 56 (for example, an image obtained by the photographing or a schematic image) linked to the device (the multifunction machine B) is sent to the automatic response AI as the user's speech content. The identifying unit 28 identifies the MFP B linked to the device image 56 as a device to be cooperated. Furthermore, when the user captures another device (for example, PC (A)) with the camera 22, the control unit 26 sets the device image 60 linked to the device (PC (A)) to the automatic response AI as the user's utterance content. send. The identifying unit 28 identifies the PC (A) linked to the device image 60 as a device to be cooperated. In this case, the specifying unit 28 specifies the cooperation function associated with the combination of the MFP B and the PC (A) in the cooperation function management table shown in FIG. 7. Information on the cooperation function is displayed in the display area 48a as the contents of the automatic response AI. It should be noted that photographing of a device to be cooperated may be performed by a photographing device other than the camera 22. In this case, device image data generated by shooting by the shooting device is transmitted to the terminal device 10 via a communication path such as a network, and the device image is sent to the automatic response AI as the user's speech content.

  In addition, when the cooperation function associated with the combination of a plurality of devices designated by the user is not registered in the cooperation function management table, the identification unit 28 determines that the usable cooperation function is not registered. . In this case, the control unit 26 causes the display area 48a to display a message indicating that the available cooperative function is not registered, as the content of the automatic response AI. Even in such a case, depending on the update status of the device or the like, the use of the cooperation function may be possible. In this case, the identification unit 28 identifies the collaboration function that has become available.

(Modification 1)
Hereinafter, the first modification will be described with reference to FIG. A screen 48 is shown in FIG. In the first modification, when information on a plurality of devices as collaboration targets is sent to the automatic response AI as the user's utterance content, an executable collaboration function is notified using the plurality of devices. Hereinafter, the first modification will be described in detail.

  Similar to FIG. 8, the screen 48 displays an image 50 linked to the automatic response AI and an image 52 linked to the user, and further, information indicating a dialog between the user and the automatic response AI is displayed. ing.

  As in FIG. 8, as the statement content 80 of the automatic response AI, the statement content such as “Which device do you want to use?” Is displayed in the display area 48a.

  When the user designates the image 82 in which a plurality of devices are represented as the statement content of the user and sends it to the automatic response AI with respect to the statement content 80 described above, the specification unit 28 is represented in the image 82 Identify multiple devices as collaboration target devices. For example, when the device image 56 associated with the MFP B and the device image 60 associated with the PC (A) are displayed in the image 82, the specifying unit 28 targets the MFP B to cooperate based on the device image 56. The PC (A) is identified as a collaboration target device based on the device image 60.

  The user may operate the screen 48 to select the device images 56 and 60 from the stock area 58, and superimpose the device images 56 and 60 on the image 82. As a result, the device images 56 and 60 are represented in the image 82, and the image 82 is sent to the automatic response AI as the user's speech content.

  As another example, the image 82 may be an image generated by the MFP B and the PC (A) being photographed together by an imaging device such as the camera 22. Also, the user takes a photo of the MFP B and PC (A) together with the camera 22 after the utterance content 80 has been issued, and the MFP 82 and the PC (A) are displayed together and the image 82 is displayed. You may send to automatic response AI as a statement content.

  When the MFP B and the PC (A) are identified as described above, the control unit 26 causes the information of the device identified as the device to be cooperated to be displayed in the display area 48a as the statement content 84 of the automatic response AI. . When the user approves the comment content 84 (for example, when the user issues a comment content 86 such as "Yes"), the cooperation function that can be executed using the PC (A) and the multifunction peripheral B The information is displayed in the display area 48a as the statement content 88 of the automatic response AI. When the user designates the cooperation function according to the statement content 90 and gives an execution instruction to the statement content 88, the control unit 26 receives the execution instruction, and transmits the execution instruction information to the collaboration target device (PC (PC Send to A) and MFP B). Thus, the cooperation function is executed by the PC (A) and the multifunction machine B. Of course, a screen for controlling a device to be cooperated (a screen for remote control) may be displayed on the display unit of the UI unit 20.

  Further, when three or more device images are displayed in the image 82, information on an executable collaboration function is notified using the three or more devices. Further, when one device image is displayed in the image 82, information on the function (single function) of the device linked to the one device image is notified.

  Moreover, when selecting the apparatus image stringed | linked to the apparatus of cooperation object from the stock area 58, the information regarding a cooperation function or a single function may be notified by control using said time limit. For example, when a plurality of device images are taken out of the stock area 58 and superimposed on the image 82 within the time limit, the specifying unit 28 can be performed using devices associated with each of the plurality of device images. Identify the collaboration function. In this case, the information on the cooperation function is displayed in the display area 48a as the contents of the automatic response AI. On the other hand, when one device image is taken out of the stock area 58 and superimposed on the image 82 within the time limit, the specifying unit 28 specifies the function (single function) that the device associated with the device image has. In this case, information on the single function is displayed in the display area 48a as the content of the automatic response AI.

  For example, in the example shown in FIG. 13, the user selects the device image 56 linked to the multifunction machine B from the stock area 58 within the time limit and overlaps the image 82 and does not overlap other device images on the image 82 In this case, the specifying unit 28 specifies the function (individual function) associated with the MFP B in the cooperation function management table shown in FIG. The control unit 26 displays information about the single function in the display area 48a as the content of the automatic response AI.

  On the other hand, when the user selects the device images 56 and 60 from the stock area 58 and overlaps them on the image 82 within the time limit, the specifying unit 28 generates a composite in the cooperation function management table shown in FIG. Specify the cooperation function associated with the combination of machine B and PC (A). The control unit 26 displays information on the cooperation function in the display area 48a as the content of the automatic response AI.

(Modification 2)
The second modification will be described below. In the second modification, when a plurality of device images are sent to the automatic response AI as the user's utterance content, the control unit 26 generates a new composite image representing a state in which the plurality of device images are superimposed on one another, A composite image may be generated that represents a state in which the device images of are combined next to each other. An example of the composite image is shown in FIG. For example, when the device images 56 and 60 are sent to the automatic response AI as the user's utterance content in that order, the control unit 26 generates a new composite image 92 by grouping the device images 56 and 60. The control unit 26 may display the composite image 92 on the display unit of the UI unit 20 or may register it in the terminal device 10 as an image displayed on the stock area 58. The composite image 92 may be an image representing a state in which the device image 60 is superimposed on the device image 56, or may be an image (for example, an icon) schematically representing the state.

  When the device images 60 and 56 are sent to the automatic response AI as the content of the user's utterance in that order, the control unit 26 generates a composite image representing a state in which the device image 56 is superimposed on the device image 60.

  Further, as shown in FIG. 14B, the control unit 26 may generate a composite image 94 representing a state in which the device images 56 and 60 are combined adjacent to each other. The control unit 26 may display the composite image 94 on the display unit of the UI unit 20, or may register it in the terminal device 10 as an image displayed on the stock area 58.

  In the first modification, when a plurality of device images are superimposed from the stock area 58 on the image 82, the control unit 26 registers the images 82 representing the plurality of device images in the terminal device 10 as a new composite image. May be

  When the composite image is generated as described above, the user may send the composite image to the automatic response AI as the user's speech content. For example, when the user sends the composite image and another device image to the automatic response AI as the user's utterance content, the specification unit 28 associates a plurality of devices associated with the composite image with the other device image. And are identified as linked devices. For example, when the user sends the composite image 92 and the device image associated with the projector C to the automatic response AI as the user's utterance content, the specifying unit 28 determines that the multifunction device B and PC (A) are associated with the composite image 92. Is identified as the device to be cooperated, and further, the projector C associated with the device image is identified as the device to be cooperated. In this case, the specifying unit 28 specifies the cooperation function associated with the combination of the PC (A), the MFP B, and the projector C in the cooperation function management table illustrated in FIG. 7. Information on the cooperation function is displayed on the screen 48 as the contents of the automatic response AI. Further, even if a new composite image in which the composite image 92 and the device image linked to the projector C overlap each other, or a new composite image in which the composite image 92 and the device image are combined adjacent to each other are generated. Good.

  By using the composite image as described above, the number of devices to be cooperated can be increased by a simple operation. Further, since the composite image represents a plurality of devices to be cooperated, displaying a composite image makes it easy for the user to visually identify which device is designated as a device to be linked.

  The above composite image may be used as an image for a shortcut when executing the cooperation function from next time onwards. For example, when the user gives an instruction to register a composite image as an image for a shortcut using the terminal device 10, the registration screen is displayed on the display unit of the UI unit 20. On the registration screen, a composite image is displayed, and information indicating registration content of a collaboration function that can be executed using a plurality of collaboration target devices is displayed. For example, the cooperation function name, the outline of the cooperation function, and the information indicating the device necessary for cooperation are displayed as the information indicating the registration content. The user may edit the registered content (for example, the cooperation function name, the outline of the cooperation function, and the like) using the terminal device 10. When the user gives a registration instruction on the registration screen, the composite image is registered as an image for shortcut. In addition, when the registered content is edited by the user, the edited content is also registered in association with the composite image. For example, a shortcut image as a composite image is generated and displayed in the stock area 58. The shortcut image is generated by, for example, the control unit 26 of the terminal device 10. When a plurality of shortcut images are generated and registered, a list of shortcut images may be displayed on the screen 48. The image for shortcut and the information indicating the registered content are stored in the terminal device 10. As another example, the shortcut image and the information indicating the registration content may be stored in the server 16 in association with the user account information. In this case, when the user account information is transmitted from the terminal device 10 to the server 16, the information (the image indicating the shortcut image and the information indicating the registered content) linked to the user account information is transmitted from the server 16 to the terminal device 10. Is displayed.

  Note that the image for shortcut may include information (for example, an image or the like) indicating the status of the device for each device to be cooperated. For example, when the device to be cooperated is capable of executing the cooperation function, an image (for example, a green or blue image) indicating that is displayed as information indicating the status. When the device to be cooperated is not in a state capable of executing the cooperation function, an image (for example, a white or red image) indicating that is displayed as information indicating the status. The state in which the cooperation function can be executed is, for example, a state in which the device is not currently used, a state in which no failure occurs, or the like. The state in which the cooperation function can not be executed is, for example, a state in which the device is used, a state in which a failure has occurred, or the like. For example, the automatic response AI acquires information indicating the status of the device to be cooperated from the device to be cooperated, and displays the status information based on the information. Acquisition of information may be performed by the server 16.

  When the user sends a shortcut image to the automatic response AI as the content of the user's comment, information on a linked function linked to the shortcut image may be displayed, or the execution of the linked function may be controlled.

  As described above, by linking and managing the cooperation function and the image for shortcut, the operation load of the user for executing the cooperation function from the next time is reduced.

(Modification 3)
Hereinafter, the third modification will be described. In the third modification, notification of the cooperation function according to the order of sending the device information to the automatic response AI is controlled. The third modification will be described in detail below with reference to FIGS.

  An example of the cooperation function management table according to the third modification is shown in FIG. In this collaboration function management table, as an example, information indicating a combination of device IDs, information indicating a collaboration target device name (for example, type of each device), information indicating a collaboration function (cooperation feature information), and connection The information indicating the order and the information indicating the priority are associated with each other. The device ID and the device name correspond to an example of device identification information. Note that an executable cooperation function may be defined by using three or more devices. In this case, a combination of three or more devices is associated with the cooperation function, and the association is registered in the cooperation function management table.

  The connection order corresponds to the order of designation of the device, and for example, corresponds to the order of sending information of the device (for example, device image, character information and voice information) to the automatic response AI as the user's speech content. For example, when the user sequentially sends the information of each device for the information of a plurality of devices as the user's utterance content to the automatic response AI, the order corresponds to the connection order. Specifically, when the user sends the information of the first device to the automatic response AI first as the user's speech content and then sends the information of the second device to the automatic response AI as the speech content of the user, the first device The designation order of is the first, and the designation order of the second device is the second.

  The priority is a priority of notification (for example, display or voice output) of information related to the cooperation function. For example, “scan transfer function” and “print function” are associated with the combination of the PC (A) and the multifunction machine B as cooperative functions. For example, when the information of each device is sent to the automatic response AI as the user's statement contents in the order of the MFP B and the PC (A), the priority of the "scan transfer function" becomes "first place", and the "print function" The priority of "" is "second place". In this case, the information on the "scan transfer function" is notified prior to the information on the "print function". For example, information on the “scan transfer function” is displayed above or higher than information on the “print function”. Conversely, when the information of each device is sent to the automatic response AI as the user's statement contents in the order of PC (A) and MFP B, the priority of the "print function" becomes "first place", and " The scan transfer function has the second priority. In this case, the information on the "print function" is displayed prior to the information on the "scan transfer function".

  FIG. 16 shows a display example according to the priority order. For example, when the device image 56, 60 is sent to the automatic response AI as the content of the user's message in the order of the MFP B and the PC (A), the priority of the "scan transfer function" becomes "first place" and "print" The priority of "function" is "second place". Therefore, the information on the “scan transfer function” is displayed on the screen 48 in a higher priority (for example, higher order) than the information on the “print function”. When the cooperation function is designated by the user on the screen 48 and an execution instruction is given, the designated cooperation function is executed.

  FIG. 17 shows another display example according to the priority order. For example, when the device image 56, 60 is sent to the automatic response AI as the user's message content in the order of PC (A) and MFP B, the priority of "print function" becomes "first place", and "scan transfer" The priority of "function" is "second place". Therefore, the information on the "print function" is displayed on the screen 48 in a higher priority (for example, higher order) than the information on the "scan transfer function".

  As described above, the notification order (for example, the display order) of the information related to the cooperation function is changed according to the order in which the device information is sent to the automatic response AI as the user's statement content. The connection order of the devices (the order to send the information of the devices to the automatic response AI) is combined with the order of the functions used by each device and the transfer order of the data moving between the linked devices. Is also used to specify the order of functions and the order of movement of data. Therefore, by changing the notification order of the information related to the cooperation function according to the connection order, the information related to the cooperation function predicted to be used by the user is notified preferentially. That is, information related to the cooperation function highly likely to be used by the user is notified preferentially. For example, when the information of each device is sent to the automatic response AI as the statement content of the user in the order of the multifunction device B and the PC (A), “use the function of the multifunction device B before PC (A), It is predicted that the cooperation function “transfer data from the multifunction machine B to the PC (A)” is used by the user. In addition, when the information of each device is sent to the automatic response AI as the statement content of the user in the order of the PC (A) and the multifunction machine B, “use the function of the PC (A) earlier than the multifunction machine B, It is predicted that the cooperative function of “transferring data from PC (A) to multifunction machine B” is used by the user. Therefore, by changing the notification order of the information on the cooperation function according to the order of selecting the device, the information on the cooperation function highly likely to be used by the user is preferentially notified.

(Modification 4)
The fourth modification will be described below. In the fourth modification, different functions are assigned to each part in the device image associated with the device. When a part in the device image is specified by the user and the information of the part is sent to the automatic response AI as the user's utterance content, information on the cooperation function that can be executed using the function assigned to the part Is notified. The fourth modification will be described in detail below.

  FIG. 18 shows an example of the device function management table according to the fourth modification. In the device function management table, as an example, a device ID, information indicating a device name (for example, type of device), information indicating a position in a device image (position of a portion), and a position in the device image Information (function information) indicating a function corresponding to the position of the portion is associated with the image ID. The position in the device image is a specific position (specific part) in the device image linked to the device. For example, the specific position in the device image schematically representing the device or the device image captured by the camera Within a particular location. Different functions are associated with specific positions (portions) in the device image. Note that an executable linkage function may be defined by using three or more parts. In this case, a combination of three or more parts is associated with the cooperation function, and the association is registered in the cooperation function management table. Of course, an executable collaboration function may be determined using the part and the entire device, and the association between the combination of the part and the entire device and the collaboration function may be registered in the collaboration function management table.

  Hereinafter, with reference to FIG. 19 and FIG. 20, the part of an apparatus is demonstrated in detail. FIG. 19 shows a device image 96 associated with the multifunction machine B as a device. FIG. 20 shows an apparatus image 98 associated with a PC (A) as an apparatus. By applying the identification technology (for example, AR technology) as described above, the multifunction peripheral B or PC (A) is identified. The identification of the part is also realized by using the above-described identification technique. For example, each part is identified by photographing each part of the device and applying the markerless AR technology.

  For example, as shown in FIG. 19A, the “print function” is assigned to the position (partial image 96 a) corresponding to the main body of the multifunction machine B in the device image 96. A “scan function” is assigned to a position (partial image 96 b) corresponding to the document cover, the document glass, and the automatic document feeder of the multifunction machine B in the device image 96. The “stapling function” is assigned to the position (partial image 96 c) corresponding to the post-processing device in the device image 96. The "stapling function" is a function for stopping the output sheet by the stapler.

  Further, as shown in FIG. 20A, the “data storage function” is assigned to the position (partial image 98a) corresponding to the main body of the PC (A) in the device image 98. A "screen display function" is assigned to a specific position (partial image 98b) corresponding to the display unit of the PC (A) in the device image 98. The "data storage function" is a function for storing data sent from another device in the PC (A). The “screen display function” is a function to display data sent from another device on the PC (A).

  Each partial image can be manipulated separately from the device image. For example, when the device image 96 is displayed on the display unit of the UI unit 20, as shown in FIG. 19B, if the user performs an operation of selecting the partial image 96b and separating it from the device image 96, The image 96 b is displayed apart from the device image 96. In FIG. 19 (b), the separated partial image 96b is shown by a broken line. For example, the partial image can be manipulated by a drag and drop operation. Specifically, the user may select a partial image, separate the partial image from the device image by a drag operation and operate the partial image, and display the partial image at the display position by performing a drop operation at a target display position. it can. In addition, when the device image 96 is displayed in the stock area 58, if the user performs an operation of selecting a partial image (for example, the partial image 96b) of the device image 96 and separating it from the device image 96, the partial image 96b It can be displayed outside the stock area 58 apart from the device image 96. Of course, even when an operation of separating a partial image from a device image is performed, the partial image may be displayed at the original position. The partial images 96a and 96c can also be displayed separately from the device image 96.

  Similarly to the device image 96, the partial images 98a and 98b can be separated from the device image 98 and displayed for the device image 98. In FIG. 20 (b), a partial image 98 b (indicated by a broken line) is separated from the device image 98.

  In the fourth modification, the user can send a partial image to the automatic response AI as the user's speech content. In this case, the identification unit 28 identifies a portion associated with the partial image as a portion to be cooperated. When the user sends a plurality of partial images to the automatic response AI as the user's utterance content, the identification unit 28 identifies a portion associated with each of the plurality of partial images as a collaboration target portion, and uses the plurality of portions. And identify the collaboration functions that can be executed.

  When the device image is displayed on the display unit of the UI unit 20, the control unit 26 displays the names (for example, printing, scanning, etc.) of the functions assigned to each position in the device image on the display unit of the UI unit 20. It may be displayed on the screen. By doing this, the information of the function assigned to each position is provided to the user. Of course, the name of the function may not be displayed.

  Hereinafter, processing using a partial image will be described by giving a specific example.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 linked to the automatic response AI and an image 52 linked to the user, and further, information indicating a dialog between the user and the automatic response AI is displayed. ing. In addition, a stock area 58 is displayed on the screen 48.

  As in FIG. 8, as the statement content 54 of the automatic response AI, the statement content such as “which device do you want to use?” Is displayed in the display area 48 a.

  When the user designates a part of the device with respect to the above-mentioned statement content 54, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the part of the device as a part to be cooperated. For example, as shown in FIG. 21, when the user designates a partial image 96b linked to a portion of the multifunction machine B as the user's utterance content and sends it to the automatic response AI, the specification unit 28 links to the partial image 96b Identify the part as a part of collaboration object. In the example shown in FIG. 21, as in the example shown in FIG. 8, the operation of causing the user to display the partial image 96 b in the display area 48 b corresponds to the action of the user issuing a speech content. The user may operate the UI unit 20 to input a character string indicating a part of the device to the terminal device 10 as the user's speech content. In this case, the identification unit 28 identifies the portion indicated by the character string as a portion to be cooperated. In addition, information on the part to be cooperated may be input to the terminal device 10 by voice as the content of the user's speech.

  For example, the user selects the partial image 96b included in the device image 96 displayed in the stock area 58, and moves the partial image 96b from the stock area 58 to the position where the user's statement is displayed by the drag operation The drop operation is performed at the display position of the content of the statement. By this operation, the control unit 26 sends the partial image 96 b to the automatic response AI as the user's speech content.

  Further, in the example shown in FIG. 21, following the partial image 96b, the user takes out the partial image 98b from the stock area 58, and moves the partial image 98b to the display position of the user's statement content. By this operation, the partial image 98b is sent to the automatic response AI as the user's speech content.

  When the user sends information of a plurality of parts on the screen 48 to the automatic response AI as the user's speech contents, that is, when the information of a plurality of parts is spoken as information of a part to be cooperated, the identification unit 28 Identify multiple parts as part of the collaboration target. In the above example, the MFP B associated with the device image 96 including the partial image 96b and the PC (A) associated with the device image 98 including the partial image 98b are identified as devices to be cooperated, and the partial image 96b The “scan function” corresponding to the “screen display function” corresponding to the partial image 98 b is identified as the function to be cooperated.

  When devices to be cooperated (for example, PC (A) and MFP B) are identified, the identification unit 28 combines the PC (A) with MFP B in the cooperation function management table shown in FIG. 7. Identify the linkage function associated with. Further, the identifying unit 28 identifies the function associated with the part in the device image sent to the automatic response AI as the user's statement contents by referring to the device function management table shown in FIG. 18. Do. Then, in the cooperation function group that can be executed using the PC (A) and the multifunction machine B, the identification unit 28 raises the priority of the cooperation function using the function associated with that portion. Lower the priority of collaboration functions that do not use that function. The control unit 26 of the terminal device 10 causes the screen 48 to display the information on the cooperation function as the content of the automatic response AI according to the priority.

  For example, when the MFP B and the PC (A) are identified as devices to be cooperated with each other and a part of the device is further identified as a portion to be cooperated, the control unit 26 selects one of the devices identified as the device to be cooperated. The information of the portion is displayed on the screen 48 as the statement content 100 of the automatic response AI. When the user approves the comment content 100 (for example, when the user issues a comment content 102 such as “Yes.”), Information on an executable collaboration function using a plurality of portions designated by the user Is displayed on the screen 48.

  FIG. 22 shows the screen 48 continued from FIG. When the cooperation function is specified as described above, the control unit 26 displays information on the cooperation function in the display area 48a as the speech content 104 of the automatic response AI. Since the "scan function" and the "screen display function" are designated by the user in that order, information on the cooperative function "scan transfer display function" that can be executed using the "scan function" and the "image display function" Is displayed preferentially (for example, at a higher rank) than information on other cooperation functions. For example, information on the “scan transfer display function” is displayed with priority over information on the cooperative function “scan transfer save function” that can be executed using the “scan function” and the “data storage function”. The “scan transfer display function” is a function of transferring data generated by scanning by the MFP B to the PC (A) and displaying the data on the screen of the PC (A). The “scan transfer storage function” is a function of transferring data generated by the scan by the MFP B to the PC (A) and storing the data in the PC (A).

  When the user designates the cooperation function according to the statement content 106 and gives an execution instruction to the statement content 104 of the automatic response AI, the control unit 26 receives the execution instruction, and executes the execution instruction information as a linkage target. Send to device. Further, the control unit 26 displays information indicating that the execution instruction has been received as the speech content 108 of the automatic response AI in the display area 48a. For example, when an execution instruction of the “scan transfer display function” is given, data generated by the scan by the MFP B is transferred to the PC (A) and displayed on the screen of the PC (A).

  A screen (remote control screen) for controlling devices to be cooperated (for example, the MFP B and the PC (A)) may be displayed on the display unit of the UI unit 20.

  According to the fourth modification, when the device to be cooperated has a plurality of functions, the function is individually designated, and the information on the cooperation function that can be executed using the designated function is prioritized Will be notified. Thereby, the cooperation function predicted to be used by the user is notified preferentially.

  Note that the cooperation function may be a function that uses a combination of the same device or parts of different devices, or a function that uses a combination of the entire device and the part of the device. It may be a function that uses a combination of the entire devices.

  Further, by combining the third and fourth modifications, the display of the information related to the cooperation function may be switched according to the order of sending the information of the device or part to the automatic response AI as the user's statement content.

  In addition, when the information of one part is sent to the automatic response AI as the user's speech content, the information on an executable single function is displayed on the display unit of the UI unit 20 using the one part.

  Hereinafter, another example of the process using the partial image will be described.

  FIG. 23 shows another example of the device function management table according to the fourth modification. In the device function management table, as an example, to identify the device ID, information indicating the device name (for example, type of device), information indicating the name of the portion of the device (for example, type of portion), and the portion The partial ID as the partial identification information of the, the information indicating the function assigned to the part (the function of the part), and the partial image ID for identifying the partial image tied to the part correspond to each other It is attached. The partial image is an image representing the appearance of the part of the device obtained by shooting with a camera. Of course, a partial image that schematically represents a part of the device may be associated with the part. For example, different functions are assigned to each part of the device.

  To give a specific example, a screen display function is assigned to the display unit of the PC (A), and the partial image ID of the partial image tied to the display unit is associated with information indicating the screen display function. It is done. The screen display function is a function to display information on the PC (A). A data storage function is assigned to the main body of the PC (A), and information indicating the data storage function is associated with the partial image ID of the partial image tied to the main body. The data storage function is a function for storing data in the PC (A).

  Further, a print function is assigned to the main body of the multifunction machine B, and information indicating the print function is associated with the partial image ID of the partial image tied to the main body. A scanning function is assigned to the reading unit of MFP B (for example, the part corresponding to the document cover, document glass, or automatic document feeder of MFP B), and a partial image of a partial image tied to the reading unit The ID is associated with information indicating a scan function. The post-processing device of the multifunction machine B is assigned a stapling function, and thereafter, the partial image ID of the partial image tied to the processing device is associated with information indicating the stapling function. The stapling function is a function to staple an output sheet.

  The functions assigned to parts of the device are identified (identified) by using, for example, markerless AR technology. For example, when a part of the device is photographed by a camera (for example, the camera 22 of the terminal device 10), the specifying unit 28 is associated with the appearance image data in the device function management table shown in FIG. Identify (identify) the function. As a result, the function assigned to the photographed part is identified (identified). For example, when the main body of the multifunction machine B is photographed by the camera 22, the specifying unit 28 specifies the print function associated with the appearance image data representing the main body of the multifunction machine B in the device function management table. . Thus, it is specified that the function assigned to the main body of the multifunction peripheral B is the print function.

  Of course, the functions assigned to parts of the device may be identified by using marker-based AR technology. For example, each part of the device is provided with a marker such as a two-dimensional barcode in which part identification information (for example, part ID) for identifying the part is encoded. When a marker provided on a part is photographed by a camera and the marker type AR technology is applied, part identification information (for example, a part ID) of the part is acquired. Thus, when the partial identification information is acquired, the identification unit 28 identifies (identifies) the function associated with the partial identification information (for example, the partial ID) in the device function management table shown in FIG. ).

  An example of the cooperation function management table according to the fourth modification is shown in FIG. The cooperation function management table is information indicating a cooperation function using functions possessed by a plurality of parts, and in the cooperation function management table, as an example, a combination of information indicating a combination of device parts and a part ID The information to be shown and the information indicating the cooperation function that can be executed using the functions of the plurality of parts included in the combination are associated with each other. Of course, in the linkage function management table, information indicating the combination of the device portion and the entire device, and information indicating the linkage function using the function of the device portion and the function of the entire device are included. It may be associated.

  A specific example will be described. A combination of the display unit of the PC (A) and the main unit of the multifunction machine B is assigned a “print function” as a cooperation function. The “print function” as the cooperation function is a function of transmitting data stored in the PC (A) to the multifunction peripheral B and printing the data by the multifunction peripheral B.

  Further, to a combination of the main body of the multifunction device B and the main body of the projector C, a “print function” as a cooperative function is assigned. The “print function” as the cooperation function is a function of transmitting the data projected by the projector C to the multifunction device B and printing the data by the multifunction device B.

  Further, to a combination of the reading unit of the multifunction device B and the main body of the projector C, a “scan projection function” as a cooperative function is assigned. The “scan projection function” as the cooperation function is, for example, a function of transmitting data generated by scanning by the multifunction device B to the projector C and projecting the data by the projector C.

  Note that the cooperative function may be a function that can be performed using functions of multiple parts included in the same device, or can be performed using functions of parts of multiple different devices. It may be a function. In addition, the cooperation function may be a function that can be performed using the function of three or more parts.

  For example, when the user sends a plurality of partial images to the automatic response AI as the user's utterance content, the specifying unit 28 links to each of the plurality of partial images in the cooperation function management table shown in FIG. Identify (identify) the cooperation function associated with the combination of parts. For example, when the partial image tied to the main body of the multifunction machine B and the partial image tied to the main body of the projector C are sent to the automatic response AI, the specifying unit 28 performs the cooperation function management table shown in FIG. In the above, the cooperative function “print function” or the like associated with the combination of the main body of the multifunction device B and the main body of the projector C is specified. As shown in FIG. 22, the information related to the cooperation function is displayed in the display area 48a as the contents of the automatic response AI.

  Further, by combining the third and fourth modifications, the display of the information related to the cooperation function may be switched according to the order in which the device image or the partial image is sent to the automatic response AI as the user's utterance content.

(Modification 5)
Hereinafter, the fifth modification will be described. In the fifth modification, the information on the cooperation function is input to the terminal device 10 by the user as the user's speech content, and then the information on the device to be cooperated is input to the terminal device 10 according to the speech content of the user. Hereinafter, the fifth modification will be described in detail.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  First, the user inputs an utterance content 110 indicating a request for execution of the cooperation function. In the example shown in FIG. 25, the cooperation function requested by the user is a cooperation function for transferring scan data to the PC. The automatic response AI analyzes the content of the statement 110 to determine that the user has requested execution of the cooperation function, and issues a statement of content 112 for inquiring the user of a device to be used for the cooperation function. The control unit 26 displays the message content 112 in the display area 48a. The user inputs, to the terminal device 10, information on the device to be used for the cooperation function with respect to the statement content 112. The user may input the information about the device to the terminal device 10 by arranging the device image as the information about the device at the display position of the user's statement content, or the information about the device to the terminal device 10 by voice. You may enter it. In the example shown in FIG. 25, the device images 56 and 60 are displayed at the display position of the user's statement content. Thereby, the control unit 26 sends the device image 56, 60 to the automatic response AI as the user's statement content, and the specifying unit 28 causes the multifunction device B to be linked to the device image 56 and the PC (A) to be linked to the device image 60. Is identified as a collaboration target device. The automatic response AI issues an utterance content 114 indicating the identified device, and the control unit 26 causes the utterance content 114 to be displayed in the display area 48a. When the user issues an utterance content 116 indicating a cooperative function execution instruction to the utterance content 114, the control unit 26 causes the MFP B and the PC (A) to execute the cooperative function requested by the user. Further, the control unit 26 displays information indicating that the execution instruction of the cooperation function has been issued as the speech content 118 of the automatic response AI in the display area 48a.

  As described above, according to the fifth modification, even when the user first sends information on the cooperation function to the automatic response AI, the cooperation function is executed by a simple operation through the dialog with the automatic response AI. Can.

(Modification 6)
The sixth modification will be described below. In the sixth modification, by using the learning function of the automatic response AI, notification of the cooperation function is controlled according to the information obtained by the automatic response AI. The automatic response AI is, for example, information related to the user (for example, information indicating the operation history of the terminal device 10, information indicating the operation history of the device, information For example, information indicating the usage history, information indicating the state of the device, etc., etc. are acquired. The automatic response AI may acquire information from the terminal device 10, may acquire information from devices such as the devices 12 and 14, or may acquire information from the server 16, or other devices The information may be acquired from (for example, a cloud) or may be acquired from another operation assistant (for example, artificial intelligence stored in another device) stored in another device.

  The control unit 26 controls the notification of the cooperation function according to the information acquired by the automatic response AI. For example, when the information indicating the usage history of the user for the device or the information (schedule information) indicating the schedule (schedule) of the user is acquired by the automatic response AI, the control unit 26 refers to the usage history and the schedule. Information on a cooperation function highly likely to be used by the user is preferentially displayed on the display unit of the UI unit 20 as the utterance content of the automatic response AI. For example, when the user sends a plurality of device images to the automatic response AI as the user's speech content, a plurality of collaboration functions that can be executed using devices linked to each of the plurality of device images are identified. The control unit 26 analyzes the usage history and the schedule, and causes the display unit of the UI unit 20 to preferentially display information on a cooperation function that is highly likely to be used by the user among the plurality of cooperation functions. For example, the control unit 26 compares the timing at which a plurality of devices have been designated as devices to be cooperated with a schedule in the past, and prioritizes information on a cooperation function which is highly likely to be used at present. Display on the display unit.

  In the past, in the morning of the day when the user went out in the afternoon, the first device image and the second device image are sent to the automatic response AI as the user's speech contents and the cooperation function Z It shall be executed. When such a usage history exists, the timing when the first device image and the second device image are sent to the automatic response AI as the user's statement content is in the morning of the day when the user is scheduled to go out in the afternoon In this case, the control unit 26 determines that there is a high possibility that the cooperation function Z described above is used, and causes the display unit of the UI unit 20 to preferentially display information related to the cooperation function Z.

  To explain this with another specific example, in the past, the first device image and the second device image were sent to the automatic response AI as the user's statement content in a certain time zone, and the recording function as the cooperation function was executed. It is assumed that the channel X program has been recorded. When such a usage history exists, the timing at which the first device image and the second device image are sent to the automatic response AI as the user's statement content is included in the time zone, the control unit 26 It is determined that the above recording function is highly likely to be used, and information on the recording function (for example, information on the function of recording channel X) is preferentially displayed on the display unit of the UI unit 20.

  In addition, when the device linked to the device image sent to the automatic response AI is in use or in failure, the control unit 26 lowers the priority of the information related to the cooperation function that can be executed using the device, and the information May be displayed on the display unit of the UI unit 20. When the device has not failed and is not in use, the control unit 26 causes the display unit of the UI unit 20 to preferentially display information on an executable cooperative function using the device. In this way, information on the ready-to-use cooperation function is preferentially provided to the user.

(Modification 7)
Hereinafter, modification 7 will be described. In the seventh modification, the automatic response AI recommends a device to be linked. For example, when the user sends a device image to the automatic response AI as the user's statement content, the identifying unit 28 identifies the device associated with the device image, and refers to the cooperation function management table to identify the device. Identify the devices that can execute the collaboration function in combination. The control unit 26 causes the display unit of the UI unit 20 to display the information on the device specified by the specifying unit 28 as the content of the automatic response AI. For example, when the user sends the device image 56 linked to the multifunction device B to the automatic response AI as the user's statement content, the specifying unit 28 links with the multifunction device B by referring to the linked function management table. Identify devices that can execute the function (for example, a PC, a projector, etc.). Information on the device (for example, a PC or a projector) is displayed on the display unit of the UI unit 20 as the contents of the automatic response AI. By doing this, the device to be cooperated is provided to the user.

  In addition, the identification unit 28 is actually combined with the device (for example, the multifunction device B) associated with the above-described device image based on the use history of the cooperation function (use history of the user or use history of other users). The used device may be specified, and the control unit 26 may display information about the device on the display unit of the UI unit 20 as the content of the automatic response AI. For example, the specification unit 28 may specify a device whose frequency (for example, the number of times of use) used in combination with the device (for example, the multifunction device B) associated with the device image is equal to or higher than a predetermined threshold. . In this way, devices with a relatively high frequency of use are provided to the user.

  In addition, the control unit 26 may cause the display unit of the UI unit 20 to preferentially display information related to a device having a high frequency of use in combination with the device (for example, the multifunction peripheral B) associated with the device image described above. .

(Modification 8)
Hereinafter, the modified example 8 will be described. In the eighth modification, a plurality of automatic responses AI may cooperate with each other. For example, the automatic response AI stored in the terminal device 10 shares information with the automatic response AI stored in another terminal device (for example, usage history information of a cooperation function, usage history information of a device, etc.) It is also good. In this case, the specifying unit 28 may specify the cooperation function by using the above-described shared information acquired by the automatic response AI, or as in the seventh modification, even if the device to be cooperated is recommended. Good. For example, the specifying unit 28 may specify a device capable of executing the cooperation function in combination with the device passed to the operation assistant by using the usage history information included in the shared information.

  Also, an automatic response AI may be defined for each user. In this case, a plurality of automatic responses AI may be stored in the same terminal device 10. Each automatic response AI manages information (for example, operation history information of the terminal device 10, use history information of a cooperation function, use history information of a device, etc.) related to the user linked to itself, and other automatic responses AI You may share information with In this case, each automatic response AI may recommend a device to be cooperated by using the shared information, or may recommend a cooperation function.

(Modification 9)
Hereinafter, modification 9 will be described. In the ninth modification, the user interacts with the automatic response AI by voice. For example, when the user utters the name of a device to be cooperated (for example, multifunction machine B, PC (A), etc.) by voice, the voice is input to the terminal device 10 and realized by the identification unit 28 (automatic response AI) The identifying unit 28) receives the voice input and identifies the cooperation function. The information related to the cooperation function may be displayed on the display unit of the UI unit 20 as the contents of the automatic response AI and may be emitted as voice information. When voice input is used, the image associated with the automatic response AI may not be displayed on the display unit of the UI unit 20. Of course, when the image associated with the automatic response AI is displayed, information about the device may be input to the terminal device 10 as voice content of the user by voice input.

Second Embodiment
An apparatus system according to the second embodiment will be described below. In the first embodiment described above, when the information of the device is sent to the automatic response AI as the user's statement content, the information on the cooperation function is notified as the statement content of the automatic response AI, but in the second embodiment, When the information of the function is sent to the automatic response AI as the user's statement content, information on the cooperation function is notified as the statement content of the automatic response AI. The information of the function is identification information for identifying the function, and is, for example, a functional image associated with the function, character information indicating the identification information, voice information or the like. For example, when the user sends a function image associated with a function to the automatic response AI, or when the user inputs a character string indicating the identification information of the function at the terminal device 10 and gives it to the automatic response AI, When the identification information of the function is given to the automatic response AI by voice, the specification unit 28 identifies the function.

  The functional image may be, for example, an image associated with software (application or the like), or may be an image associated with a device on which processing is performed (a device present in the real space). The functional image is, for example, an image such as an icon, and can be said to be a virtual image in the sense of distinguishing it from an image (for example, a photo) representing the actual appearance of the device as hardware or an image representing the device itself. For example, when a functional image tied to software is sent to the automatic response AI as the user's speech content, information indicating an executable cooperative function using the software tied to the functional image is the speech content on the automatic response AI Be notified. Hereinafter, the second embodiment will be described in detail.

  FIG. 26 shows an example of the cooperation function management table according to the second embodiment. The cooperation function management table is information indicating cooperation functions that can be executed using a plurality of functions. In this cooperation management table, as an example, a function ID (for example, an ID for identifying software), information indicating a function name (for example, software name), and a cooperation function that can be executed using a plurality of functions. The pieces of information shown are associated with one another. By referring to this cooperation function management table, cooperation functions that can be executed using a plurality of functions are identified.

  For example, software α as a function is software that sends data, and software β as a function is software that applies a password to data. By combining the softwares α and β, for example, a cooperative function of transmitting data after applying a password is realized. Similarly, an executable linkage function is defined for other software.

  Note that an executable linkage function may be defined by using three or more functions (for example, software). In this case, a combination of three or more functions is associated with the cooperation function, and the association is registered in the cooperation function management table.

  Hereinafter, the second embodiment will be described in detail by giving a specific example.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 120 such as “which function do you want to use?” Is displayed in the display area 48 a as the utterance content of the automatic response AI.

  When the user designates a function with respect to the message content 120, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the function as a cooperation target function. For example, as shown in FIG. 27, the user designates a functional image 122 associated with a function, and arranges the functional image 122 at a position where the user's speech content is displayed, whereby the control unit 26 displays the functional image Send 122 to the automatic response AI as the user's speech content. In response to this operation, the identification unit 28 identifies the function associated with the functional image 122 as the function to be cooperated. In the example illustrated in FIG. 27, the operation of causing the user to display the functional image 122 in the display area 48b corresponds to the action of the user emitting the content of the statement. Note that the user may operate the UI unit 20 to input a character string indicating a function to the terminal device 10 as the user's speech content. In this case, the identification unit 28 identifies the function indicated by the character string as the cooperation target function. In addition, information of the function to be cooperated may be input to the terminal device 10 as a speech content of the user by voice.

  The functional image 122 is an image associated with the password setting function (for example, an image associated with password setting software). When the functional image 122 is selected by the user and an activation instruction of the password setting function is given, for example, password setting software is activated, and password setting processing for data or the like is performed.

  For example, as in the first embodiment, the control unit 26 displays the stock area 124 in the display area 48b for the user, and displays a functional image in the stock area 124. For example, the control unit 26 causes the stock area 124 to display a functional image or the like associated with the software installed in the terminal device 10. Also, when there are other functional images that can not be displayed in the stock area 124, the user scrolls the functional images displayed in the stock area 124 so that the other functional images are in the stock area 124. Is displayed on.

  Similar to the first embodiment, the user selects a function image (the function image 122 in the example shown in FIG. 27) linked to the function to be cooperated from the stock area 124 and moves it to the outside of the stock area 124 The image 122 can be sent to the automatic response AI as the user's speech content. For example, the user moves the functional image 122 from the stock area 124 to a position where the user's speech content is displayed by a drag operation, and performs a drop operation at the display position of the speech content. By this operation, the control unit 26 sends the functional image 122 to the automatic response AI as the user's speech content.

  Further, in the example shown in FIG. 27, following the functional image 122, the user selects the functional image 126 from the stock area 124, and moves the functional image 126 to the display position of the user's speech content. By this operation, the functional image 126 is sent to the automatic response AI as the user's speech content. The functional image 126 is an image associated with the data transmission function (for example, an image associated with data transmission software). When the functional image 126 is selected by the user and an activation instruction of the data transmission function is given, for example, data transmission software is activated, and data to be transmitted is transmitted to the transmission destination.

  When the user sends information of a plurality of functions to the automatic response AI on the screen 48, that is, when information of a plurality of functions is spoken as information of a function to be cooperated, the specifying unit 28 is shown in FIG. In the linked function management table, the linked function associated with the plurality of functions is specified. The control unit 26 displays information on the cooperation function in the display area 48a as the contents of the automatic response AI.

  In the above example, the functional images 122 and 126 are sent to the automatic response AI as the function to be cooperated. In this case, the specifying unit 28 identifies the function associated with the functional image 122 and the function associated with the functional image 126 as the function to be cooperated. Alternatively, the functional image 122 may be sent to the automatic response AI first, and then the functional image 126 may be sent to the automatic response AI, or the functional image 126 may be sent to the automatic response AI first, and then the functional image 122 may be automatic response May be sent to AI.

  As described above, when the functional images 122 and 126 are sent to the automatic response AI and the password function and the data transmission function are identified as the cooperation target function, the control unit 26 selects one of the functions identified as the cooperation target function. The information is displayed in the display area 48a as the statement content 128 of the automatic response AI. When the user approves the message content 128 (for example, when the user issues the message content 130 such as “Yes.”), The cooperation function that can be executed using the PC (A) and the MFP B The information is displayed in the display area 48a.

  The identification unit 28 is shown in FIG. 26 at the timing when the functional image 122 or 126 is sent to the automatic response AI as information of the function to be cooperated, or at the timing when the user approves the message content 128. In the cooperation function management table, a cooperation function (for example, “a function of applying a password to data transmission and the like”) associated with the combination of the password function and the data transmission function is specified. As a result, an executable linkage function is identified using the password function and the data transmission function. The information on the cooperation function specified in this manner is displayed in the display area 48a as the content of the automatic response AI. Note that the process of specifying the function or the cooperation function may be performed by the server 16.

  FIG. 28 shows the screen 48 continued from FIG. When the cooperation function is specified as described above, the control unit 26 of the terminal device 10 causes the screen 48 to display information on the cooperation function as the statement content 132 of the automatic response AI.

  When the user designates the cooperation function according to the speech contents 134 and gives an execution instruction to the speech contents 132, the control unit 26 receives the execution instruction and executes the cooperation function. In addition, the control unit 26 causes the screen 48 to display information indicating that the execution instruction has been received as the speech content 136 of the automatic response AI. As another example, a button image for instructing execution of the cooperation function may be displayed on the screen 48, and the user may give an execution instruction of the cooperation function by pressing the button image. When the password function and the data transmission function are selected as the cooperation target function, the control unit 26 activates the password setting software, sets a password for the data to be processed, and further activates the data transmission software. Send the data to the destination.

  Note that the functional image displayed in the stock area 124 in FIG. 27 may be edited by the user, or information regarding the function associated with the functional image may be edited by the user. For example, when the user designates a functional image displayed in the stock area 124 and gives an editing instruction, the control unit 26 causes the display unit of the UI unit 20 to display an editing screen. For example, when the user long-presses the functional image (for example, when the length of time during which the functional image is pressed becomes equal to or more than the threshold), the control unit 26 presses the button image for displaying the editing screen. In the case where the user inputs an editing instruction into the terminal device 10 by voice or the like, the editing screen is displayed on the display unit of the UI unit 20. On the editing screen, it is possible to edit the functional image. For example, when a function is further assigned to each part of the functional image, it is also possible to cut out a partial image from the functional image. In addition, it is also possible to edit information on the function associated with the functional image.

  As another example, the cooperation target function may be set in advance as a basic cooperation function. The information on the basic cooperation function may be stored in advance in the storage unit 24 of the terminal device 10. The user may specify the basic cooperation function using the UI unit 20. When the basic cooperation function is set, the user sends a functional image linked to a cooperation target function other than the basic cooperation function to the automatic response AI as a speech content. Thereby, the function to be cooperated is specified (identified), and the cooperation function that can be executed using the basic cooperative function and the function is specified (identified).

  As described above, according to the second embodiment, on the screen 48 for interacting with the automatic response AI, the information on the collaboration function is provided by sending the information of the function to be cooperated to the automatic response AI as the dialogue content. Be done. As described above, the information on the cooperation function is provided by the simple method using the interaction with the automatic response AI.

  When three or more functional images are sent as the interactive contents to the automatic response AI, information on a collaboration function that can be executed using the three or more functions is provided.

  In addition, when information of a plurality of functions is sent to the automatic response AI as the user's speech content within a predetermined time limit, the specifying unit 28 specifies an executable cooperation function using the plurality of functions. If the information of one function is sent to the automatic response AI as the user's speech content within the time limit, the function may be specified as a single function. In this case, information on the cooperation function or the single function specified by the specifying unit 28 is displayed in the display area 48a as the content of the automatic response AI. The start time of the time limit may be, for example, the time when the automatic response AI inquires the user of the function to be cooperated, as in the case of the speech content 120, or the user is the first function after the generation of the speech content 120. It may be the point when the image is sent to the automatic response AI.

  For example, in the example shown in FIG. 27, within the time limit from when the automatic response AI inquires the user of the function to be cooperated as in the statement content 120, the user sets only the functional image 122 as the statement content of the user to the automatic response AI. When it is sent, the identification unit 28 identifies the function (single function) associated with the functional image 122. The control unit 26 displays information about the single function in the display area 48a as the content of the automatic response AI.

  On the other hand, if the user sends the functional image 122 or 126 as the user's utterance content to the automatic response AI within the time limit from when the automatic response AI inquires the user of the function of the cooperation function as in the utterance content 120 The unit 28 specifies the cooperation function associated with the combination of the password function and the data transmission function in the cooperation function management table shown in FIG. The control unit 26 displays information on the cooperation function in the display area 48a.

  If the start time of the time limit is when the user sends the first functional image (for example, the functional image 122) to the automatic response AI, the same process as described above is performed.

  By providing the time limit as described above, it is possible to switch between providing the cooperation function or the single function.

  In addition, when the cooperation function associated with the combination of a plurality of functions designated by the user is not registered in the cooperation function management table, the identification unit 28 determines that the available cooperation function is not registered. . In this case, the control unit 26 causes the display area 48a to display a message indicating that the available cooperative function is not registered, as the content of the automatic response AI. Even in such a case, depending on the update status of the function, etc., it may be possible to use the cooperation function. In this case, the identification unit 28 identifies the collaboration function that has become available.

  Also, in the second embodiment, the first to ninth modifications of the first embodiment may be applied.

  The process at the time of applying the modification 1 to 2nd Embodiment is shown by FIG. By applying the first modification, when information on a plurality of functions to be cooperated is sent to the automatic response AI as the user's statement content, an executable cooperation function is notified using the plurality of functions. Ru. Hereinafter, with reference to FIG. 29, a process in the case where the first modification is applied to the second embodiment will be described.

  Similar to FIG. 27, as the statement contents 138 of the automatic response AI, the statement contents such as “Which function do you want to use?” Are displayed in the display area 48a.

  When the user designates the image 140 in which a plurality of functions are represented as the statement content of the user and sends it to the automatic response AI with respect to the statement content 138 described above, the identifying unit 28 is represented in the image 140 Identify multiple functions as linked devices. For example, when the functional image 122 associated with the password function and the functional image 126 associated with the data transmission function are represented in the image 140, the specifying unit 28 sets the password function as the cooperation target function based on the functional image 122. It identifies and the data transmission function is identified as a cooperation target function based on the functional image 126.

  The user may operate the screen 48 to select the functional images 122 and 126 from the stock area 124, and may overlay the functional images 122 and 126 on the image 140. As a result, the functional images 122 and 126 are represented in the image 140, and the image 140 is sent to the automatic response AI as the user's speech content.

  When the password function and the data transmission function are identified as described above, the control unit 26 displays information of the function identified as the function to be cooperated in the display area 48a as the speech content 142 of the automatic response AI. When the user approves the comment content 142 (for example, when the user issues a comment content 144 such as “Yes.”), The information on the cooperation function that can be executed using the password function and the data transmission function is , And is displayed on the screen 48 as the statement content 146 of the automatic response AI. When the user designates the cooperation function according to the statement content 148 and gives an execution instruction to the statement content 146, the control unit 26 activates the password setting software and the data transmission software. Thus, the cooperation function is executed by the password setting software and the data transmission software.

  In addition, when three or more functional images are displayed in the image 140, information on an executable collaboration function is notified using the three or more functions. In addition, when one functional image is displayed in the image 140, information on a function (unique function) associated with the one functional image is notified.

  In addition, when a functional image linked to a function to be cooperated is selected from the stock area 124, information on the cooperative function or a single function may be notified by control using the above-mentioned time limit. For example, when a plurality of functional images are taken out of the stock area 124 and superimposed on the image 140 within the time limit, the specifying unit 28 can be performed using a function associated with each of the plurality of functional images. Identify the collaboration function. In this case, the information on the cooperation function is displayed in the display area 48a as the contents of the automatic response AI. On the other hand, when one functional image is extracted from the stock area 124 and superimposed on the image 140 within the time limit, the specifying unit 28 specifies a function (unique function) associated with the functional image. In this case, information on the single function is displayed in the display area 48a as the content of the automatic response AI.

  For example, in the example shown in FIG. 29, the user selects the functional image 122 associated with the password function from the stock area 124 within the time limit and overlaps the image 140 and does not overlap other functional images on the image 140 In this case, the specifying unit 28 specifies the password function associated with the functional image 122 as a single function. The control unit 26 displays information about the single function in the display area 48a as the content of the automatic response AI.

  On the other hand, when the user selects functional images 122 and 126 from stock area 124 and overlaps them on image 140 within the time limit, specifying unit 28 generates a password in the cooperation function management table shown in FIG. Identify the cooperation function associated with the combination of the function and the data transmission function. The control unit 26 displays information on the cooperation function in the display area 48a as the content of the automatic response AI.

  Further, by applying the modification 2, when a plurality of functional images are sent to the automatic response AI as the user's utterance content, a composite image representing a state in which the plurality of functional images are superimposed on each other is generated. Alternatively, a composite image in which the plurality of functional images are combined adjacent to each other may be generated (see FIG. 14 (b)). The composite image may be displayed in the stock area 124. In addition, the composite image may be registered in the terminal device 10 as a shortcut image.

  Further, by applying the modification 3, the cooperation function to be notified may be changed according to the order of sending the function information to the automatic response AI. In this case, the connection order and the priority of functional images are determined also in the cooperation function management table according to the second embodiment, as in the cooperation function management table shown in FIG.

  Further, by applying the fourth modification, a different function may be assigned to each part in the functional image. In this case, when the partial image linked to the function is sent to the automatic response AI as the user's speech content, information on the collaboration function that can be executed using the function linked to the partial image is displayed. Also, the combination of the third and fourth modifications may be applied to the second embodiment.

  Further, by applying the fifth modification, the information on the cooperation function is input to the terminal device 10 by the user as the statement content of the user first, and then the information on the function to be cooperated is the terminal device as the statement content of the user 10 may be input.

  Moreover, the notification of a cooperation function may be controlled according to the information which automatic response AI acquired by applying the modification 6. FIG. For example, the automatic response AI is information indicating the usage history of the function by the learning function, information indicating the execution status of the software for realizing the function, information on the user (information indicating the operation history and action history), the schedule of the user May be acquired, and notification of the collaboration function may be controlled using these pieces of information.

  Moreover, the function used as cooperation object may be recommended by the modification 7 being applied. For example, when the user sends a functional image to the automatic response AI as the user's utterance content, the identifying unit 28 identifies the function associated with the functional image and refers to the cooperation function management table to identify the function. Identify the functions that can execute the collaboration function in combination. Further, the specifying unit 28 may recommend a function based on the use history of the cooperation function, or may recommend a function having a relatively high frequency of use.

  Further, by applying the modification 8, a function used for the cooperation function may be recommended based on the information shared by the automatic response AI with another automatic response AI.

  Moreover, the modification 9 may be applied, and the information regarding the function of cooperation object may be input into the terminal device 10 by audio | voice input.

Third Embodiment
The device system according to the third embodiment will be described below. The third embodiment corresponds to a combination of the first embodiment and the second embodiment described above. That is, when the device and function information (e.g., device image and function image) is sent to the automatic response AI as the user's statement content, an executable cooperation function is notified using the device and function. The third embodiment will be described in detail below.

  FIG. 30 shows an example of the cooperation function management table according to the third embodiment. The cooperation function management table is information indicating a cooperation function that can be executed using the device and the function (for example, software). In this collaboration function management table, as an example, a collaboration function that can be executed using a combination of IDs (combination of a device ID and a function ID), information indicating a device name and a function name, and a device and a function The pieces of information shown are associated with one another. By referring to the linked function management table, linked functions that can be executed using the device and the function are identified. For example, by combining the PC (A) as the device A and the data transmission function as the function α, for example, a cooperative function of transmitting data using the PC (A) is realized. Similar executable functions are defined for other devices and functions. Note that an executable cooperative function may be defined by using a total of three or more devices and functions. In this case, a combination of three or more devices or functions is associated with the cooperation function, and the association is registered in the cooperation function management table.

  The third embodiment will be described below with reference to a specific example.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 150 such as “which function do you want to use?” Is displayed in the display area 48 a as the utterance content of the automatic response AI.

  When the user designates a device with respect to the contents of this statement 150, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the device as a function to be cooperated. For example, as shown in FIG. 31, the user designates the device image 60 linked to the PC (A) as a device, and arranges the device image 60 at a position where the user's speech content is displayed, whereby the control unit 26 sends the device image 60 to the automatic response AI as the user's speech content. In response to this operation, the identification unit 28 identifies the PC (A) associated with the device image 60 as a device to be cooperated. In the example illustrated in FIG. 31, the operation of causing the user to display the device image 60 in the display area 48 b corresponds to the action of the user issuing a speech content. The user may operate the UI unit 20 to input a character string indicating the device to the terminal device 10 as the user's speech content. In this case, the identification unit 28 identifies the device indicated by the character string as the device to be cooperated. In addition, information on the device to be cooperated may be input to the terminal device 10 by voice as the content of the user's speech.

  Further, as in the first embodiment, the control unit 26 may display the stock area 58 in the display area 48b for the user, and may display the device image in the stock area 58. In this case, the user may select a device image from within the stock area 58 and move the device image to the display position of the user's statement content. In this case, the control unit 26 sends the device image to the automatic response AI as the user's speech content.

  Further, in the example shown in FIG. 31, the user specifies a function following the device image 60 after the statement content 152 of the automatic response AI. In this case, the identification unit 28 identifies the function as a cooperation target function. For example, when the user designates a functional image 154 linked to a function and arranges the functional image 154 at the display position of the user's speech content, the control unit 26 automatically uses the functional image 154 as the user's speech content. Send to response AI. In response to this operation, the identification unit 28 identifies the function associated with the functional image 154 as the function to be cooperated. In the example illustrated in FIG. 31, the operation of causing the user to display the functional image 154 in the display area 48 b corresponds to the act of the user issuing a speech content. Note that the user may operate the UI unit 20 to input a character string indicating a function to the terminal device 10 as the user's speech content. In this case, the identification unit 28 identifies the function indicated by the character string as the cooperation target function. In addition, information of the function to be cooperated may be input to the terminal device 10 as a speech content of the user by voice.

  The functional image 154 is an image associated with the image forming function (for example, an image associated with the image forming software or an image associated with the function of the multifunction peripheral). When the functional image 154 is selected by the user and an activation instruction of the image forming function is given, for example, the image forming software is activated, and a print instruction and a scan instruction are given to the multifunction peripheral.

  When the user sends the information of the device and function to the automatic response AI on the screen 48, that is, when the information of the device and function is uttered as the information of collaboration object, the specifying unit 28 performs the cooperation shown in FIG. In the function management table, the cooperative function associated with the combination of the device and the function is specified. The control unit 26 displays information on the cooperation function in the display area 48a as the contents of the automatic response AI.

  In the above-described example, the device image 60 and the functional image 154 are sent to the automatic response AI as the collaboration object. In this case, the identification unit 28 (the identification unit 28 realized by the automatic response AI) identifies the PC (A) attached to the device image 60 and the image forming function attached to the functional image 154 as a collaboration target. Alternatively, the device image 60 may be sent to the automatic response AI first, and then the functional image 154 may be sent to the automatic response AI. Alternatively, the functional image 154 may be sent to the automatic response AI, and then the device image 60 may be automatic response. May be sent to AI.

  As described above, when the device image 60 and the functional image 154 are sent to the automatic response AI and the device and the function to be cooperated are identified, the control unit 26 automatically performs the information on the device and the function which are identified as the cooperation object. The message content of the response AI may be displayed in the display area 48a. When the user approves the content of this statement (for example, when the user issues a statement such as "Yes"), the information on the cooperation function that can be executed using the PC (A) and the image forming function is It may be displayed in the display area 48a.

  The identifying unit 28 is a linked function (for example, “scan transfer function”, “print function”, etc.) associated with a combination of the PC (A) linked to the device image 60 and the image forming function linked to the functional image 154. Identify). As a result, an executable cooperative function is specified using the PC (A) and the image forming function. The information on the cooperation function specified in this manner is displayed in the display area 48a as the content of the automatic response AI. Note that the process of specifying the function or the cooperation function may be performed by the server 16.

  The control unit 26 displays information on the cooperation function in the display area 48a as the statement content 156 of the automatic response AI.

  When the user designates the cooperation function according to the statement content and gives an execution instruction to the statement content 156, the control unit 26 receives the execution instruction and executes the cooperation function. For example, when an instruction to execute a scan transfer function is given, a multifunction machine is designated by the user using, for example, image forming software, and data generated by the scan by the multifunction machine is transmitted from the multifunction machine to the PC. It is transferred to (A).

  As another example different from the above, when, for example, the device image associated with the multifunction machine B and the functional image associated with the data transmission function are sent to the automatic response AI as the user's utterance content, the identifying unit 28 Identify B and the data transmission function as collaboration targets. In this case, the specifying unit 28 specifies an executable cooperative function using the MFP B and the data transmission software by referring to the cooperative function management table shown in FIG. The information indicating the cooperation function is displayed in the display area 48a as the content of the automatic response AI.

  As another example, a device to be cooperated may be preset as a basic cooperation device, or a function to be cooperated may be predetermined as a basic cooperation function. The information on the basic cooperation device and the basic cooperation function may be stored in advance in the storage unit 24 of the terminal device 10. The user may designate the basic cooperation device or the basic cooperation function using the UI unit 20. When the basic cooperation device or the basic cooperation function is set, the user sends information on the cooperation device or function other than the basic cooperation device or the basic cooperation function to the automatic response AI as the user's comment content. As a result, the device or function to be cooperated is identified (identified).

  As described above, according to the third embodiment, on the screen 48 for interacting with the automatic response AI, the information on the collaboration function is sent by transmitting the information of the device and the function to be cooperated as the interactive content to the automatic response AI. Is provided. As described above, the information on the cooperation function is provided by the simple method using the interaction with the automatic response AI.

  In addition, when a total of three or more device images and function images are sent to the automatic response AI as the user's statement content, the information indicating the cooperation function that can be executed using the three or more devices and functions is an automatic response It may be displayed as the AI's statement content.

  In addition, when the information on the device and the function is sent to the automatic response AI as the user's utterance content within the predetermined time limit, the specifying unit 28 can execute the cooperative function that can be executed using the device and the function. If the information of one device or one function is sent to the automatic response AI as the user's speech content within the time limit, the function of the one device or the one function is specified as a single function. May be In this case, information on the linked function or the single function specified by the specifying unit 28 is displayed on the display unit of the UI unit 20. The start time of the time limit may be, for example, the time when the automatic response AI inquires the user of the cooperation target, as in the case of the statement content 150, or after the generation of the statement content 150, the user It may be the time when the functional image is sent to the automatic response AI.

  For example, in the example shown in FIG. 31, the user sends only the device image 60 as the user's utterance content to the automatic response AI within the time limit from the time when the automatic response AI inquires the user for cooperation as in the statement content 150 In this case, the identifying unit 28 identifies a function (single function) that the device linked to the device image 60 has. The control unit 26 displays information about the single function in the display area 48a as the content of the automatic response AI.

  On the other hand, when the user sends the device image 60 and the functional image 154 to the automatic response AI as the user's utterance content within the time limit from when the automatic response AI inquires the user for the cooperation function as in the utterance content 150 The unit 28 specifies the cooperative function associated with the combination of the PC (A) and the image forming function in the cooperative function management table shown in FIG. The control unit 26 displays information on the cooperation function in the display area 48a.

  If the start time of the time limit is when the user sends the first image (e.g., the device image 60) to the automatic response AI, the same process as described above is performed.

  By providing the time limit as described above, it is possible to switch between providing the cooperation function or the single function.

  Further, when the cooperation function associated with the combination of the device and the function designated by the user is not registered in the cooperation function management table, the identification unit 28 determines that the usable cooperation function is not registered. Do. In this case, the control unit 26 causes the display area 48a to display a message indicating that the available cooperative function is not registered, as the content of the automatic response AI. Even in such a case, the linked function may be available depending on the update status of the device or function. In this case, the identification unit 28 identifies the collaboration function that has become available.

  In addition, in the third embodiment as well, the first to ninth modifications of the first embodiment may be applied.

  The process at the time of applying the modification 1 to 3rd Embodiment is shown by FIG. By applying the first modification, when the information on the device and function as the cooperation target is sent to the automatic response AI as the user's statement content, the device and the cooperative function that can be executed using the function are notified. Be done. Hereinafter, with reference to FIG. 32, a process in the case where the first modification is applied to the third embodiment will be described.

  Similar to FIG. 31, as the statement contents 158 of the automatic response AI, the statement contents such as “which function do you want to use?” Are displayed in the display area 48a.

  When the user designates the image 160 representing the device and the function as the statement content of the user for the above statement content 158 and sends it to the automatic response AI, the identifying unit 28 is represented in the image 160 Identify devices and functions as collaboration targets. For example, when the device image 60 and the functional image 154 are displayed in the image 160, the specifying unit 28 identifies the PC (A) as a device to be cooperated based on the device image 60, and based on the functional image 154. Identify the image forming function as a function to be linked.

  The user may operate the screen 48 to select the device image 60 from the stock area 58 and superimpose the device image 60 on the image 160. Similarly, the user may operate the screen 48 to select the functional image 154 from the stock area 124 and superimpose the functional image 154 on the image 160. As a result, the device image 60 and the functional image 154 are represented in the image 160, and the image 160 is sent to the automatic response AI as the user's statement content.

  As described above, when the PC (A) and the image forming function are identified as the cooperation target, the control unit 26 sets the information of the device and the function identified as the cooperation target in the display area 48a as the utterance content of the automatic response AI. It may be displayed. When the user approves the content of this statement (for example, when the user issues a statement such as "Yes"), the information on the cooperation function that can be executed using the PC (A) and the image forming function is , And may be displayed in the display area 48a as the statement content 162 of the automatic response AI. When the user designates the cooperation function according to the utterance content 164 and gives an execution instruction to the utterance content 162, the control unit 26 executes the designated cooperation function.

  Further, when a total of three or more device images and functional images are displayed in the image 160, information on a collaboration function that can be executed using a total of three or more devices and functions is notified. Also, when one device image or one functional image is displayed in the image 160, a function (single function) that the function associated with the one device image has, or a function associated with the one functional image Information on (single function) is notified.

  In addition, when a device image or a functional image is selected from the stock area, information on a cooperation function or a single function may be notified by control using the above-mentioned time limit. For example, when the device image and the functional image are extracted from the stock area and superimposed on the image 160 within the time limit, the specifying unit 28 uses the device associated with the device image and the function associated with the functional image. Identify feasible collaboration functions. In this case, the information on the cooperation function is displayed in the display area 48a as the contents of the automatic response AI. On the other hand, when one device image or one functional image is taken out of the stock area and superimposed on the image 160 within the time limit, the specifying unit 28 has the function (single function) that the device associated with the device image has Or, specify the function (single function) linked to the functional image. In this case, information on the single function is displayed in the display area 48a as the content of the automatic response AI.

  Further, by applying the modification 2, when the device image and the functional image are sent to the automatic response AI as the user's utterance content, the composite image representing a state in which the device image and the functional image are superimposed on each other. It may be generated (see FIG. 14A), or a composite image may be generated in which the device image and the functional image are combined adjacent to each other (see FIG. 14B). The composite image may be displayed in the stock area. In addition, the composite image may be registered in the terminal device 10 as a shortcut image.

  Further, by applying the modification 3, the cooperation function to be notified may be changed according to the order of sending the information of the device and the function to the automatic response AI. In this case, the connection order and the priority of the device image and the function image are determined in the cooperation function management table according to the third embodiment as well as the cooperation function management table shown in FIG.

  Moreover, by applying the modification 4, different functions may be assigned to each part in the device image, or further different functions may be assigned to each part in the functional image. In this case, when the partial image linked to the part of the device is sent to the automatic response AI as the user's speech content, information on an executable cooperation function is displayed using the part. Similarly, when a partial image tied to a function is sent to the automatic response AI as the user's speech content, information on an executable collaboration function is displayed using the function tied to the partial image. Also, the combination of the third and fourth modifications may be applied to the third embodiment.

  In addition, as the fifth modification is applied, the information on the cooperation function is input to the terminal device 10 by the user as the user's speech contents first, and then, the information on the cooperation target device and function is the user's speech contents It may be input to the terminal device 10.

  Moreover, the notification of a cooperation function may be controlled according to the information which automatic response AI acquired by applying the modification 6. FIG. For example, the automatic response AI is information indicating the usage history of the device and the function, information indicating the execution status of the device, information indicating the execution status of the software for realizing the function, information on the user (operation history and Information indicating an action history), information indicating a schedule of the user, and the like may be acquired, and notification of the cooperation function may be controlled using such information.

  Moreover, the function used as cooperation object may be recommended by the modification 7 being applied. For example, when the user sends a device image to the automatic response AI as the user's statement content, the identification unit 28 identifies the device associated with the device image, and refers to the cooperation function management table to identify the device Identify the functions that can execute the collaboration function in combination. The feature is recommended to the user. Similarly, devices to be linked may be recommended. For example, when the user sends a functional image to the automatic response AI as the user's utterance content, the identifying unit 28 identifies the function associated with the functional image and refers to the cooperation function management table to identify the function. Identify the devices that can execute the collaboration function in combination. The device is recommended to the user. In addition, the identification unit 28 may recommend a device or a function based on the use history of the cooperation function, or may recommend a device or a function with a relatively high frequency of use.

  Further, by applying the modification 8, a function used for the cooperation function may be recommended based on the information shared by the automatic response AI with another automatic response AI.

  Moreover, the modification 9 may be applied, and the information regarding the apparatus and function of cooperation object may be input into the terminal device 10 by audio | voice input.

Fourth Embodiment
The device system according to the fourth embodiment will be described below. In the fourth embodiment, when the information of the file (data) is sent to the automatic response AI as the user's statement content, the information on the cooperation function is notified as the statement content of the automatic response AI. The information of the file is identification information for identifying the file, and is, for example, a file image attached to the file, character information indicating the identification information, voice information or the like. For example, when the user sends a file image associated with a file to the automatic response AI, when the user inputs a character string indicating identification information of the file to the terminal device 10 and gives the automatic response AI, or the user When the identification information of the file is given to the automatic response AI by voice, the identification unit 28 identifies the file. The fourth embodiment will be described in detail below.

  The linked function management information according to the fourth embodiment is information for managing a linked function executed by linking a plurality of functions. The cooperation function is a function realized by linking a plurality of software. Of course, the cooperation function may be a function realized by linking a function of an apparatus as hardware and a function realized by software.

  The cooperation function according to the fourth embodiment will be described in more detail. The cooperation function is a function to be executed on a plurality of files (data) when the files are selected by the user. For example, for each individual file, software (application) for performing processing on the file is linked. The cooperation function is a function realized by causing an application associated with each of a plurality of files selected by the user to cooperate.

  The file is, for example, a document file, an image file, a moving image file, a spreadsheet file, a file for presentation, a figure (drawing) file, an audio file or the like.

  A document file is a file having a document format (document format), and is a file composed of character strings and other information. Images, tables and figures may be included in the document file as other information. The document format is, for example, a text format or a document format. Data having text format (for example, data having a string “. Txt” linked as an extension) and data having a document format (eg having a string “. Doc” or “. Docx” as an extension Data) etc. correspond to the document file.

  An image file is a file having an image formation (image format). The image file may be image data having a raster format, or may be image data having a vector format. The image format is, for example, JPEG format, BMP (bit map) format, PDF format, GIF format, TIFF format, PNG format, PICT format or the like. Data having the JPEG format (data in which the character string “.jpeg” is linked as an extension), data having the BMP format (data in which the character string “.bmp” is linked as an extension), data having a PDF format (data Data with string ". Pdf" linked as an extension), data with GIF format (data with string ". Gif" as an extension), data with TIFF format (string as an extension ".tiff" linked data, data with PNG format (data linked with character string ". png" as extension), data with PICT format linked with character string ". pict" as extension Data) etc. correspond to the image file. Of course, image data having a format other than these may be used as an image file according to the fourth embodiment.

  The moving image file is a file having a moving image format (moving image format). The moving picture format is, for example, AVI format, MP4 format, MOV format, MPEG2-TS format, MPEG2-PS format, WMV format, FLV format, or the like. Data having the AVI format (data in which the character string “.avi” is linked as an extension), data having the MP4 format (data in which the character string “.mp4” is linked as an extension), data having the MOV format (data Data in which the character string ". Mov" is linked as an extension), data having the MPEG2-TS format (data in which the character string ". M2ts" is linked as an extension), data having an MPEG 2- PS format (extension As a string “.mpeg” is associated with data), data having a WMV format (data associated with a string “.wmv” as an extension), data having an FLV format (a string “.flv as an extension Data etc. with which "is attached corresponds to a moving image file. Of course, moving image data having a format other than these may be used as a moving image file according to the fourth embodiment.

  The spreadsheet file is a file having a tabular format (table format), and is a file configured by a table, a graph, and other information. Strings, images and figures may be included in the spreadsheet file as other information. For example, it is data in which character strings ". Csv", ". Xls", and ". Xlsx" are linked as an extension.

  The presentation file is a file having a sheet format (sheet format) for presentation. For example, data associated with the character string “.ppt” or “.pptx” as an extension corresponds to the presentation file. The presentation file is composed of, for example, a character string, an image (still image or moving image), a figure, a sound or the like.

  A graphic file is a file having a graphic (drawing) format (graphic format). The figure file is, for example, a file created by figure creation software (for example, two-dimensional CAD, three-dimensional CAD, etc.). For example, data in the DXF format (format) (data to which the character string “.dxf” is linked as an extension), data having a DWG format (data to which a string “.dwg” is linked as an extension), etc. is there. In addition, data linked with “.vsd” or “.vsdx” as an extension may be treated as a graphic file.

  The audio file is a file having an audio format (audio format). The audio format is, for example, uncompressed audio format (WAV format, AIFF format, BWF format etc.), lossy compressed audio format (mp3 format, AAC format, WMA format etc.), lossless compressed audio format (TAK format, FLAC format etc.) Etc. Data with uncompressed audio format (data with ".wav" etc. attached as an extension), data with lossless compressed audio format (data with a text ".mp3" etc attached with an extension), non-compressed Data having a compressed audio format (data in which a character string “.tak” or the like is linked as an extension) or the like corresponds to the audio file.

  In the fourth embodiment, the cooperation function is determined for each file format (attribute), and the cooperation function management information is, for example, information indicating a combination of a plurality of file formats and information indicating the cooperation function (cooperation function (Information) is information indicating correspondence between The collaboration function information is, for example, a collaboration function ID or a collaboration function name. The file format is the above-mentioned document format, image format, moving image format, tabular format, sheet format, graphic format, audio format, and the like. By referring to the cooperation function management information, the cooperation function corresponding to the combination of a plurality of file formats is identified (identified).

  The cooperation function may be a function realized by linking a plurality of different functions (for example, different softwares) to each other, or may be realized by linking the same function (for example, the same software) It may be a function. The cooperation function may be a function that can not be used before cooperation. The function that can not be used before cooperation may be a function that can be used by linking the same function, or a function that can be used by combining different functions. For example, by linking a document creation function (document creation software) and an image display function (image display software (for example, image viewer)), as a cooperation function, an image insertion (attachment) function to a document, or The superimposing function of the document (character string, text) is realized. That is, by linking the document creation function and the image display function, an image insertion function and a text overlay function are realized.

  The category of the concept of linkage functions may include a combination function that enables execution of new functions by linking a plurality of functions. For example, an extended display function as a combined function may be realized by combining a plurality of display functions. In addition, by combining a plurality of shooting functions, a shooting area expansion function as a combination function may be realized. In addition, by combining the call function and the translation function, a translation call function (function of translating the contents of a call) as a combination function may be realized. As described above, the category of the concept of linkage functions includes functions that can be realized by linking the same type of functions with each other, and functions that can be achieved by linking different types of functions with each other. It is also good.

  The cooperation function managed by the cooperation function management information may be a function executable by the software installed in the terminal device 10 or may be installed in an external device (for example, a server or another terminal device) It may be a function that can be executed by certain software, or a function that can be executed by the software installed in the terminal device 10 and the software installed in the external device.

  Note that, as another example of the linked function management information, information indicating a combination of a plurality of functions (plural software) and information indicating a linked function that can be executed by linking the plurality of functions (linked function information) , May be associated.

  When a file image associated with each of a plurality of files is sent to the automatic response AI as the user's comment content, the identification unit 28 corresponds to the combination of the format (file format) of the plurality of files in the cooperation function management information. Identify the attached linkage function. Thereby, an executable cooperation function is identified.

  The cooperative function management information according to the fourth embodiment will be described in detail below with reference to FIG. FIG. 33 shows an example of the cooperation function management table according to the fourth embodiment. In the linked function management table, as an example, information indicating a combination of a plurality of file formats and information indicating a linked function (linked function information) are associated with each other. For example, when the user sends information of a plurality of files to the automatic response AI as the user's statement content, the identification unit 28 is associated with a combination of file formats of the plurality of files in the linkage function management table. Identify the linked functions. Information on the cooperation function is displayed on the display unit of the UI unit 20. Note that although a combination of two file formats is shown in FIG. 33, an association between a combination of three or more file formats and a linkage function may be registered in the linkage function management table.

  Hereinafter, each cooperation function shown in FIG. 33 will be described.

  The “document format” in FIG. 33 is a text format (the extension is “.txt”) or a document format (the extension is “.doc” or “.docx”). The “table format” is a format in which the extension is expressed by “.csv”, “.xls” or the like. The image format is a format in which the extension is expressed by “.jpeg” or “.pdf”. The video format is a format in which the extension is expressed by ".avi", ".mpeg" or the like. The sheet format is a format in which the extension is expressed by “.ppt” or the like. Although the graphical format and the audio format are not shown in FIG. 30, the association between the combination including these and the cooperation function may be registered in the cooperation function management table. Of course, the association between the combination including the file format other than the file format shown in FIG. 33 and the linkage function may be determined and registered in the linkage function management table.

  Hereinafter, each cooperation function shown in FIG. 33 will be described. When the information of document file A and spreadsheet file B is sent by the user to automatic response AI as the user's comment contents, "the function of inserting a table into the document" associated with the combination of the document format and the table format And "the function of inserting a document into a table" are identified as the cooperation function. The "function to insert a table into a document" is a function to insert the table represented by the spreadsheet file B into the document represented by the document file A. "Function to insert a document into a table" is a function to insert the document represented by the document file A into the table represented by the above-mentioned spreadsheet file B.

  When the information of the document file A and the image file C is sent to the automatic response AI as the user's comment contents by the user, “a function to insert an image into the document” associated with the combination of the document format and the image format The "function to superimpose a document on an image" is specified as a cooperation function. The “function to insert an image into a document” is a function to insert the image represented by the image file C into the document represented by the document file A. The “function to superimpose a document on an image” is a function to superimpose the document represented by the document file A on the image represented by the image file C.

  When the information of the document file A and the moving image file D is sent by the user to the automatic response AI as the user's comment contents, “the function of inserting the moving image into the document” associated with the combination of the document format and the moving image format The "function to insert a still image into a document" is specified as the cooperation function. The “function to insert a moving image into a document” is a function to insert the moving image represented by the moving image file D into the document represented by the document file A. The “function of inserting a still image into a document” is a function of inserting one or more frames (still images) constituting the moving image file D into the document represented by the document file A. For example, in the case where the document file A corresponds to a document file which does not correspond to a moving image, that is, a file to which a moving image can not be inserted, the “moving image into a document” function is executed. One or more frames (still images) in D are inserted into the document represented by document file A.

  Multiple files having the same file format may be selected by the user. For example, when the information of the document files A1 and A2 is sent by the user to the automatic response AI as the user's comment contents, the "documents combined function associated with the combination of the document format and the document format" cooperates Identified as a function. “Function to merge documents” merges the documents represented by the document files A1 and A2 (for example, combines two documents), while leaving the document files A1 and A2, or the document files A1 and A2 , And another new document file A3 is generated.

  When the information of spreadsheet files B1 and B2 is sent by the user to the automatic response AI as the user's comment contents, the "table-to-table combination function" associated with the combination of the table format and the table format is a cooperation function Identified as “Function to merge tables” merges the tables represented by the spreadsheet files B1 and B2 (for example, by combining two tables), while leaving the spreadsheet files B1 and B2, or This is a function to generate another new spreadsheet file B3 without leaving B1 and B2.

  When the information of moving image files D1 and D2 is sent by the user to the automatic response AI as the user's statement content, the "function to unite moving images" associated with the combination of the moving image format and the moving image format is a linked function. It is identified. "Function to combine moving images" combines moving images represented by moving image files D1 and D2 (for example, combining two moving images), while leaving moving image files D1 and D2, or moving image files D1 and D2 , And another new moving image file D3 is generated.

  In addition, when the information of the presentation file E and the document file A is sent to the automatic response AI as the user's comment contents by the user, “the document is associated with the combination of the sheet format and the document format” "Function to be inserted" is specified as a cooperation function. The "function for inserting a document into a sheet for presentation" is a function for inserting the document represented by the document file A into the sheet represented by the file for presentation E.

  Hereinafter, the fourth embodiment will be described in detail by giving a specific example.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 166 such as “which file do you want to use?” Is displayed in the display area 48 a as the utterance content of the automatic response AI.

  When the user designates a file for the message content 166, the identifying unit 28 (the identifying unit 28 realized by the automatic response AI) identifies the file as a file to be cooperated. For example, as shown in FIG. 34, the user designates the file image 168 linked to the document file A, and arranges the file image 168 at a position where the user's speech content is displayed. The file image 168 is sent to the automatic response AI as the user's speech content. In response to this operation, the identification unit 28 identifies the document file A linked to the file image 168 as a file to be cooperated. In the example shown in FIG. 34, the operation of causing the user to display the file image 168 in the display area 48b corresponds to the action of the user emitting the content of the statement. The user may operate the UI unit 20 to input a character string indicating a file name to the terminal device 10 as the user's statement content. In this case, the identification unit 28 identifies the file having the file name indicated by the character string as the file to be cooperated. In addition, information of a file to be cooperated may be input to the terminal device 10 as voice content of the user by voice.

  For example, as in the first embodiment, the control unit 26 displays the stock area 170 in the display area 48b for the user, and displays a file image in the stock area 170. For example, the control unit 26 displays a file image associated with the file stored in the terminal device 10 in the stock area 170. In addition, when there are other file images that can not be displayed in the stock area 170, the user scrolls the file images displayed in the stock area 170 to make the other file images in the stock area 170. Is displayed on.

  Similar to the first embodiment, the user selects a file image (file image 168 in the example shown in FIG. 34) associated with the file to be cooperated from the stock area 170 and moves the file image to the outside of the stock area 170 The image 168 can be sent to the automatic response AI as the user's speech content. For example, the user moves the file image 168 from the stock area 170 to a position where the user's speech content is displayed by a drag operation, and performs a drop operation at the display position of the speech content. By this operation, the control unit 26 sends the file image 168 to the automatic response AI as the user's speech content.

  Further, in the example shown in FIG. 34, following the file image 168, the user selects the file image 172 linked to the image file C from the stock area 170, and moves the file image 172 to the display position of the user's statement content. ing. By this operation, the file image 172 is sent to the automatic response AI as the user's speech content.

  When the user sends information of a plurality of files to the automatic response AI on the screen 48, that is, when information of a plurality of files is uttered as information of a file to be cooperated, the specifying unit 28 is shown in FIG. In the linked function management table, the linked function associated with the plurality of files is specified. The control unit 26 displays information on the cooperation function in the display area 48a as the contents of the automatic response AI.

  In the above example, the file images 168 and 172 are sent to the automatic response AI as the function to be linked. In this case, the identification unit 28 (the identification unit 28 realized by the automatic response AI) identifies the document file A attached to the file image 168 and the image file C attached to the file image 172 as files to be cooperated. Alternatively, the file image 168 may be sent to the automatic response AI first, and then the file image 172 may be sent to the automatic response AI, or the file image 172 may be sent first to the automatic response AI, and then the file image 168 may be automatic response May be sent to AI.

  As described above, when the file images 168 and 172 are sent to the automatic response AI and the document file A and the image file C are identified as the file to be cooperated, the control unit 26 determines the file to be identified as the file to be cooperated The above information may be displayed in the display area 48a as the contents of the automatic response AI. When the user approves the content of this statement (for example, when the user issues a statement such as "Yes"), information on the collaboration function that can be executed using document file A and image file C is displayed. It is displayed in the area 48a.

  The identifying unit 28 identifies the file format of the document file A based on the extension of the document file A (for example, the extension associated with the document file A as attribute information of the document file A). Similarly, the identification unit 28 identifies the file format of the image file C based on the extension of the image file C (for example, the extension associated with the image file C as attribute information of the image file C). The extension may be included in the file name. When the extension of the document file A corresponds to the extension indicating the document format such as “.txt” or “.docx”, the identification unit 28 identifies the file format of the document file A as the document format Do. When the extension of the image file C corresponds to the extension representing the image format such as “.jpeg” or the like, the specification unit 28 identifies the file format of the image file C as the image format. Then, the specifying unit 28 sets the cooperation function (“function to insert an image into a document” and “document to be associated”) associated with the combination of the document format and the image format in the linkage function management table shown in FIG. Identify the function to be superimposed on the image. The information on the cooperation function specified in this manner is displayed in the display area 48a as the content of the automatic response AI. Note that the process of specifying the file and the cooperation function may be performed by the server 16.

  The control unit 26 displays information related to the cooperation function in the display area 48a as the statement contents 174 of the automatic response AI.

  When the user designates the cooperation function according to the statement content and gives an execution instruction to the statement content 174, the control unit 26 (the control unit 26 realized by the automatic response AI) receives the execution instruction and performs the cooperation. Perform a function For example, when an instruction to execute “a function to insert an image into a document” is given by the user, the control unit 26 inserts the image represented by the image file C into the document represented by the document file A.

  More specifically, the control unit 26 activates the document application and the image application. The document application is software (program) having a function of generating, editing, and displaying a file having a document format. The image application is software (program) having a function of generating, editing, and displaying a file having an image format. The document application may be associated with the document file A in advance, and the image application may be associated with the image file C in advance. The control unit 26 copies the image from the image file C by using the function of the image application, and pastes the image in the document represented by the document file A by using the function of the document application. wear. This generates a document into which the image has been inserted. The control unit 26 may extract a partial image representing a specific part from the image represented by the image file C by using a function having an image application, and may paste the partial image into the document. The area to which the image is attached in the document may be a predetermined area, an area specified by the user, or it may be inferred that a character string related to the image is described. Region may be used. For example, the control unit 26 infers or searches for a character string representing the feature portion represented in the image, and in the document, an area in which a character string having the same or similar meaning as the character string is described is It is inferred that it is an area where a character string related to an image is described. Of course, the area may be estimated by another method.

  Even when an instruction to execute the function of "superimpose a document on an image" is given by the user, all or a part of the document represented by the document file A is copied by the function of the document application, The copied document is superimposed on the image represented by the image file C. The portion to be copied in the document may be predetermined or designated by the user. Also, the portion of the image on which the document is superimposed may be predetermined or may be designated by the user.

  The control unit 26 may generate a new file by executing the cooperation function. In this case, the plurality of files selected as collaboration targets are not edited. For example, when the document file A and the image file C are selected as files to be cooperated and, for example, an execution instruction of "function to insert an image into a document" is given, the control unit 26 selects the document represented by the document file A By pasting the image represented by the image file C into it, a new document file AC is generated. In this case, the control unit 26 may generate a new file image (for example, an icon) associated with the document file AC, and display the file image on the screen 48. In this case, the document file A and the image file C themselves are not edited.

  Also in the case where an instruction to execute “the function of superposing a document on an image” is given, the control unit 26 similarly causes the whole or one of the documents represented by the document file A on the image represented by the image file C A new image file CA is generated by superimposing parts. Also in this case, the document file A and the image file C themselves are not edited.

  As another example, the control unit 26 may edit (including processing and the like) the file itself selected as the cooperation target by executing the cooperation function without generating a new file. In the above example, the control unit 26 pastes the image represented by the image file C into the document represented by the document file A, and sets the document file in which the pasting of the image is reflected as the document file A. save. In this case, the document file A is updated.

  As another example, a file to be cooperated may be preset as a basic cooperation file. The information of the basic cooperation file may be stored in advance in the storage unit 24 of the terminal device 10. The user may specify the basic cooperation file using the UI unit 20. When the basic cooperation file is set, the user sends information on the file to be cooperated other than the basic cooperation file to the automatic response AI as the user's comment content. Thus, the file to be cooperated is identified (identified).

  As described above, according to the fourth embodiment, on the screen 48 for interacting with the automatic response AI, the information on the collaboration function is provided by sending the information of the file to be cooperated as the interactive content to the automatic response AI. Be done. As described above, the information on the cooperation function is provided by the simple method using the interaction with the automatic response AI.

  In addition, when three or more file images are sent to automatic response AI as a user's speech content, the information regarding the cooperation function matched with the combination of the format of three or more files may be displayed.

  In the fourth embodiment, the display of information related to the cooperation function and the cooperation function to be executed may be changed according to the master-slave relationship between the files. For example, a file capable of editing the content itself represented by the file corresponds to the main file, and a file not capable of editing the content itself corresponds to the secondary file. As an example, when the document itself represented by the document file can be edited, the document file corresponds to the main file. If the image itself represented by the image file can not be edited (that is, although the content can be superimposed on the image but the image itself can not be edited), the image file is It corresponds to the secondary file. The same is true for other files. Of course, even if it is a document file, a document file which can not edit a document corresponds to a secondary file, and even if it is an image file, an image file which can edit an image corresponds to a main file.

  The identifying unit 28 may determine whether the file corresponds to the main file or the sub file based on the file format (i.e., the extension). For example, the identification unit 28 determines that the file having the document format, the file having the table format, the file having the sheet format, and the file having the graphic format correspond to the main file, and the file having the image format, the moving image format It may be determined that the file having the file name and the file having the audio format correspond to the secondary file. Of course, the determination as to whether the file corresponds to either the main file or the secondary file can not be made indiscriminately according to the file format, but the file is mainly determined by individually determining the file selected as the cooperation target. It may be determined whether the file or the secondary file corresponds. For example, even if the file has an image format, the file may correspond to a main file as long as the file can be edited.

  For example, when the file image 168 linked to the document file A and the file image 172 linked to the image file C are sent to the automatic response AI as the user's comment contents, the specification unit 28 sets the extension linked to the document file A It is determined whether the document file A corresponds to the main file or the secondary file based on the image file C, and the image file C corresponds to the main file or the secondary file based on the extension associated with the image file C. Decide what to do. The extension of the document file A is, for example, “.docx”, and the extension of the image file C is, for example, “.jpeg”. In this case, since the document file A has a document format and corresponds to an editable file, the specifying unit 28 determines that the document file A corresponds to the main file. Further, since the image file C has an image format and corresponds to a non-editable file, the specifying unit 28 determines that the image file C corresponds to a secondary file. Of course, the determination unit 28 determines whether the document file A and the image file C correspond to editable files by individually determining the document file A and the image file C without making a determination based on the extension. You may decide whether or not.

  Since the document file A corresponds to the main file and the image file C corresponds to the secondary file, the image represented in the image file C corresponds to the content to be pasted and the document represented in the document file A is the image It corresponds to the content of paste destination. In this case, the specification unit 28 specifies “a function of inserting an image into a document” as a cooperation function. The control unit 26 displays information on "the function of inserting an image into a document" in the display area 48a as an utterance content of the automatic response AI. When a plurality of cooperation functions are specified by the specifying unit 28, the control unit 26 causes information on the plurality of cooperation functions to be displayed in the display area 48a as the statement content of the automatic response AI.

  In addition, among the plurality of cooperation functions registered in the cooperation function management table shown in FIG. 33, control unit 26 performs another cooperation on the information related to the cooperation function specified based on the master-slave relationship of the file. The message content of the automatic response AI may be displayed in the display area 48a in preference to the information on the function. The document file A and the image file C will be described by way of example. "Function to insert an image into a document" and "Function to overlap a document on an image" as linked functions corresponding to the combination of the document format and the image format Are registered in the cooperation function management table (see FIG. 33). In this case, the specifying unit 28 specifies the “function to insert an image into a document” specified based on the master-slave relationship of the file as a cooperative function with “1st place” in priority, and “superimposes the document on the image "Function" is specified as a "2nd place" linked function with the priority "2". According to the priority order, the control unit 26 gives priority to (for example, above) information on the "function to insert an image into a document" over the information on "a function to superimpose a document on an image" It is displayed in the display area 48a.

  In addition, when a plurality of cooperation functions are specified based on the master-slave relationship of the file, the control unit 26 determines the priority of each cooperation function based on the usage history of each cooperation function by the user, and gives priority to each cooperation function. Information on each cooperation function may be displayed on the screen 48 as the contents of the automatic response AI according to the order. For example, the control unit 26 causes the display area 48 a to be displayed in the display area 48 a with higher priority (for example, higher rank) on information related to the frequently used cooperative function.

  Further, when a plurality of linkage functions are specified based on the master-slave relationship of files, the control unit 26 determines the relationship between the plurality of files selected as linkage targets (that is, the relationship of the plurality of file formats). The priority order of each cooperation function may be determined, and information on each cooperation function may be displayed in the display area 48a as the content of the automatic response AI according to the priority order of each cooperation function. For example, the control unit 26 determines, based on a plurality of file formats, the degree of the possibility of being used for each cooperation function included in the plurality of cooperation functions identified by the identification unit 28. The control unit 26 causes the display area 48 a to be displayed with priority (for example, higher rank) in the information regarding the cooperation function that is likely to be used. For example, in the combination of the document format and the image format, it is predicted that the function of inserting an image into a document is more likely to be used than the function of superposing a document on an image. In this case, the control unit 26 causes the information related to the "function to insert an image into a document" to be displayed in the display area 48a in preference to the information related to the "function to overlap a document on an image".

  In addition, when the information of a plurality of files is sent to the automatic response AI as the user's comment contents within a predetermined time limit, the specifying unit 28 specifies an executable collaboration function using the plurality of files. If the information of one file is sent to the automatic response AI as the user's speech contents within the time limit, the single file may be used as an executable single function. In this case, information on the linked function or the single function specified by the specifying unit 28 is displayed on the display unit of the UI unit 20. The start time of the time limit may be, for example, the time when the automatic response AI inquires the user for the cooperation target, as in the case of the statement content 166, or after the generation of the statement content 166, the user can obtain the first file image. It may be at the time of sending to the automatic response AI.

  For example, in the example shown in FIG. 34, the user sends only the file image 168 as the user's utterance content to the automatic response AI within the time limit from the time when the automatic response AI inquires the user for cooperation as in the statement content 166 In this case, the identification unit 28 identifies an executable function (single function) using the file associated with the file image 168. The control unit 26 displays information about the single function in the display area 48a as the content of the automatic response AI.

  On the other hand, when the user sends the file image 168 and the file image 172 to the automatic response AI as the user's utterance content within the time limit from when the automatic response AI inquires the user for the cooperation function as in the utterance content 166 The unit 28 specifies the cooperation function associated with the combination of the document file (file having the document format) and the image file (file having the image format) in the cooperation function management table shown in FIG. . The control unit 26 displays information on the cooperation function in the display area 48a.

  If the start time of the time limit is when the user sends the first image (e.g., file image 168) to the automatic response AI, the same process as described above is performed.

  By providing the time limit as described above, it is possible to switch between providing the cooperation function or the single function.

  In addition, when the cooperation function associated with the combination of a plurality of file formats designated by the user is not registered in the cooperation function management table, the identification unit 28 determines that the usable cooperation function is not registered. Do. In this case, the control unit 26 causes the display area 48a to display a message indicating that the available cooperative function is not registered, as the content of the automatic response AI. Even in such a case, the linked function may be available depending on the update status of the device or software. In this case, the identification unit 28 identifies the collaboration function that has become available.

  Further, in the fourth embodiment as well, the first to ninth modifications of the first embodiment may be applied.

  The process at the time of applying the modification 1 to 4th Embodiment is shown by FIG. By applying the first modification, when information on a plurality of files to be cooperated is sent to the automatic response AI as the user's statement content, an executable collaboration function is notified using the plurality of files. Ru. The process in the case where the first modification is applied to the fourth embodiment will be described below with reference to FIG.

  As in FIG. 34, as the statement contents 176 of the automatic response AI, the statement contents such as “which file do you want to use?” Are displayed in the display area 48 a.

  When the user designates the image 178 in which a plurality of files are represented as the statement content of the user and sends it to the automatic response AI with respect to the statement content 176 described above, the specification unit 28 is represented in the image 178 Identify multiple files as linked files. For example, when the file images 168 and 172 are represented in the image 178, the specifying unit 28 identifies the document file A as a file to be cooperated based on the file image 168, and the image file C based on the file image 172. Is identified as a file to be linked.

  The user may operate the screen 48 to select the file images 168 and 172 from the stock area 170 and superimpose the file images 168 and 172 on the image 178. As a result, the file images 168 and 172 are represented in the image 178, and the image 178 is sent to the automatic response AI as the user's speech content.

  When the document file A and the image file C are identified as collaboration targets as described above, the control unit 26 may display information of the files identified as collaboration targets on the screen 48 as the content of the automatic reply AI. . When the user approves the content of this statement (for example, when the user issues a statement such as "Yes"), information on the collaboration function that can be executed using the document file A and the image file C is The message content 180 of the automatic response AI may be displayed in the display area 48a. When the user designates the cooperation function according to the speech content 182 and gives an execution instruction to the speech content 180, the control unit 26 executes the designated cooperation function.

  Further, when three or more file images are displayed in the image 178, information on an executable collaboration function is notified using three or more files. In addition, when one file image is displayed in the image 178, information on an executable function (single function) is notified using the file associated with the one file image.

  Further, when the file image is selected from the stock area, information on the cooperation function or the single function may be notified by the control using the above-mentioned time limit. For example, when a plurality of file images are extracted from the stock area and superimposed on the image 178 within the time limit, the specifying unit 28 specifies an executable cooperation function using the plurality of files. In this case, the information on the cooperation function is displayed in the display area 48a as the contents of the automatic response AI. On the other hand, when one file image is taken out of the stock area and superimposed on the image 178 within the time limit, the specifying unit 28 can execute the function (single function) using the file associated with the file image. Identify In this case, information on the single function is displayed in the display area 48a as the content of the automatic response AI.

  In addition, by applying the modification 2, when a plurality of file images are sent to the automatic response AI as the user's utterance content, a composite image representing a state in which the plurality of file images are superimposed on each other is generated. Alternatively, a composite image in which the plurality of file images are combined adjacent to each other may be generated (see FIG. 14B). The composite image may be displayed in the stock area 170. In addition, the composite image may be registered in the terminal device 10 as a shortcut image.

  Further, by applying the modification 3, the cooperation function to be notified may be changed according to the order of sending information of a plurality of files to the automatic response AI. In this case, the connection order and the priority of a plurality of file images are determined also in the cooperation function management table according to the fourth embodiment, as in the cooperation function management table shown in FIG.

  Further, by applying the modification 4, a different function may be assigned to each part in the file image. In this case, when the partial image is sent to the automatic response AI as the user's speech content, information on an executable collaboration function is displayed using the function associated with the part. Also, the combination of the third and fourth modifications may be applied to the fourth embodiment.

  Further, by applying the fifth modification, the information about the cooperation function is input to the terminal device 10 by the user as the user's speech contents first, and then, the information regarding the file to be cooperated as the user's speech contents. 10 may be input.

  Moreover, the notification of a cooperation function may be controlled according to the information which automatic response AI acquired by applying the modification 6. FIG. For example, the automatic response AI is information indicating the use history of the file, information indicating the execution status of the device, information indicating the execution status of the software for realizing the function, information on the user (operation history and action history) Information indicating the user's schedule, etc. may be acquired, and notification of the cooperation function may be controlled using such information.

  Moreover, the function used as cooperation object may be recommended by the modification 7 being applied. For example, when the user sends a file image to the automatic response AI as the user's comment content, the identifying unit 28 identifies the file associated with the file image, and refers to the cooperation function management table to determine the format of the file. Identify the functions that can execute the collaboration function in combination with. Further, the specifying unit 28 may recommend a file based on the use history of the cooperation function, or may recommend a file with a relatively high frequency of use.

  Further, by applying the modification 8, a function used for the cooperation function may be recommended based on the information shared by the automatic response AI with another automatic response AI.

  Moreover, the modification 9 may be applied, and the information of the file to be cooperated may be input to the terminal device 10 by voice input.

  Further, in the fourth embodiment, when the information indicating the names (file names) of a plurality of files is sent to the automatic response AI as the user's comment contents, the specifying unit 28 selects the cooperation function corresponding to the plurality of file names. It may be specified. In the cooperation function management table for that, the combination of file names and the cooperation function are associated with each other, and the identifying unit 28 associates the combination with a plurality of file name combinations by referring to the cooperation function management table. Identify the linked functions that are being

  Further, the specification unit 28 may specify the file to which the content is to be added, based on the name of the file sent to the automatic response AI. For example, in the name (file name) of the file sent to the automatic response AI, the character strings "regulatory book", "summary", "management", "statistics", "graph", "graph", "table", "list", etc. When the identification unit 28 includes a character string that indicates that the data is to be collected (that is, a file that is assumed or predicted to be added with the content of other data), the identifying unit 28 , As a destination file for adding content. In addition, the identification unit 28 may be a file having a file name not including the above-described summary character string, or a file having a file name including a character string indicating that the file is a file providing content (material). , As a file containing content to be added (material to be added). For example, a file having a file name including the character string "receipt", "slip", "personal data", "device data", etc. does not correspond to the above summary file, and content (material) It is determined that the file is provided.

  For example, in the case where document files A1 and A2 are sent to automatic response AI as user's comment contents as files to be cooperated, document file A1 has a file name including the character string for summary described above, and is a document file When A2 does not include a character string for summarizing and has a file name including a character string for providing material, the specification unit 28 writes the document file A2 on the material side into the document file A1 for collecting, copies, and statistical processing Identify the cooperation function that performs etc. as a priority cooperation function. The control unit 26 displays the preferential cooperation function on the screen 48 in preference to other cooperation functions.

  In addition, the combination of cooperation object is not limited to the combination in said 1st Embodiment to 4th Embodiment. For example, if at least two combinations of equipment, functions (including functions performed by equipment and functions performed by software), files, and software are selected as collaboration targets, using the combination Notification of executable collaboration functions may be controlled. For example, when the device image and the file image are sent to the automatic response AI as the user's utterance content, the specifying unit 28 can execute the process using the device associated with the device image and the file associated with the file image. The cooperation function may be specified, and the control unit 26 may display the information on the cooperation function on the screen 48 as the statement content of the automatic response AI as a notification of the information on the cooperation function, or may output it as voice information. Good. Describing the specific example, when the device image associated with the multifunction machine B and the file image associated with the document file are sent to the automatic response AI as the user's statement content, the specification unit 28 determines that the MFP B and the document A cooperative function that can be executed using a file (for example, a function of printing a document file by the multifunction machine B, a function of transmitting a document file by facsimile by the multifunction machine B, etc.) is specified. The control unit 26 controls notification of information on the cooperation function. Similarly, when the functional image associated with software and the file image are sent to the automatic response AI as the user's comment content, the specification unit 28 uses the function associated with the functional image and the file associated with the file image. The control unit 26 may specify an executable cooperative function, and control the notification of information on the cooperative function. Describing the specific example, when the functional image associated with the document application and the file image associated with the document file are sent to the automatic response AI as the user's statement content, the identifying unit 28 determines the document application and the document file. Specify an executable linkage function (eg, a function for editing a document file by a document application, a function for outputting, etc.). In addition, when the device image, the function image, and the file image are sent to the automatic response AI as the user's utterance content, the specifying unit 28 includes the device associated with the device image and the function (for example, software) associated with the function image. The executable cooperation function may be specified using the file associated with the file image, and the control unit 26 may control notification of information on the cooperation function. A specific example will be described. When the device image tied to the multifunction machine B, the functional image tied to the document application, and the file image tied to the document file are sent to the automatic response AI as the user's comment content, The identification unit 28 can execute an executable linkage function using the MFP B, the document application, and the document file (for example, edit the document file by the document application and output the edited file to the MFP B. Specify functions such as printing and facsimile transmission of files. The combination in the above-mentioned specific example is only an example, and the cooperation function which can be performed combining equipment, a function, a file, and software except the above may be defined. Also, information of equipment, functions, files and software may be transmitted to the automatic response AI by voice.

  The above-described first to fourth embodiments and the first to ninth modifications may be applied when setting an automatic execution process by RPA (Robotics Process Automation). For example, when the information on the device, function, or file is sent to the automatic response AI by the setter, the combination of the sent information and the associated function associated with the combination are stored as the setting content, Processing is executed according to the setting contents. As RPA, for example, routine processing such as work that can be executed by data input or a combination of multiple application software, processing such as data collection and analysis, processing to learn and judge based on data, etc. are performed. Good.

Fifth Embodiment
The device system according to the fifth embodiment will be described below. In the fifth embodiment, when an instruction to use a device (use request) is given to the automatic response AI on an interface (for example, the screen 48) in which the user interacts with the automatic response AI (for example, automatic response is made to the device information When sent to AI), the display of the other party in the response is switched to the display of the device. The fifth embodiment will be described in detail below.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, as the statement content of the automatic response AI, the statement content 184 such as "How did it happen?" Is displayed in the display area 48a. Since the message content 184 is the message content of the automatic response AI, the control unit 26 displays the image 50 linked to the automatic response AI in the display area 48 a and links the message content 184 to the image 50 of the automatic response AI The message content is displayed in the display area 48a. Note that this automatic response AI is a device-independent automatic response AI and can be said to be a general-purpose automatic response AI.

  When the user designates a device with respect to the utterance content 184, the identification unit 28 (the identification unit 28 realized by the automatic response AI) identifies the device. For example, it is assumed that the user issues an utterance content 186 such as “I want to use the multifunction device B” as the utterance content indicating the usage instruction (usage request) of the device. The control unit 26 causes the display area 48b to display the message content 186 including the information of the MFP B as the user's message content, and sends the message content 186 to the automatic response AI. Thereby, the information of the multifunction machine B is sent to the automatic response AI. The statement content 186 is displayed in the display area 48b as the statement content linked to the image 52 of the user. The user may input a character string indicating the device as the user's statement content by operating the UI unit 20 to the terminal device 10, or, as in the first embodiment described above, a device image associated with the device For example, an appearance image or an image such as an icon may be designated as the user's speech content. When specifying a device image, as shown in FIG. 8, the user selects a device image linked to the device to be used from the stock area 58, and moves the device image to the display position of the user's statement content By doing this, the device image may be designated as the user's utterance content.

  The identifying unit 28 identifies the function of the device designated by the user, for example, by referring to the device function management table shown in FIG. In the above example, since the multifunction machine B is designated, the specification unit 28 identifies the function that the multifunction machine B has.

  When the user specifies a device and the device is identified, the control unit 26 displays a display of the conversation partner (for example, an image associated with the conversation partner or a character string indicating the conversation partner), a general-purpose automatic response AI display Display of the device designated by the user from (for example, an image associated with the general-purpose automatic response AI or a character string indicating the general-purpose automatic response AI) (for example, an image associated with the device or a character string indicating the device) Switch to

  Even when the display of the conversation partner is switched from the display of the general-purpose automatic response AI to the display of the device, the user's dialog partner itself is the general-purpose automatic response AI. That is, the contents of the conversation partner's utterance are created by the general-purpose automatic response AI mounted on the terminal device 10.

  In the above example, since the multifunction machine B is designated as the device to be used, the control unit 26 substitutes the device image 188 tied to the multifunction machine B as the dialogue partner in place of the image 50 tied to the automatic response AI. It is displayed in the display area 48a as an image to be linked. The device image 188 may be an image (for example, an external appearance image of the multifunction device B) generated by photographing the multifunction device B, or may be an image (for example, an icon) imitating the multifunction device B. Of course, instead of the device image 188 or together with the device image 188, the control unit 26 may display a character string indicating the multifunction device B in the display area 48a as information indicating the other party of interaction.

  Further, instead of the image 50 linked to the automatic response AI, the control unit 26 may display a device image representing the status of the device in the display area 48a as an image linked to the conversation partner. In the above-described example, the control unit 26 causes the device image 188 representing the state of the multifunction device B to be displayed in the display area 48 a as an image associated with the other party in dialogue. The control unit 26 acquires information indicating the state of the device from the device to be used, and controls the display of the device image indicating the state. For example, when the device is executing a process, the control unit 26 displays the device image representing the state of executing the process as an image linked to the other party of interaction, and the device is not executing the process. In this case, a device image representing a state in which the process is not performed is displayed as an image tied to the interaction partner, and when the device state corresponds to the sleep state, a device image representing the sleep state is displayed as an image tied to the conversation partner When the power of the device is turned off, a device image representing that the power is turned off is displayed as an image linked to the other party in the dialogue. By doing this, the user can visually confirm the state of the device. For example, when the MFP B is performing printing, the control unit 26 causes the display area 48a to display a device image 188 representing a state in which the MFP B is printing as a string associated with the other party. The image representing the state of the device may be created in advance and stored in the storage unit 24 or may be created by the control unit 26. The control unit 26 may display a character string indicating the state of the device in the display area 48a instead of or together with the device image.

  Further, the control unit 26 causes the display area 48a to display the statement contents 190 such as “Which function do you want to use?” As the statement contents of the MFP B. The statement content 190 itself is the statement content created by the general-purpose automatic response AI. The control unit 26 displays the message content 190 as the message content linked to the device image 188 of the multifunction device B in the display area 48 a in order to produce effects as if the multifunction device B is making a statement content 190 Let The control unit 26 may display a list of functions possessed by the multifunction device B identified by the identifying unit 28 in the display area 48 a as the message content of the multifunction device B. The MFP B is, for example, a device having an image forming function such as a print function and a copy function, so the control unit 26 displays a character string or an image showing a list of the image forming functions as the message content of the MFP B. It may be displayed in the area 48a.

  The user inputs processing content to be executed using the MFP B to the terminal device 10 as the user's utterance content in response to the utterance content 190 described above. In the example shown in FIG. 36, the statement content 192 indicating the processing content such as “I want to print the file X” is input to the terminal device 10 as the statement content of the user.

  The control unit 26 analyzes the user's comment content 192 to specify the processing content requested by the user, and causes the MFP B to execute the processing content. For example, the control unit 26 transmits a print job including data of the file X to the multifunction peripheral B. The MFP B that has received the print job prints the file X in accordance with the print job. The control unit 26 may cause the UI unit 20 to display a screen for setting printing conditions (for example, conditions such as the number of copies, color printing, black and white printing, etc.). The print job includes the conditions set on the screen, and the multifunction peripheral B performs printing in accordance with the conditions.

  When the printing of the file X is completed, the control unit 26 causes the display area 48a to display the message content 194 indicating that the printing is completed, as the message content of the MFP B. Also at this time, the control unit 26 displays the device image 188 linked to the multifunction machine B in the display area 48a, and the message content 194 as the message content linked to the device image 188 of the multifunction machine B in the display area 48a. Display. Note that the statement content 194 itself is the statement content created by the general-purpose automatic response AI.

  When the processing by the multifunction machine B is completed, the control unit 26 substitutes the device image 188 tied to the multifunction machine B and displays the image 50 tied to the general-purpose automatic response AI as an image tied to the dialogue partner in the display area 48a. Display on. In addition, the control unit 26 causes the display area 48a to display, as the statement content of the automatic response AI, the statement content 196 indicating a general question such as "What happened?"

  As described above, according to the fifth embodiment, when an instruction to use a device (use request) is given to the automatic response AI on the screen 48 that interacts with the automatic response AI (for example, the information on the device is transmitted to the automatic response AI) If sent, the display of the other party in the response is switched to the display of the device. By doing this, the user can visually confirm whether or not an instruction can be given to the device. For example, when the user sends information of the device to the automatic response AI to give an instruction to the device, an image linked to the device is displayed as an image linked to the interaction partner. By doing this, the user can visually confirm whether or not the instruction is actually given to the device.

(Modification 10)
Hereinafter, modification 10 will be described. In the tenth modification, when the instruction to use the device is given to the automatic response AI as the user's statement content, the display of the conversation partner is switched from the display of the general-purpose automatic response AI to the display of the device and The automatic response AI (for example, a program) is switched from the general-purpose automatic response AI to the automatic response AI mounted on the device.

  In the above example, when the automatic response AI is mounted on the multifunction machine B (when the program of the automatic response AI is installed on the multifunction machine B), the control unit 26 automatically performs the process as the user's interactive partner. The response AI is switched from a general-purpose automatic response AI mounted on the terminal device 10 (a program of the automatic response AI installed on the terminal device 10) to the automatic response AI mounted on the multi-function peripheral B.

  In this case, the control unit 26 transmits information indicating the content of the user's utterance to the MFP B via the communication path N. The automatic response AI installed in the MFP B receives the information indicating the content of the user's speech transmitted from the terminal device 10, analyzes the content of the user's speech, and creates a reply such as an answer to the content of the speech. . The automatic response AI of the MFP B transmits, to the terminal device 10, information indicating the content of the message including the reply. For example, the statement contents 190 and 194 shown in FIG. 36 are created by the automatic response AI installed in the multifunction machine B, and information indicating the statement contents 190 and 194 is transmitted from the multifunction machine B to the terminal device 10 Be done. The control unit 26 links the message content (for example, the message content 190 and 194) sent from the multifunction device B to the device image 188 of the multifunction device B as the message of the multifunction device B and displays the message content in the display area 48a. .

  When the processing by the multifunction machine B is completed, the control unit 26 controls the image tied to the interaction partner from the device image 188 tied to the multifunction machine B to the general-purpose automatic response AI 50, as shown in FIG. Further, the automatic response AI as the user's interactive partner is switched from the automatic response AI mounted on the multifunction device B to the general-purpose automatic response AI mounted on the terminal device 10.

  According to the modification 10, when the automatic response AI is mounted on the device, since the dialog is performed between the automatic response AI and the user, the processing of the automatic response AI mounted on the terminal device 10 is performed during that time Load is reduced. In addition, when the automatic response AI installed in the device has a higher degree of control ability and information collection capability with respect to the device than the general-purpose automatic response AI, the automatic response AI installed in the device is By interacting with the user as the user's interaction partner, more useful information can be provided to the user.

(Modification 11)
Hereinafter, the modification 11 will be described with reference to FIG. In the eleventh modification, when the use instruction (use request) of the part of the device is given to the automatic response AI (for example, when the information of the part of the device is sent to the automatic response AI), display of the other party in the response Is switched to the display of the part. Hereinafter, modification 10 will be described in detail.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  Similar to the example shown in FIG. 36, the statement content 198 such as "How was it done?" Is displayed in the display area 48a as the statement content of the automatic response AI, and the statement indicating the use instruction of the device with respect to the statement content 198 As the content, a statement content 200 such as "I want to use the multifunction device B" is displayed on the display area 48b as the statement content of the user. Since the message content 200 including the information of the multifunction machine B is sent to the automatic response AI, as in the fifth embodiment described above, the image associated with the other party of conversation is an image 50 associated with the general-purpose automatic response AI. The image is switched to the device image 188 linked to the multifunction device B. In addition, an utterance content 202 such as "Which function do you want to use?" Is displayed in the display area 48a as the utterance content of the MFP B, and an utterance content 204 such as "I want to print file X" is the utterance content of the user. It is displayed in the display area 48b.

  The identifying unit 28 identifies the processing content requested by the user by analyzing the user's comment content 204, and causes the MFP B to execute the processing content. As in the fifth embodiment, the control unit 26 transmits a print job including data of the file X to the multifunction peripheral B, and causes the multifunction peripheral B to execute the print job.

  Further, the control unit 26 switches the display of the conversation partner to the display of the part of the device for executing the process. Since the print function is assigned to the “main unit” of the multifunction machine B, the control unit 26 switches the display of the conversation partner to the display of the “main unit” of the multifunction machine B. For example, instead of the device image 188 tied to the multifunction device B, the control unit 26 causes the partial image 206 tied to the “main body” of the multifunction device B to be displayed in the display area 48a as an image tied to the conversation partner. It should be noted that, when the message content 204 indicating the processing content is sent to the automatic response AI, a usage instruction (use request) of the part of the device for executing the processing is given to the automatic response AI. That is, the information on the part of the device is sent to the automatic response AI. In the above example, since the content of processing “I want to print” is included in the message content 204, the information on the “main unit” of the MFP B to which the print function is assigned is sent to the automatic response AI. It will be

  Further, instead of the image 50 linked to the automatic response AI, the control unit 26 may display a partial image representing the state of the part of the device in the display area 48a as an image linked to the dialogue partner. In the above-described example, the control unit 26 causes the partial image 206 representing the state of the “main unit” of the multifunction peripheral B to be displayed in the display area 48 a as an image associated with the other party. The control unit 26 acquires information indicating the state of each part of the device from the device to be used, and controls display of a partial image indicating the state. For example, when a part of the device is executing a process, the control unit 26 displays a partial image representing that the process is being executed as an image tied to the other party, and the part of the device performs the process. If it is not executed, a partial image representing that the process is not executed is displayed as an image linked to the other party of interaction, and if the state of the device part corresponds to the sleep state, the partial image representing the sleep state is used as the other party for interaction When the power of the device is turned off, a partial image representing that the power is turned off is displayed as an image tied to the other party. By doing this, the user can visually confirm the state of the part of the device. For example, when the “main body” of the multifunction machine B is printing, the control unit 26 is an image in which the partial image 206 representing a state in which the “main body” of the multifunction machine B is printing Are displayed in the display area 48a. The image representing the state of the part of the device may be created in advance and stored in the storage unit 24 or may be created by the control unit 26. The control unit 26 may display a character string indicating the state of the part of the device in the display area 48a instead of or together with the partial image.

  When the printing of the file X is completed, the control unit 26 causes the display area 48a to display the message content 208 indicating that the printing is completed, as the message content of the “main unit” of the multifunction device B. Also at this time, the control unit 26 causes the display unit 48 to display the partial image 206 tied to the "main unit" of the multifunction peripheral B in the display area 48a, and the message contents 208 to the partial image 206 of the "main unit" of the multifunction peripheral B. It is displayed in the display area 48a as the comment content to which the string is attached.

  When the processing by the multifunction machine B is completed, the control unit 26 replaces the partial image 206 tied to the “main part” of the multifunction machine B, and ties the image 50 tied to the general-purpose automatic response AI to the conversation partner Are displayed in the display area 48a. In addition, the control unit 26 causes the display area 48a to display the utterance content 210 indicating a general question such as "How did it happen?" As the utterance content of the automatic response AI.

  As another example, when the information of the part of the device (for example, a partial image associated with the part or a character string indicating the part) is sent to the automatic response AI as the user's utterance content 200, the control unit 26 The display of may be switched from the display of the automatic response AI to the display of that portion. For example, when the contents of a statement such as “I want to use the main unit (or the printing unit) of the MFP B” as the contents 200 of the user are sent to the automatic response AI, the control unit 26 Alternatively, the image 50 associated with the automatic response AI may be switched to the partial image 206 associated with the “main body” of the multifunction machine B.

  As described above, according to the modification 11, when the use instruction (use request) of the device part is given to the automatic response AI on the screen 48 which interacts with the automatic response AI (for example, the information of the device part is automatically When sent to the response AI), the display of the conversation partner is switched to the display of the part of the device. By doing this, the user can visually confirm whether or not an instruction can be given to the part.

  When the part of the device is displayed as the interaction partner, the user's interaction partner may be the general-purpose automatic response AI mounted on the terminal device 10, or from the general-purpose automatic response AI It may be switched to the automatic response AI mounted on the part.

(Modification 12)
Hereinafter, the modified example 12 will be described with reference to FIG. In the modified example 12, when the user gives an instruction to use a plurality of devices (use request) to the automatic response AI (for example, when information of a plurality of devices is sent to the automatic response AI as information to be cooperated) Is switched from the display of the general-purpose automatic response AI to the display of a plurality of devices. The modification 12 will be described in detail below.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  Similar to the example shown in FIG. 36, the statement content 212 such as "How did it happen?" Is displayed in the display area 48a as the statement content of the automatic response AI.

  When the user designates a plurality of devices as devices to be cooperated with respect to the content of this statement 212, the specification unit 28 (the specification unit 28 realized by the automatic response AI) identifies the plurality of devices as devices to be cooperation. Do. For example, it is assumed that the user issues an utterance content 214 such as “I want to use the MFP B and the PC (A)” as the utterance content indicating the usage instruction (utilization request) of a plurality of devices. The control unit 26 causes the display area 48b to display the message content 214 including the information of the MFP B and the PC (A) as the user's message content, and sends the message content 214 to the automatic response AI. As a result, the information on the MFP B and the PC (A) is sent to the automatic response AI as the information on the cooperation target device. The user may input a character string indicating a plurality of devices to the terminal device 10 as the user's utterance content by operating the UI unit 20, and may be linked to the devices as in the first embodiment described above. A device image (for example, an image such as an appearance image or an icon) may be designated as the user's utterance content. When specifying a device image, as shown in FIG. 8, the user selects a plurality of device images from the stock area 58, and moves the plurality of device images to the display position of the user's statement content. The plurality of device images may be designated as the user's utterance content.

  The identifying unit 28 identifies an executable collaboration function using a plurality of devices designated by the user, for example, by referring to the collaboration function management table shown in FIG. 7. In the above example, since the MFP B and the PC (A) are specified, the “scan transfer function” and the “print function” can be executed as cooperative functions that can be executed using the MFP B and the PC (A). Identified (see FIG. 7).

  When the user designates a plurality of devices and the plurality of devices are identified, the control unit 26 changes the display of the conversation partner from the display of the general-purpose automatic response AI to the display of the plurality of devices specified by the user. Switch.

  In the above example, since the multifunction device B and the PC (A) are designated as the devices to be cooperated, the control unit 26 substitutes the image 50 associated with the automatic response AI with the device image associated with the multifunction device B. A device image 216 linked to the H.188 and PC (A) is displayed in the display area 48a as an image linked to the other party. The device image 216 may be an image generated by photographing the PC (A) (for example, an appearance image of the PC (A)) or an image (for example, an icon) imitating the PC (A) It is also good. Of course, the control unit 26 displays a character string indicating the MFP B and the PC (A) in the display area 48a as information indicating the other party of the conversation instead of the device image 188 or 216 or together with the device image 188 or 216. May be

  In addition, the control unit 26 causes the display area 48a to display the statement contents 218 such as "which function do you want to use?" As the statement contents of the MFP B and the PC (A). The utterance content 218 itself is the utterance content created by the general-purpose automatic response AI. The control unit 26 directs the statement contents 218 of the device image 188 of the multifunction machine B to that of the PC (A) in order to produce effects as if the multifunction machine B and the PC (A) are speaking the statement contents 218. It is displayed in the display area 48 a as the content of the message associated with the device image 216. The control unit 26 may display a list of the linked functions specified by the specifying unit 28 in the display area 48a as the content of the statement of the MFP B and the PC (A). The cooperation functions that can be executed using the MFP B and the PC (A) are the “scan transfer function” and the “print function” (see FIG. 7). A character string or an image indicating “print function” may be displayed in the display area 48 a as the content of the statement of the multifunction device B and the PC (A).

  With respect to the above-mentioned statement contents 218, the user inputs the cooperation function that he / she wants to execute to the terminal device 10 as the statement contents of the user. In the example shown in FIG. 38, an utterance content 220 indicating a scan transfer function such as “I want to send scanned data to the PC (A)” is input to the terminal device 10 as the utterance content of the user.

  The control unit 26 analyzes the user's comment contents 220 to specify the cooperation function requested by the user, and causes the MFP B and the PC (A) to execute the cooperation function. The process for the execution is the same as the process according to the first embodiment described above.

  When the scan transfer is completed, the control unit 26 causes the display area 48a to display the message content 222 indicating that the scan transfer is completed as the message content of the MFP B and the PC (A). Also at this time, the control unit 26 displays the device image 188 tied to the multifunction machine B and the device image 216 tied to the PC (A) in the display area 48 a, and the message content 222 of the message image of the multifunction machine B It is displayed in the display area 48 a as the content of an utterance linked to the device image 216 of the PC (A) 188 and the PC (A). The statement content 222 itself is the statement content created by the general-purpose automatic response AI.

  When the processing by the multifunction machine B and the PC (A) is completed, the control unit 26 replaces the device image 188 tied to the multifunction machine B and the device image 216 tied to the PC (A) with a general-purpose automatic response AI. The image 50 linked to is displayed in the display area 48a as an image linked to the conversation partner. In addition, the control unit 26 causes the display area 48a to display the utterance content 224 indicating a general question such as "How did it happen?" As the utterance content of the automatic response AI.

  As described above, according to the modification 12, when the use instruction (use request) of a plurality of devices to be cooperated is given to the automatic response AI on the screen 48 which interacts with the automatic response AI (for example, a plurality of cooperation objects) When the information of the device of (1) is sent to the automatic response AI), the display of the conversation partner is switched to the display of the plurality of devices. By doing this, the user can visually confirm whether or not an instruction can be given to the plurality of devices. For example, when the user sends information of a plurality of devices to the automatic response AI in order to give instructions to a plurality of devices, an image associated with each device is displayed as an image associated with the interaction partner. By doing this, the user can visually confirm whether or not the instruction is actually given to a plurality of devices.

  When a plurality of devices are displayed as a conversation partner, the user's dialog partner may be a general-purpose automatic response AI installed in the terminal device 10, or the plurality of general-purpose automatic responses AI may be used. It may be switched to the automatic response AI mounted on the device. For example, when the automatic response AI is mounted on the multifunction device B, the automatic response AI mounted on the multifunction device B may function as a conversation partner, or the automatic response AI is mounted on the PC (A) If so, the automatic response AI mounted on the PC (A) may function as the other party of the dialogue. When the automatic response AI is mounted on both the multifunction device B and the PC (A), the automatic response AI mounted on either of the multifunction device B or the PC (A) may function as a dialogue partner.

  In the example shown in FIG. 38, although the information of two devices is sent to the automatic response AI, the information of three or more devices (for example, a character string indicating three or more device images or three or more devices) May be sent to the automatic response AI. In this case, in place of the image 50 associated with the automatic response AI, three or more device images are displayed in the display area 48a as an image associated with the interaction partner.

  Further, the modifications 11 and 12 may be combined. That is, when a plurality of parts are designated as a part to be cooperated and the information of the plurality of parts is sent to the automatic response AI, the control unit 26 switches the display of the conversation partner to the display of the plurality of parts. Good. A plurality of parts in the same device may be designated, or a plurality of parts may be designated from among the plurality of devices.

(Modification 13)
Hereinafter, the modification 13 will be described with reference to FIGS. 39 and 40. FIGS. 39 and 40 show images linked to the user with whom the user interacts. Here, it is assumed that information of a plurality of devices to be cooperated (for example, the MFP B and the PC (A)) is sent to the automatic response AI as the user's interactive content.

  An image of a basic form is shown in FIG. 39 (a). As in the above-described modified example 12, when the information of the multifunction peripheral B and the PC (A) is sent to the automatic response AI as the user's utterance content, the device image 188 tied to the multifunction peripheral B and the image on the PC (A) are The attached device image 216 is displayed in the display area 48a as an image associated with the other party.

  As shown in FIG. 39 (a), in the state where the device image 188, 216 is displayed as an image tied to the other party of conversation, as shown in FIG. When it is displayed in the display area 48a, the control unit 26 causes the device image 188 tied to the multifunction device B to be displayed in the display area 48a larger than the device image 216 tied to the PC (A). For example, when the PC (A) is not executing the process included in the cooperation function and the MFP B is executing the process included in the cooperation function, the information regarding the process is the statement content of the MFP B It is displayed in the display area 48a. In this case, the device image 188 is displayed larger than the device image 216. By so doing, the user can visually confirm the device that is performing the process included in the cooperation function. When both the MFP B and the PC (A) execute the process included in the cooperation function, the device images 188 and 216 have the same size, as shown in FIG. 39 (a). Is displayed.

  On the other hand, as shown in FIG. 39C, when the message content 228 is displayed in the display area 48a as the message content of the PC (A), the control unit 26 makes the device image 216 larger than the device image 188. Display in the display area 48a. For example, when the PC (A) is executing the process included in the cooperation function, and the MFP B is not executing the process included in the cooperation function, information on the process is displayed as the statement content of the MFP B It is displayed in the area 48a.

  Further, in the example shown in FIG. 38 and FIG. 39 (a), the device images 188, 216 are arranged and displayed so as to be aligned in the horizontal direction, but as shown in FIG. 39 (d), the device image 188, 216 may be arranged and displayed so as to line up vertically. Of course, the device images 188 and 216 may be displayed in an arrangement other than these.

  Further, as shown in FIG. 39 (e), when the message contents 230 are displayed in the display area 48a as the message contents of the MFP B, the device image 188 linked to the MFP B is linked to the PC (A). It may be displayed larger than the device image 216. When the statement content is displayed in the display area 48 a as the statement content of the PC (A), the device image 216 is displayed larger than the device image 188.

  Also, the display order of the device images may be changed according to the order of the user's speech. For example, when the user speaks in the order of “MFP B” and “PC (A)”, the device image 188 in association with MFP B and the device image 216 in association with PC (A) are in the order of device images 188, 216 may be displayed. For example, the previously designated device (for example, the multifunction peripheral B) may be displayed on the left or upper side, and the later designated device (for example, PC (A)) may be displayed on the right or lower side. Of course, the display order may be determined by the order of identification of devices, the order of registration, etc., regardless of the order of specification. Also, the device image of the device that is uttering (the device to which the content of the remark is linked) may be displayed at a predetermined position (for example, the left side or the upper side). By so doing, the user can visually confirm the device that is speaking (for example, the device that is performing processing).

  FIG. 40 shows still another display example. For example, when the modification examples 11 and 12 are combined and information on a plurality of parts to be cooperated is sent to the automatic response AI as the utterance content of the user, the control unit 26 displays Switch to display. FIG. 40 (a) shows an example of the display. For example, when the “main unit” of the multifunction device B and the “display unit” of the PC (A) are designated as the cooperation target portion, and the information thereof is sent to the automatic response AI as the user's utterance content, the multifunction device B A partial image 206 tied to the "body" and a partial image 232 tied to the "display" of the PC (A) are displayed in the display area 48a as an image tied to the other party. When a plurality of portions are displayed as a conversation partner, the display example shown in any of FIGS. 39 (a) to 39 (e) may be applied.

  In addition, when the device is executing the process, the control unit 26 may display a partial image tied to the portion of the device executing the process in the display area 48a as an image tied to the other party . For example, when the “main body” of the multifunction machine B is executing printing, as shown in FIG. 40 (b), the control unit 26 sets the partial image 206 tied to the “main body” to the other party of the dialogue. It is displayed in the display area 48a as an image to be linked. Similarly, when the “reading unit” of the MFP B is executing a scan, the control unit 26 displays the partial image linked to the “reading unit” in the display area 48 a as an image linked to the other party Let In this way, the user can visually confirm the portion in which the process is being performed. In addition, the control unit 26 ties the partial image tied to the portion (for example, the “main unit” of the multifunction device B) executing the process (for example, print job etc.) instructed by the user on the screen 48 It may be displayed in the display area 48a as an attached image. By so doing, the user can visually confirm the part executing the process instructed by the user himself.

  Further, as shown in FIG. 40C, an image 50a representing the automatic response AI itself and a background image 50b representing a background other than the automatic response AI are displayed in the image 50 associated with the general-purpose automatic response AI. It is also good. When the information of the device (e.g., the multifunction device B) is sent to the automatic response AI, the entire image 50 may be replaced with the device image 188 as shown in FIG. 40 (d). As another example, as shown in FIG. 40 (e), the image 50 may be replaced by an apparatus image 188 representing the composite machine B itself and the background image 50b. That is, the background image 50b may be displayed as it is.

Sixth Embodiment
The device system according to the sixth embodiment will be described below. In the sixth embodiment, when an instruction to use a function (usage request) is sent to the automatic response AI on an interface (for example, the screen 48) in which the user interacts with the automatic response AI (for example, automatic response of information on the function) If sent to AI), the display of the other party in the response is switched to the display of the function. The sixth embodiment will be described in detail below.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 234 such as “How did it happen?” Is displayed on the screen 48 as the utterance content of the automatic response AI. Since the message content 234 is the message content of the automatic response AI, the control unit 26 displays the image 50 linked to the automatic response AI in the display area 48 a and links the message content 234 to the image 50 of the automatic response AI The message content is displayed in the display area 48a. Note that this automatic response AI is a device-independent automatic response AI and can be said to be a general-purpose automatic response AI.

  When the user designates a function with respect to the utterance content 234, the identification unit 28 (the identification unit 28 realized by the automatic response AI) identifies the function. For example, it is assumed that the user issues an utterance content 236 such as “I want to use a password function.” As the utterance content indicating a usage instruction (a usage request) of the function. The control unit 26 causes the display area 48b to display the utterance content 236 including the information of the password function as the utterance content of the user, and sends the utterance content 236 to the automatic response AI. Thereby, the information of the password function is sent to the automatic response AI. The statement content 236 is displayed in the display area 48b as the statement content linked to the image 52 of the user. The user may input a character string indicating a function to the terminal device 10 as the user's utterance content by operating the UI unit 20, or the functional image associated with the function as in the second embodiment. (For example, an image such as an icon) may be designated as the user's speech content. When specifying a functional image, as shown in FIG. 27, the user selects a functional image linked to the function to be used from the stock area 124, and moves the functional image to the display position of the user's statement content By doing this, the functional image may be designated as the user's speech content.

  When the user specifies a function and the function is identified, the control unit 26 displays a dialog partner (for example, an image associated with the dialog partner or a character string indicating the dialog partner), and displays a general-purpose automatic response AI. Display of a function designated by the user from the image (for example, an image associated with the general-purpose automatic response AI or a character string indicating the general-purpose automatic response AI) (for example, an image associated with the function or a character string indicating the function) Switch to

  Even when the display of the conversation partner is switched from the display of the general-purpose automatic response AI to the display of the function, the user's dialog partner itself is the general-purpose automatic response AI. That is, the contents of the conversation partner's utterance are created by the general-purpose automatic response AI mounted on the terminal device 10.

  In the above example, since the password function is specified as the function to be used, the control unit 26 links the function image 238 linked to the password function to the conversation partner instead of the image 50 linked to the automatic response AI. It is displayed in the display area 48a as an image. The functional image 238 is, for example, an icon. Of course, instead of the functional image 238 or together with the functional image 238, the control unit 26 may display a character string indicating the password function in the display area 48a as information indicating the other party of the dialogue.

  Also, the control unit 26 interacts with the functional image representing the state of the function (for example, the state of the software realizing the function and the state of the device realizing the function) instead of the image 50 associated with the automatic response AI. It may be displayed in the display area 48a as an image tied to the other party. In the above example, the control unit 26 causes the functional area 238 indicating the state of the password setting software to be displayed in the display area 48a as an image associated with the other party. The control unit 26 acquires information indicating the state of the software or the device from the software or device realizing the function, and controls the display of the functional image indicating the state. For example, when the software or device for realizing the function is executing processing, the control unit 26 displays a functional image representing the state of executing the processing as an image linked to the other party of interaction, and If the software or device that implements the function is not executing the process, a functional image representing that the process is not executed is displayed as an image linked to the other party of interaction, and the state of the device that implements the function is in the sleep state If applicable, the functional image representing the sleep state is displayed as an image associated with the interaction partner, and if the state of the software that realizes the function corresponds to the freeze state, the image relating the functional image representing the freeze state to the interaction partner When the power of the device that realizes the function is turned off, a functional image representing that the power is turned off is displayed as an image linked to the other party in the dialogueIn this way, the user can visually confirm the state of the function. For example, when the password setting software sets a password for another data, the control unit 26 sets a function image 238 representing that the password setting software is performing password setting processing as an image associated with the other party of interaction. It is displayed in the display area 48a. The image representing the state of the device may be created in advance and stored in the storage unit 24 or may be created by the control unit 26. The control unit 26 may display a character string indicating the state of the function in the display area 48a instead of or together with the device image.

  In addition, the control unit 26 causes the display area 48a to display the comment contents 240 such as "which file do you want to use the password for?" As the comment contents of the password function. The statement content 240 itself is the statement content created by the general-purpose automatic response AI. The control unit 26 causes the display content 48 to be displayed in the display area 48 a as the content of the message linked to the functional image 238 of the password function, in order to produce the effect as if the password function is making a statement of the content 240.

  The user specifies a file to which a password is to be set for the above statement content 240. In the example shown in FIG. 41, the utterance content 242 indicating the content such as “file X” is input to the terminal device 10 as the utterance content of the user.

  The control unit 26 analyzes the user's comment content 242 to specify the processing content requested by the user, and causes the password setting software to execute the processing content. For example, the control unit 26 activates the password setting software, and sets the password in the file X designated by the user by the password setting software. The control unit 26 may cause the UI unit 20 to display a screen for setting a password.

  When the setting of the password is completed, the control unit 26 causes the display area 48a to display the statement content 244 indicating that the setting of the password is completed as the statement content of the password function. Also at this time, the control unit 26 displays the functional image 238 linked to the password function in the display area 48a, and displays the message content 244 in the display area 48a as the message content linked to the functional image 238 of the password function. . The statement content 244 itself is the statement content created by the general-purpose automatic response AI.

  When the process by the password function is completed, the control unit 26 displays the image 50 linked to the general-purpose automatic response AI in the display area 48a as an image linked to the conversation partner instead of the function image 238 linked to the password function. Let In addition, the control unit 26 causes the display area 48a to display an utterance content 246 indicating a general question such as "How did it happen?" As the utterance content of the automatic response AI.

  As described above, according to the sixth embodiment, when an instruction to use a function (use request) is given to the automatic response AI on the screen 48 that interacts with the automatic response AI (for example, information on the function is transmitted to the automatic response AI) If sent, the display of the other party in the response is switched to the display of the function. In this way, the user can visually confirm whether or not an instruction can be given to the function. For example, when the user sends information of the function to the automatic response AI to give an instruction to the function, an image linked to the function is displayed as an image linked to the interaction partner. In this way, the user can visually confirm whether the instruction is actually given to the function.

  Modification 10 may be applied to the sixth embodiment. For example, when the program of the automatic response AI is assigned to each function and the program of each automatic response AI is installed in the terminal device 10, the control unit 26 displays the other party of the dialogue by displaying the general-purpose automatic response AI. The automatic response AI as the user's interactive partner may be switched from the general-purpose automatic response AI to the automatic response AI assigned to the function, as well as the display of the function to the display of the function.

  Modification 11 may be applied to the sixth embodiment. For example, in the case where a function is further assigned to each part in the functional image, if the image linked to the part is sent to the automatic response AI as the user's utterance content, the image linked to the part interacts with the dialog It may be displayed in the display area 48a as an image tied to the other party.

  Modification 12 may be applied to the sixth embodiment. For example, when the user gives an instruction to use a plurality of functions (usage request) to the automatic response AI (for example, when information on a plurality of functions is sent to the automatic response AI as information to be cooperated) The display of is switched from the display of the general purpose automatic response AI to the display of a plurality of functions. The process in the case where the modification 12 is applied to the sixth embodiment will be described below with reference to FIG.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  As in the example shown in FIG. 41, the statement content 248 such as "How did it happen?" Is displayed in the display area 48a as the statement content of the automatic response AI.

  When the user designates a plurality of functions as the cooperation target function with respect to the comment content 248, the specification unit 28 (the specification unit 28 realized by the automatic response AI) identifies the plurality of functions as the cooperation target function. Do. For example, it is assumed that the user issues an utterance content 250 such as “I want to use a password function and a data transmission function” as the utterance content indicating a usage instruction (a usage request) of a plurality of functions. The control unit 26 causes the display area 48b to display the statement content 250 including the information of the password function and the data transmission function as the statement content of the user, and sends the statement content 250 to the automatic response AI as the information to be cooperated. Thereby, the information on the password function and the data transmission function is sent to the automatic response AI as the information on the function to be cooperated. The user may input a character string indicating a plurality of functions to the terminal device 10 as the content of the user's utterance by operating the UI unit 20, as in the second embodiment described above. A functional image (for example, an icon) may be designated as the user's utterance content. When specifying a functional image, as shown in FIG. 27, the user selects a plurality of functional images from the stock area 124, and moves the plurality of functional images to the display position of the user's statement content. The plurality of functional images may be designated as the user's utterance content.

  The identifying unit 28 identifies an executable collaboration function by using a plurality of functions designated by the user, for example, by referring to the collaboration function management table shown in FIG. In the above example, since the password function and the data transmission function are specified, for example, “a function of setting a password for transmission and transmitting” or the like is specified as the cooperation function.

  When the user designates a plurality of functions and the plurality of functions are identified, the control unit 26 changes the display of the conversation partner from the display of the general-purpose automatic response AI to the display of the plurality of functions specified by the user. Switch.

  In the above example, since the password function and the data transmission function are designated as the function to be cooperated, the control unit 26 substitutes the image 50 associated with the automatic response AI with the functional image 238 associated with the password function and data A function image 252 associated with the transmission function is displayed in the display area 48 a as an image associated with the other party in dialogue. Of course, the control unit 26 may display a character string indicating the password function and the data transmission function in the display area 48a as information indicating the conversation partner instead of or together with the function images 238 and 252. Good.

  In addition, the control unit 26 causes the display area 48a to display the statement content 254 such as "which file do you want to process?" As the statement content of the password function and the data transmission function. The statement content 254 itself is the statement content created by the general-purpose automatic response AI. In order to produce the effect of the password function and the data transmission function as if the statement content 254 is being described, the control unit 26 combines the statement content 254 with the function image 238 of the password function and the function image 252 of the data transmission function. Is displayed in the display area 48a as the content of the message linked to the message. The control unit 26 may display a list of the linked functions specified by the specifying unit 28 in the display area 48a as the content of the message of the password function and the data transmission function.

  The user specifies a file to be processed with respect to the above statement content 254. In the example shown in FIG. 42, the statement content 256 indicating the content such as “file X” is input to the terminal device 10 as the statement content of the user.

  The control unit 26 analyzes the user's comment content 256 to specify the processing content requested by the user, and causes the password setting software and the data transmission software to execute the processing content. For example, the control unit 26 activates password setting software and data transmission software, sets the password to the file X designated by the user by the password setting software, and sets the file X to which the password is set by the data transmission software. It transmits to a transmission destination (a transmission destination designated by the user, a predetermined transmission destination, etc.). The control unit 26 may cause the UI unit 20 to display a screen for setting a password and a screen for setting a transmission destination.

  When the execution of the cooperation function is completed, the control unit 26 causes the display area 48a to display the statement content 258 indicating that the execution of the cooperation function is completed as the statement content of the password function and the data transmission function. Also at this time, the control unit 26 displays the functional image 238 associated with the password function and the functional image 252 associated with the data transmission function in the display area 48a, and transmits the message content 258 and the functional image 238 with the password function. It is displayed in the display area 48 a as a statement content linked to the function image 252 of the function. The statement content 258 itself is the statement content created by the general-purpose automatic response AI.

  When the execution of the cooperation function is completed, the control unit 26 replaces the image 50 associated with the general-purpose automatic response AI with the other party instead of the function image 238 associated with the password function and the function image 252 associated with the data transmission function. It is displayed in the display area 48a as an image to be linked. In addition, the control unit 26 causes the display area 48a to display the utterance content 260 indicating a general question such as "How did it happen?" As the utterance content of the automatic response AI.

  As described above, when the modification example 12 is applied to the sixth embodiment, an instruction to use a plurality of functions to be cooperated (use request) is given to the automatic response AI on the screen 48 that interacts with the automatic response AI. (For example, when the information of a plurality of functions to be cooperated is sent to the automatic response AI as the dialogue content), the display of the dialogue partner is switched to the display of the plurality of functions. By doing this, the user can visually confirm whether or not an instruction can be given to the plurality of functions. For example, when the user sends information of a plurality of functions to the automatic response AI in order to give instructions to a plurality of functions, an image associated with each function is displayed as an image associated with the interaction partner. By doing this, the user can visually confirm whether the instruction is actually given to a plurality of functions.

  When a plurality of functions are displayed as the interaction partner, the user's interaction partner may be a general-purpose automatic response AI, or even if the general-purpose automatic response AI is switched to the automatic response AI for each function. Good.

  In the example shown in FIG. 42, although information of two functions is sent to the automatic response AI, information of three or more functions (for example, a character string indicating three or more function images or three or more functions) May be sent to the automatic response AI. In this case, in place of the image 50 linked to the automatic response AI, three or more functional images are displayed in the display area 48a as an image linked to the interactive partner.

  Also, the combination of the modifications 11 and 12 may be applied to the sixth embodiment. That is, in the case where the function is further assigned to each part in the functional image, when the image linked to the plurality of parts is sent to the automatic response AI as the user's utterance content, the images of the plurality of parts interact It may be displayed in the display area 48a as an image tied to the other party.

  Further, the modification 13 may be applied to the sixth embodiment. That is, also in the sixth embodiment, each device image may be displayed according to the display example shown in FIG. 39 and FIG.

Seventh Embodiment
The device system according to the seventh embodiment will be described below. In the seventh embodiment, when an instruction to use devices and functions (usage request) is given to the automatic responses AI on an interface (for example, the screen 48) in which the user interacts with the automatic responses AI (for example, the devices and functions When information is sent to the automatic response AI), the display of the other party in the response is switched to the display of the device and function. The seventh embodiment will be described in detail below.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 262 such as “How did it happen?” Is displayed on the screen 48 as the utterance content of the automatic response AI. Since the message content 262 is the message content of the automatic response AI, the control unit 26 displays the image 50 linked to the automatic response AI in the display area 48a and links the message content 262 to the image 50 of the automatic response AI The message content is displayed in the display area 48a. Note that this automatic response AI is a device-independent automatic response AI and can be said to be a general-purpose automatic response AI.

  When the user designates a device and a function as a collaboration target with respect to the statement content 262, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the device and the function as a collaboration target. For example, it is assumed that the user issues an utterance content 264 such as “I want to use the multifunction device B and the password function” as the utterance content indicating the usage instruction (usage request) of the device and the function. The control unit 26 causes the display area 48b to display the message content 264 including information on the MFP B and the password function as the user's message content, and sends the message content 264 to the automatic response AI. As a result, the information on the multifunction peripheral B and the password function is sent to the automatic response AI as the information to be cooperated. The statement content 264 is displayed in the display area 48b as the statement content linked to the image 52 of the user. The user may input a character string indicating the device and the function to the terminal device 10 as the user's statement content by operating the UI unit 20, and may be tied to the device as in the third embodiment described above. The functional image associated with the device image and the function may be designated as the user's utterance content. When specifying a device image, as shown in FIG. 8, the user selects a device image linked to the device to be linked from the stock area 58, and moves the device image to the display position of the user's statement content By doing this, the device image may be designated as the user's utterance content. Similarly, when specifying a functional image, as shown in FIG. 27, the user selects a functional image associated with the function to be used from the stock area 124, and displays the functional content of the user's statement content By moving to the position, the functional image may be designated as the user's speech content.

  The identifying unit 28 identifies an executable cooperative function by using the device and function designated by the user, for example, by referring to the cooperative function management table shown in FIG. In the above example, since the multifunction device B and the password function are specified, for example, “a function of applying a password to data generated by scanning by the multifunction device B” or the like is specified as the cooperation function.

  When the user designates a device and a function as collaboration targets, and the device and the function are identified as collaboration targets, the control unit 26 designates the display of the conversation partner from the display of the general-purpose automatic response AI by the user. Switch to the device and function display.

  Even when the display of the conversation partner is switched from the display of the general-purpose automatic response AI to the display of the device and the function, the user's dialog partner itself is the general-purpose automatic response AI. That is, the contents of the conversation partner's utterance are created by the general-purpose automatic response AI mounted on the terminal device 10.

  In the above example, since the multifunction device B and the password function are designated as the cooperation target, the control unit 26 substitutes the image 50 attached to the automatic response AI for the device image 188 attached to the multifunction device B and the password function. Is displayed in the display area 48a as an image associated with the other party. Of course, instead of the device image 188 and the functional image 238, or together with the device image 188 and the functional image 238, the control unit 26 displays the character string indicating the multifunction device B and the password function as information indicating the conversation partner. It may be displayed inside.

  In addition, the control unit 26 causes the display area 48a to display the statement content 266 such as "which function do you want to use?" As the statement content of the MFP B and the password function. The utterance content 266 itself is the utterance content created by the general-purpose automatic response AI. In order to produce effects as if the multifunction device B and the password function are speaking the content 266, the control unit 26 combines the content 266 with the device image 188 of the multifunction device B and the functional image 238 of the password function. Is displayed in the display area 48a as the content of the message linked to the message. The control unit 26 may display the list of the linked functions specified by the specifying unit 28 in the display area 48a as the content of the statement of the MFP B and the password function.

  With respect to the above-mentioned statement content 266, the user inputs the cooperation function that he / she wants to execute into the terminal device 10 as the statement content of the user. In the example illustrated in FIG. 43, a statement content 268 indicating a cooperation function such as “I want to put a password on scanned data” is input to the terminal device 10 as the statement content of the user.

  The control unit 26 analyzes the user's comment contents 268 to specify the cooperation function requested by the user, and causes the MFP B and the password function (password setting software) to execute the cooperation function. For example, the control unit 26 transmits a scan job to the MFP B, and causes the MFP B to execute a scan. The data generated by this is transmitted from the multifunction device B to the terminal device 10. Furthermore, the control unit 26 activates password setting software, and sets the password in the data by the password setting software.

  When the execution of the cooperation function is completed, the control unit 26 causes the display area 48a to display the statement contents 270 indicating that the execution of the cooperation function is completed as the statement contents of the MFP B and the password function. Also at this time, the control unit 26 displays the device image 188 linked to the multifunction machine B and the functional image 238 linked to the password function in the display area 48 a, and the message content 270 with the device image 188 of the multifunction machine B It is displayed in the display area 48 a as the content of the message linked to the function image 238 of the password function. The statement content 270 itself is the statement content created by the general-purpose automatic response AI.

  When the execution of the cooperation function is completed, the control unit 26 substitutes the image 50 associated with the general-purpose automatic response AI as the dialogue partner, instead of the device image 188 associated with the multifunction machine B and the function image 238 associated with the password function. It is displayed in the display area 48a as an image to be linked. In addition, the control unit 26 causes the display area 48a to display an utterance content 272 indicating a general question such as "How is it done?" As the utterance content of the automatic response AI.

  As described above, according to the seventh embodiment, when an instruction to use devices and functions (use request) is given to the automatic responses AI on the screen 48 that interacts with the automatic responses AI (for example, information on the devices and functions is provided) When sent to the automatic response AI), the display of the conversation partner is switched to the display of the device and function. By doing this, the user can visually confirm whether or not an instruction can be given to the device and the function. For example, when the user sends device and function information to the automatic response AI in order to give instructions to the device and function, an image linked to the device and function is displayed as an image linked to the other party in the dialogue. By doing this, the user can visually confirm whether or not the instruction is actually given to the device and the function.

  In the example shown in FIG. 43, information of two devices and functions in total is sent to the automatic response AI, but information of three or more devices and functions in total may be sent to the automatic response AI. In this case, in place of the image 50 associated with the automatic response AI, a total of three or more device images and device images are displayed in the display area 48a as an image associated with the interaction partner.

  Modification 10 may be applied to the seventh embodiment. For example, when the program of the automatic response AI is assigned to each device or function, the control unit 26 switches the display of the conversation partner from the display of the general-purpose automatic response AI to the display of the device and the function. The automatic response AI as the interaction partner may be switched from the general-purpose automatic response AI to the function or the automatic response AI assigned to the function.

  Modification 11 may be applied to the seventh embodiment. For example, when the information of the part of the device is sent to the automatic response AI as the user's utterance content, the image linked to the part may be displayed as the image linked to the other party of interaction. In addition, when a function is further assigned to each part in the functional image, if the image linked to the part is sent to the automatic response AI as the user's speech content, the image linked to the part interacts with the dialog It may be displayed in the display area 48a as an image tied to the other party.

  Moreover, the modification 13 may be applied to 7th Embodiment. That is, also in the seventh embodiment, the device image or the functional image may be displayed according to the display example shown in FIG. 39 and FIG.

Eighth Embodiment
An apparatus system according to the eighth embodiment will be described below. In the eighth embodiment, when an instruction to use a file (use request) is given to the automatic response AI on an interface (for example, the screen 48) in which the user interacts with the automatic response AI (for example, automatic response is made to file information). When sent to AI), the display of the other party in the response is switched to the display of the file. The eighth embodiment will be described in detail below.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  For example, an utterance content 274 such as “How did it happen?” Is displayed on the screen 48 as the utterance content of the automatic response AI. Since the message content 274 is the message content of the automatic response AI, the control unit 26 displays the image 50 linked to the automatic response AI in the display area 48 a and links the message content 274 to the image 50 of the automatic response AI The message content is displayed in the display area 48a. Note that this automatic response AI is a device-independent automatic response AI and can be said to be a general-purpose automatic response AI.

  When the user designates a file for the message content 274, the identifying unit 28 (the identifying unit 28 realized by the automatic response AI) identifies the file. For example, it is assumed that the user issues an utterance content 276 such as “I want to edit the document file A” as the utterance content indicating a file use instruction (usage request). The control unit 26 causes the display area 48b to display the message content 276 including the information of the document file A as the user's message content, and sends the message content 276 to the automatic response AI. Thereby, the information of the document file A is sent to the automatic response AI. The statement content 276 is displayed in the display area 48b as the statement content linked to the image 52 of the user. The user may input a character string (for example, a file name) indicating a file to the terminal device 10 as the content of the user's utterance by operating the UI unit 20, as in the fourth embodiment described above. A file image (for example, an image of an icon or the like) linked to may be designated as the user's speech content. When specifying a file image, as shown in FIG. 34, the user selects a file image linked to the file to be used from the stock area 170, and moves the file image to the display position of the user's statement contents By doing this, the file image may be designated as the user's message content.

  When the user designates a file and the file is identified, the control unit 26 switches the display of the conversation partner from the display of the general-purpose automatic response AI to the display of the file specified by the user.

  Even when the display of the dialog partner is switched from the display of the general-purpose automatic response AI to the display of the file, the user's dialog partner itself is the general-purpose automatic response AI. That is, the contents of the conversation partner's utterance are created by the general-purpose automatic response AI mounted on the terminal device 10.

  In the above example, since the document file A is designated as the file to be used, the control unit 26 replaces the image 50 associated with the automatic response AI with the file image 278 associated with the document file A as the interaction partner. It is displayed in the display area 48a as an image to be linked. The file image 278 is, for example, an icon. Of course, instead of the file image 278 or together with the file image 278, the control unit 26 may display a character string (for example, file name) indicating the document file A in the display area 48a as information indicating the other party.

  Further, the control unit 26 causes the display area 48a to display the statement content 280 such as "Do you want to start the document application?" As the statement content of the document file A. The statement content 280 itself is the statement content created by the general-purpose automatic response AI. The control unit 26 displays the utterance content 280 as the utterance content linked to the file image 278 of the document file A in the display area 48a in order to produce the effect that the document file A is uttering the utterance content 280. Let

  When the user instructs the start of the document application as in the case of the statement 282 in response to the statement 280, the control unit 26 starts the document application according to the instruction. At this time, the control unit 26 starts up the document application to open the document file A (displays the document file A in an editable manner) as the message content of the document file A in the display area 48a. Display. Also at this time, the control unit 26 causes the file image 278 linked to the document file A to be displayed in the display area 48a, and the message content 284 as the message content linked to the file image 278 of the document file A in the display area 48a. Display. The statement content 284 itself is the statement content created by the general-purpose automatic response AI.

  Thereafter, the user edits the document file A using the document application. When the editing is completed, the control unit 26 causes the image 50 associated with the general-purpose automatic response AI to be displayed in the display area 48a as an image associated with the conversation partner instead of the file image 278 associated with the document file A. .

  As described above, according to the eighth embodiment, when an instruction to use a file (use request) is given to the automatic response AI on the screen 48 that interacts with the automatic response AI (for example, information on the file is transmitted to the automatic response AI) (If sent), the display of the conversation partner is switched to the display of the file. In this way, the user can visually confirm whether or not the process can be executed on the file.

  In addition, a functional image associated with software (application) for processing a file may be displayed in the display area 48a as an image associated with the other party with the file image associated with the file. In the above example, the functional image associated with the document application may be displayed in the display area 48 a together with the file image 278 associated with the document file A.

  The tenth modification may be applied to the eighth embodiment. For example, when the program of the automatic response AI is assigned to each file and the program of each automatic response AI is installed in the terminal device 10, the control unit 26 displays the other party of the dialogue by displaying the general-purpose automatic response AI. The automatic response AI as the user's interactive partner may be switched from the general-purpose automatic response AI to the automatic response AI assigned to the file, as well as the display of the file to the display of the file.

  Modification 12 may be applied to the eighth embodiment. For example, when the user gives an instruction to use a plurality of files (request for use) to an automatic response AI (for example, when information on a plurality of files is sent to the automatic response AI as information to be cooperated) The display of is switched from the display of the general purpose automatic response AI to the display of a plurality of files. The process in the case where the modification 12 is applied to the eighth embodiment will be described below with reference to FIG.

  A screen 48 is shown in FIG. Similar to FIG. 8, the screen 48 displays an image 50 associated with the automatic response AI, an image 52 associated with the user, and information indicating interaction between the user and the automatic response AI.

  Similar to the example shown in FIG. 44, the statement content 286 such as "How did it happen?" Is displayed in the display area 48a as the statement content of the automatic response AI.

  When the user designates a plurality of files as the files to be cooperated with respect to the content 286 of the utterance, the specifying unit 28 (the specifying unit 28 realized by the automatic response AI) identifies the plurality of files as the files to be cooperated. Do. For example, it is assumed that the user issues an utterance content 288 such as “I want to edit the document file A and the image file C” as the utterance content indicating a usage instruction (a usage request) of a plurality of files. The control unit 26 causes the display area 48b to display the statement content 288 including the information of the document file A and the document file C as the statement content of the user, and sends the statement content 288 to the automatic response AI as the information to be cooperated. Thereby, the information of the document file A and the image file C is sent to the automatic response AI as the information of the file to be cooperated. The user may input a character string (for example, file name) indicating a plurality of files to the terminal device 10 as the user's comment content by operating the UI unit 20, as in the fourth embodiment described above. A file image (for example, an icon) associated with the file may be designated as the user's speech content. When specifying a file image, as shown in FIG. 34, the user selects a plurality of file images from the stock area 170 and moves the plurality of file images to the display position of the user's statement content. The plurality of file images may be designated as the user's message content.

  The identifying unit 28 identifies an executable collaboration function using a plurality of files designated by the user, for example, by referring to the collaboration function management table shown in FIG. In the above example, since the document file A and the image file C are specified, for example, “a function to insert an image into a document” or “a function to superimpose a document on an image” is specified as a cooperation function.

  When the user designates a plurality of files and the plurality of files are identified, the control unit 26 changes the display of the dialogue partner from the display of the general-purpose automatic response AI to the display of a plurality of files specified by the user. Switch.

  In the above example, since the document file A and the image file C are designated as the files to be cooperated, the control unit 26 changes the image 50 associated with the automatic response AI to a file image 278 associated with the document file A. And the file image 290 linked to the image file C are displayed in the display area 48a as an image linked to the other party of interaction. Of course, in place of the file images 278 and 290, or together with the file images 278 and 290, the control unit 26 displays a character string (for example, file name) indicating the document file A and the image file C as information indicating the other party. It may be displayed inside.

  In addition, the control unit 26 causes the display area 48a to display the statement content 292 such as "which function is to be used?" As the statement content of the document file A and the image file C. The statement content 292 itself is the statement content created by the general-purpose automatic response AI. The control unit 26 directs the contents 292 of the document file A to the file image 278 of the document file A and the file image of the image file C in order to produce the document file A and the image file C as if the statement contents 292 are stated. It is displayed in the display area 48 a as the content of the message linked to the number 290. The control unit 26 may display a list of linked functions specified by the specifying unit 28 in the display area 48 a as the contents of the document file A and the image file C.

  With respect to the above-mentioned statement content 292, the user inputs the cooperation function that he / she wants to execute into the terminal device 10 as the statement content of the user. In the example shown in FIG. 45, statement content 294 indicating a cooperation function such as “I want to insert an image into a document” is input to the terminal device 10 as the statement content of the user.

  The control unit 26 analyzes the user's comment content 294 to specify the cooperation function requested by the user, and controls the execution of the cooperation function. In the above example, the control unit 26 activates the document application and the image application, and inserts the image file C into the document file A.

  When the execution of the cooperation function is completed, the control unit 26 causes the display area 48a to display the statement content 296 indicating that the execution of the cooperation function is completed as the statement content of the document file A and the image file C. Also at this time, the control unit 26 displays the file image 278 linked to the document file A and the file image 290 linked to the image file C in the display area 48 a, and the contents 296 of the file image 278 of the document file A. And the file contents 290 of the image file C are displayed in the display area 48a as the contents of the message. The statement content 296 itself is the statement content created by the general-purpose automatic response AI.

  When the execution of the cooperation function is completed, the control unit 26 replaces the file image 278 associated with the document file A and the file image 290 associated with the image file C with the image 50 associated with the general-purpose automatic response AI Are displayed in the display area 48a as an image tied to the image. In addition, the control unit 26 causes the display area 48a to display an utterance content 298 indicating a general question such as "How is it done?" As the utterance content of the automatic response AI.

  The control unit 26 may cause the display unit of the UI unit 20 to display the document editing screen and the image editing screen by activating the document application and the image application. By executing the above-described cooperation function, the document editing screen displays the document file A in a state in which the image file C is inserted. The user may edit the document file A or the image file C on the document editing screen. The control unit 26 substitutes the image 50 linked to the general-purpose automatic response AI as the image linked to the dialogue partner in place of the file images 278 and 290 when the editing ends or when the editing starts. It is displayed in the display area 48a.

  In addition, a functional image associated with software (application) for processing a file may be displayed in the display area 48a as an image associated with the other party with the file image associated with the file. In the above example, the functional image associated with the document application and the functional image associated with the image application may be displayed together with the file images 278 and 290 in the display area 48a.

  As described above, when the modification example 12 is applied to the eighth embodiment, an instruction to use a plurality of files to be cooperated (use request) is given to the automatic response AI on the screen 48 that interacts with the automatic response AI. (For example, when information on a plurality of files to be cooperated is sent to the automatic response AI), the display of the conversation partner is switched to the display of the plurality of files. By doing this, the user can visually confirm whether or not the processing can be performed on the plurality of files.

  When a plurality of files are displayed as a dialog partner, the user's dialog partner may be a general-purpose automatic response AI or even if the general-purpose automatic response AI is switched to the automatic response AI for each file. Good.

  In the example shown in FIG. 45, although the information of two files is sent to the automatic response AI, the information of three or more files (for example, a character string indicating three or more file images or three or more files) May be sent to the automatic response AI. In this case, in place of the image 50 linked to the automatic response AI, three or more file images are displayed in the display area 48a as an image linked to the interactive partner.

  Further, the modification 13 may be applied to the eighth embodiment. That is, also in the eighth embodiment, each device image may be displayed according to the display example shown in FIG. 39 and FIG.

  In addition, the combination of cooperation object is not limited to the combination in said 5th Embodiment thru | or 8th Embodiment. For example, information on at least two of the device, the function (including the function executed by the device and the function executed by the software), the file, and the software is sent to the automatic response AI as the information to be cooperated In this case, the display of the general-purpose automatic response AI may be switched to the display of the combination.

  For example, when the device and file information (e.g., device image and file image) are sent to the automatic response AI as the user's utterance content, the control unit 26 links the image associated with the other party to the general-purpose automatic response AI. The image 50 is switched to the device image and the file image. Describing the specific example, when the information of the multifunction device B and the document file is sent to the automatic response AI as the user's statement content, the control unit 26 converts the image linked to the other party into the general-purpose automatic response AI. The image 50 to be linked is switched to the device image to be linked to the multifunction machine B and the file image to be linked to the document file.

  Similarly, when software and file information (e.g., a functional image and a file image linked to software) are sent to the automatic response AI as the user's utterance content, the control unit 26 performs general-purpose automatic The image 50 linked to the response AI is switched to the functional image and the file image. Describing the specific example, when the information of the document application and the document file is sent to the automatic response AI as the user's statement content, the control unit 26 links the image associated with the other party to the general-purpose automatic response AI. The image 50 to be attached is switched to the function image attached to the document application and the file image attached to the document file.

  In addition, when the information of the device, the function and the file is sent to the automatic response AI as the user's statement content, the control unit 26 displays the image associated with the other party from the general-purpose automatic response AI Switch to To give a specific example, when information on MFP B, the document application, and the document file is sent to automatic response AI as the user's statement content, control unit 26 can use a general-purpose image associated with the other party. The image 50 associated with the automatic response AI is switched to a device image associated with the multifunction machine B and a function image associated with the document application and a file image associated with the document file.

  The combination in the above specific example is only an example, and when information of a combination of devices, functions, files, and software other than the above is sent to the automatic response AI, the display of the general-purpose automatic response AI switches to the display of the combination It may be done.

  Each of the terminal device 10, the devices 12 and 14, and the server 16 described above is realized by cooperation of hardware and software as an example. Specifically, each of the terminal device 10, the devices 12 and 14, and the server 16 includes one or more processors such as a CPU (not shown). The functions of the respective units of the terminal device 10, the devices 12, 14 and the server 16 are realized by the one or more processors reading and executing the program stored in the storage device (not shown). The program is stored in the storage device via a recording medium such as a CD or a DVD or via a communication path such as a network. As another example, each part of the terminal device 10, the devices 12 and 14, and the server 16 may be realized by hardware resources such as a processor, an electronic circuit, or an application specific integrated circuit (ASIC). A device such as a memory may be used in the implementation. As another example, each unit of the terminal device 10, the devices 12, 14 and the server 16 may be realized by a digital signal processor (DSP), a field programmable gate array (FPGA) or the like.

  10 terminal device, 12, 14 devices, 16 servers, 26 control units, 28 identification units.

Claims (22)

  An information processing apparatus comprising control means for switching a display of the other party in the response to a display of the device when the user gives an instruction to use the device in the interface for interacting with the other party. The control means switches the other party in response to the device while switching the display.
An information processing apparatus according to claim 1, characterized in that. The control means switches an image associated with the other party in response to the image associated with the device as the display switching.
The information processing apparatus according to claim 1 or 2, characterized in that: The image associated with the device is an image representing the state of the device,
The information processing apparatus according to claim 3, characterized in that: The image representing the state of the device is an image representing a state in which the device is performing processing.
The information processing apparatus according to claim 4, characterized in that: The image associated with the device is an image associated with a portion of the device that is performing processing.
The information processing apparatus according to claim 3, characterized in that: The control means switches the display of the other party in the response to the display of the part of the device when the user gives an instruction to use the part of the device on the interface to the other party.
An information processing apparatus according to claim 1, characterized in that. The control means switches, as the display switching, an image associated with the interactive partner in response to an image associated with the part of the device.
The information processing apparatus according to claim 7, characterized in that: The control means is configured to, when the user gives an instruction to use the plurality of devices to execute the cooperation function using the plurality of devices in the interface, display the other party in the response in response Switch to the display of multiple devices,
An information processing apparatus according to claim 1, characterized in that. The control means switches an image associated with the other party in response to the images of the plurality of devices as the display switching.
The information processing apparatus according to claim 9, characterized in that:   An information processing apparatus comprising: control means for switching a display of the other party in the response to a display of the function when the user gives an instruction to use the function in the interface for interacting with the other party. The control means switches the other party in response to the function while switching the display.
The information processing apparatus according to claim 11, characterized in that: The control means switches an image linked to the interactive partner in response to an image linked to the function as the display switching.
The information processing apparatus according to claim 11 or 12, characterized in that: The image associated with the function is an image representing the state of the function,
The information processing apparatus according to claim 13, characterized in that: The image representing the state of the function is an image representing a state in which the function is executing processing.
The information processing apparatus according to claim 14, characterized in that: When the user gives an instruction to use the plurality of functions in order to execute a cooperation function using a plurality of functions in the interface, the control means displays a display of the other in the response. Switch to the display of multiple functions,
The information processing apparatus according to claim 11, characterized in that: The control means switches an image linked to the conversation partner in response to an image of the plurality of functions as switching of display.
The information processing apparatus according to claim 16, characterized in that:   Information processing having control means for switching the display of the other party in the response to the display of the device and the function when the user gives an instruction to use the device and the function in the interface for interacting with the other party. apparatus.   An information processing apparatus comprising: control means for switching display of the other party in response to display of the file when the user gives an instruction to use the file in the interface for interacting with the other party.   An information processing apparatus comprising control means for switching a display of the other party in the response to a display of the software when the user gives an instruction to use the software to the other side in the interface for interacting with the other party.   In the interface for the user to interact with the interaction partner, when the usage instruction of at least two combinations of the device, the function, the file and the software is given to the interaction partner, the display of the interaction partner in response is the combination of An information processing apparatus having control means for switching to display. Computer,
Control means for switching the display of the other party in the response to the display of the device when the user gives an instruction to use the device on the other side of the interface in the interface for interacting with the other person;
A program to function as