JP2019068442A - Information processing apparatus and program - Google Patents

Abstract

To provide an information processing apparatus and a program that can change functions using an image.SOLUTION: A function is changed according to a portion where a first image linked to a first function and a second image linked to a second function overlap each other. The first image and the second image are displayed on display means. A user overlaps all or part of the first image on all or part of the second image on the display means. A display color of the overlapped portion is changed.SELECTED DRAWING: Figure 34

Description

  The present invention relates to an information processing apparatus and program.

  A system in which the device body and the user interface are separated may be used. For example, Patent Document 1 discloses a technique for displaying an image indicating guidance of an operation on an operation panel that can be wirelessly communicated with an apparatus body.

  Patent Document 2 discloses a relay apparatus that relays data between a client apparatus and a service providing apparatus that provides a service.

  Patent Document 3 discloses a portable terminal that generates an augmented reality image in which additional information including state information of an image forming apparatus is superimposed on a captured image captured by an imaging unit.

  Patent Document 4 discloses a technology for displaying the usage status of each cloud server by a user on a display of a device operated by the user at the time of login in a communication system that communicates with a plurality of cloud servers.

Patent No. 5737906 JP, 2014-10769, A JP, 2014-48672, A JP 2014-238786 A

  An object of the present invention is to change functions using an image.

  The invention according to claim 1 is an information processing apparatus having control means for changing a function according to a portion where a first image attached to a first function and a second image attached to a second function overlap each other.

  The invention according to claim 2 is the information processing apparatus according to claim 1, wherein the function uses a function corresponding to the overlapping portion.

  In the invention according to claim 3, the first image and the second image are displayed on the display means, and the whole or the part of the first image is displayed on the whole or the part of the second image by the user on the display means. The information processing apparatus according to claim 1, wherein the information processing apparatus is overlapped.

  The invention according to claim 4 is the information processing apparatus according to any one of claims 1 to 3, wherein the control means changes the display color of the overlapped portion.

  The invention according to claim 5 is a program that causes a computer to function as control means that changes a function according to a portion where a first image attached to a first function and a second image attached to a second function overlap each other. .

  According to the invention which concerns on Claim 1-5, a function can be changed using an image.

FIG. 1 is a block diagram showing an image forming system according to a first embodiment of the present invention. FIG. 1 is a block diagram showing an image forming apparatus according to a first embodiment. It is a block diagram showing a server concerning a 1st embodiment. It is a block diagram showing a terminal unit concerning a 1st embodiment. FIG. 1 is a schematic view showing an appearance of an image forming apparatus. It is a figure which shows an example of an apparatus function management table. It is a figure which shows an example of a cooperation function management table. It is a figure which shows an example of the function display screen displayed on a terminal device. It is a figure which shows the apparatus of cooperation object. It is a figure which shows an example of the screen displayed on a terminal device in 1st Embodiment. It is a figure which shows an example of the screen displayed on a terminal device in 1st Embodiment. It is a figure which shows an example of the apparatus function management table which concerns on the modification 1. FIG. It is a figure which shows an example of the screen displayed on a terminal device in the modification 1. FIG. It is a figure which shows an example of the screen displayed on a terminal device in the modification 1. FIG. It is a figure which shows an example of the screen displayed on a terminal device in the modification 1. FIG. It is a figure which shows an example of the screen displayed on a terminal device in modification 2. FIG. It is a figure which shows an example of the screen displayed on a terminal device in modification 2. FIG. It is a figure which shows an example of the screen displayed on a terminal device in modification 2. FIG. FIG. 18 is a diagram showing an example of a screen displayed on the terminal device in the modification 3. FIG. 18 is a diagram showing an example of a screen displayed on the terminal device in the modification 4; FIG. 6 is a sequence diagram showing connection processing in the image forming system according to the first embodiment. It is a block diagram showing the server concerning a 2nd embodiment. It is a figure which shows an example of a cooperation function management table. It is a figure which shows an example of the screen displayed on a terminal device in 2nd Embodiment. It is a figure which shows an example of the screen displayed on a terminal device in 2nd Embodiment. FIG. 18 is a diagram showing an example of a screen displayed on the terminal device in the fifth modification. FIG. 18 is a diagram showing an example of a screen displayed on the terminal device in the fifth modification. FIG. 18 is a diagram showing an example of a screen displayed on the terminal device in the fifth modification. It is a figure which shows an example of the screen displayed on a terminal device in the modification 6. FIG. It is a figure which shows an example of the screen displayed on a terminal device in the modification 6. FIG. It is a figure which shows an example of the screen displayed on a terminal device in the modification 6. FIG. FIG. 21 is a diagram illustrating an example of a screen displayed on a terminal device in a modification 7; It is a figure which shows an example of the screen displayed on a terminal device in the modification 8. FIG. It is a figure which shows an example of the screen displayed on a terminal device in modification 10. FIG. It is a figure which shows an example of the screen displayed on a terminal device.

First Embodiment
An image forming system as an information processing system according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 shows an example of the image forming system according to the first embodiment.

  The image forming system according to the first embodiment includes an image forming apparatus 10 as an example of an apparatus, a terminal apparatus 12 as an example of an apparatus, a server 14 and a terminal apparatus 16 as an example of an information processing apparatus. . The image forming apparatus 10, the terminal devices 12 and 16, and the server 14 are connected to one another via a communication path N such as a network. Of course, the image forming apparatus 10, the terminal devices 12 and 16, and the server 14 may be connected to other devices via different communication paths. Although two devices (the image forming apparatus 10 and the terminal device 12) are included in the image forming system in the example illustrated in FIG. 1, one or more devices may be included in the image forming system. Also, a plurality of servers 14 and a plurality of terminal devices 16 may be included in the image forming system.

  The image forming apparatus 10 is an apparatus having an image forming function. Specifically, the image forming apparatus 10 is an apparatus provided with at least one of a scan function, a print function, a copy function, and a facsimile function. In addition, the image forming apparatus 10 has a function of transmitting and receiving data to and from another apparatus.

  The terminal devices 12 and 16 are devices such as a PC (personal computer), a tablet PC, a smartphone, and a mobile phone, and have a function of transmitting and receiving data to and from other devices. The terminal device 16 functions as a user interface unit (UI unit), for example, when using an apparatus such as the image forming apparatus 10 or the like.

  The server 14 is a device that manages the function of each device. For example, the server 14 manages a collaboration function that is executed by linking a plurality of functions. The server 14 also has a function of transmitting and receiving data to and from other devices.

  In the example shown in FIG. 1, the image forming apparatus 10 and the terminal device 12 are included in the image forming system as an example of the device, but this is merely an example. A display device such as a projector, a device such as a telephone, a watch, a surveillance camera, or the like may be included as an apparatus in the image forming system. Of course, the image forming apparatus 10 and the terminal device 12 may not be included in the image forming system.

  The server 14 may manage, for each user, functions available to the user. For example, the function purchased by the user is a function available to the user, and the server 14 may manage the function purchase history for each user. Of course, since there are functions available free of charge, functions of additional update data, functions specially managed by the administrator, etc., the server 14 not only manages functions according to the presence or absence of purchase. The purchase process of the function is performed by, for example, the server 14. Of course, the purchase process may be performed by another device.

  In the image forming system according to the first embodiment, a cooperation function to be executed is specified by causing a plurality of functions to cooperate, and information on the plurality of cooperation functions is displayed in the order of priority.

  Hereinafter, each device included in the image forming system according to the first embodiment will be described in detail.

  First, the configuration of the image forming apparatus 10 will be described in detail with reference to FIG. The configuration of the image forming apparatus 10 is shown in FIG.

  The communication unit 18 is a communication interface, and has a function of transmitting data to another device and a function of receiving data from the other device. The communication unit 18 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

  The image forming unit 20 performs a function related to image formation. Specifically, the image forming unit 20 executes at least one of the scan function, the print function, the copy function, and the facsimile function. By executing the scan function, the document is read and scan data (image data) is generated. By executing the print function, an image is printed on a recording medium such as paper. By executing the copy function, the original is read and printed on the recording medium. By executing the facsimile function, image data is transmitted by facsimile or received by facsimile. Further, a cooperation function in which a plurality of functions are cooperated may be executed. For example, a scan transfer function combining the scan function and the transmission function (transfer function) may be performed. By executing the scan transfer function, a document is read to generate scan data (image data), and the scan data is transmitted to a transmission destination (for example, an external device such as the terminal device 16). Of course, this cooperation function is only an example, and another cooperation function may be executed.

  The storage unit 22 is a storage device such as a hard disk or an SSD. The storage unit 22 indicates, for example, information indicating a command for image formation (for example, job information etc.), image data to be printed, scan data generated by executing a scan function, and addresses of other devices. Device address information, server address information indicating the address of the server 14, various control data, various programs, and the like are stored. Of course, they may be stored in separate storage devices, or may be stored in one storage device.

  The UI unit 24 is a user interface unit, and includes a display unit and an operation unit. The display unit is, for example, a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel or a keyboard. The image forming apparatus 10 may not include the UI unit 24 or may include a hardware user interface unit (hardware UI unit) as hardware without including the display unit. The hardware UI unit is, for example, a hardware key specialized for numeric input (for example, a ten key), a hardware key specialized for direction indication (for example, a direction indication key), or the like.

  The control unit 26 controls the operation of each unit of the image forming apparatus 10.

  Next, the configuration of the server 14 will be described in detail with reference to FIG. The configuration of the server 14 is shown in FIG.

  The communication unit 28 is a communication interface, and has a function of transmitting data to another device and a function of receiving data from the other device. The communication unit 28 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

  The storage unit 30 is a storage device such as a hard disk or an SSD. The storage unit 30 stores, for example, device function information 32, cooperation function information 34, various data, various programs, device address information indicating the address of each device, server address information indicating the address of the server 14, etc. ing. Of course, they may be stored in separate storage devices, or may be stored in one storage device. The device function information 32 and the cooperation function information 34 stored in the storage unit 30 are provided to the terminal device 16 periodically or at designated timing, whereby the information stored in the terminal device 16 is updated. May be The device function information 32 and the cooperation function information 34 will be described below.

  The device function information 32 is information indicating the function of each device (for example, the image forming apparatus 10 and the terminal device 12) included in the image forming system, and, for example, device identification information for identifying the device and the device It is information which shows correspondence with function identification information for identifying a function which it has. The device identification information is, for example, a device ID, a device name, information indicating a type of device, a model number, position information, and the like. The function identification information is, for example, a function ID or a function name. For example, when an image forming apparatus 10 has a scan function, a print function, a copy function, and a scan transfer function, the device identification information of the image forming apparatus 10 indicates function identification information indicating a scan function, and a print function. The function identification information, the function identification information indicating the copy function, and the function identification information indicating the scan transfer function are associated with each other. By referring to the device function information 32, the function that each device has is identified (identified).

  The cooperation function information 34 is information indicating a cooperation function executed by causing a plurality of functions to cooperate. By linking a plurality of functions, one or more linking functions are executed. The cooperation function may be executed by linking a plurality of functions of one device (for example, the image forming apparatus 10 or the terminal device 12), or a plurality of devices (for example, the image forming apparatus 10 and the terminal device 12) It may be implemented by linking a plurality of functions that it has. Further, the terminal device (in the present embodiment, the terminal device 16) issuing the operation instruction may be included in the device to be identified, and the function of the terminal device may be used as part of the cooperation function.

  The cooperation function information 34 includes, for example, information indicating a combination of function identification information for identifying each function to be cooperated to execute the cooperation function, cooperation function identification information for identifying the cooperation function, and The information indicates the correspondence between information indicating the order of connecting each function used for the collaboration function and information indicating the display priority of the collaboration function. The cooperation function identification information is, for example, a cooperation function ID or a cooperation function name. The display priority is the order in which the terminal device 16 displays information on the cooperation function (for example, cooperation function identification information).

  In the case of linking a plurality of devices, the linking function information 34 is information indicating a linking function to be executed by linking a plurality of functions possessed by the plurality of devices, and each device linked to execute the linking function Information indicating the combination of the device identification information for identifying the information, the cooperation function identification information, the information indicating the order of connecting the respective devices used for the cooperation function, and the information indicating the display priority of the cooperation function It is the information which shows the correspondence of and.

  The linkage function may be a function executed by linking a plurality of different functions with each other, or may be a function executed by linking the same functions. The cooperation function may be a function that can not be used before cooperation. The function that can not be used before the cooperation may be a function that can be used by using the same function among the functions possessed by the cooperation target device, or a function that can be used by combining different functions. May be For example, by linking an apparatus (printer) having a print function and an apparatus (scanner) having a scan function, a copy function as a cooperation function is realized. That is, the copy function is realized by linking the print function and the scan function. In this case, the copy function is associated with the combination of the print function and the scan function. In the cooperation function information 34, for example, cooperation function identification information for identifying a copy function as a cooperation function, apparatus identification information for identifying an apparatus having a print function, and an apparatus for identifying an apparatus having a scan function. A combination of device identification information is associated with each other.

  The storage unit 30 may store function purchase history information. The function purchase history information is information indicating a purchase history of a function by each user, that is, information indicating a function purchased by each user, for example, user identification information for identifying the user, and purchase by the user It is information indicating the correspondence with the function identification information indicating the function. The user identification information is, for example, user account information such as a user ID and a name. The features purchased by a user are the features available to the user. By referring to the function purchase history information, the functions purchased by each user, that is, the functions available to each user are identified. The function purchase history information is updated, for example, each time the user purchases a function.

  The control unit 36 controls the operation of each unit of the server 14. Further, the control unit 36 includes a specifying unit 38.

  The specifying unit 38 receives device identification information for identifying a device, and specifies function identification information of a function associated with the device identification information in the device function information 32 stored in the storage unit 30. Thereby, the function that the device has is identified. For example, device identification information is transmitted from the terminal device 16 to the server 14, and the identification unit 38 identifies function identification information of a function associated with the device identification information. Information related to the function (for example, function identification information, function explanatory information, etc.) is transmitted from the server 14 to the terminal device 16 and displayed, for example. Thereby, the information regarding the function which the apparatus specified by the apparatus identification information has is displayed on the terminal device 16.

  Further, the specifying unit 38 receives device identification information for identifying each device to be cooperated, and the cooperation function associated with the combination of the respective device identification information in the cooperation function information 34 stored in the storage unit 30. Identify the linkage function identification information of As a result, the cooperation function to be executed is identified by causing the functions of the respective devices to be cooperated to cooperate. For example, a plurality of pieces of device identification information are transmitted from the terminal device 16 to the server 14, and the identification unit 38 identifies cooperation function identification information of the cooperation function associated with the plurality of pieces of device identification information. Information related to the cooperation function (for example, cooperation function identification information, explanation information of the cooperation function, etc.) is transmitted from the server 14 to the terminal device 16 and displayed, for example. Thereby, the information regarding the cooperation function performed by the plurality of devices specified by the plurality of device identification information is displayed on the terminal device 16.

  Further, the specifying unit 38 refers to the cooperation function information 34 and determines the display priority of the information on the cooperation function based on the connection order of each function used for the cooperation function. For example, by linking a plurality of functions, a plurality of linking functions may be realized. In this case, based on the connection order of each function, the display priority of information on each cooperation function included in the plurality of cooperation functions is determined. The terminal device 16 displays the information related to the cooperation function in accordance with the priority. For example, the image linked with the device is displayed on the terminal device 16, and the images are connected by the user. When linking a plurality of functions of a plurality of devices, a plurality of images linked to the plurality of devices are linked by the user. The order of connecting the images corresponds to the order of connecting the functions. The identifying unit 38 identifies the display priority of the information related to the cooperation function based on the order of connecting the images.

  When the purchase history of the function is managed, the identification unit 38 receives the user identification information for identifying the user, and associates the user identification information in the function purchase history information stored in the storage unit 30 with the user identification information. The function identification information of each function being identified may be specified. Thereby, the function group purchased by the user, that is, the function group available by the user is specified (identified). For example, user identification information is transmitted from the terminal device 16 to the server 14, and the identification unit 38 identifies function identification information of each function associated with the user identification information. The function identification information of each function (for example, information indicating the name of each function) is transmitted from, for example, the server 14 to the terminal device 16 and displayed. Thereby, information (function identification information) for identifying a function available to the user specified by the user identification information is displayed on the terminal device 16. For example, the identification unit 38 receives the device identification information and the user identification information, identifies the function identification information of the function associated with the device identification information in the device function information 32, and also identifies the user in the function purchase history information. The function identification information of the function associated with the identification information is specified. As a result, the functions possessed by the device identified by the device identification information that are usable by the user identified by the user identification information are identified. The function identification information indicating the function is transmitted from, for example, the server 14 to the terminal device 16 and displayed.

  The control unit 36 may execute a process of purchasing a function and manage the history of the purchase. For example, when a paid function is purchased by a user, the control unit 36 applies a charging process to the user and creates function purchase history information indicating a history of the purchase.

  Next, the configuration of the terminal device 16 will be described in detail with reference to FIG. The configuration of the terminal device 16 is shown in FIG.

  The communication unit 40 is a communication interface, and has a function of transmitting data to another device and a function of receiving data from the other device. The communication unit 40 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

  The camera 42 as a photographing means generates image data (for example, still image data and moving image data) by photographing an object to be photographed. In addition to using the camera of the terminal device 16, the communication unit 40 receives image data captured by an external camera connected to a communication path such as a network, and the UI unit 46 displays the image data, and the image data is displayed. May be operated.

  The storage unit 44 is a storage device such as a hard disk or an SSD, and various programs, various data, address information of the server 14, address information of each device (for example, address information of each image forming apparatus 10 or terminal device 12), identification It stores information on the linked device, information on the linked function, etc.

  The UI unit 46 is a user interface unit, and includes a display unit and an operation unit. The display unit is, for example, a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel, a keyboard, or a mouse. Of course, a user interface (for example, a touch-type display or one that displays a keyboard or the like on a display electronically) that doubles as a display unit and an operation unit may be used.

  The control unit 48 controls the operation of each unit of the terminal device 16. The control unit 48 functions as, for example, a display control unit (control unit), and causes the display unit of the UI unit 46 to display various types of information.

  On the display unit of the UI unit 46, for example, an image captured by the camera 42, an image linked to a device identified as a device to be used (for example, a device to be cooperated), and the like are displayed. The image linked to the device may be an image (still image or moving image) representing the device taken by the camera 42, or may be an image (for example, an icon) schematically showing the device. For example, data of an image schematically represented may be stored in the server 14 and provided from the server 14 to the terminal device 16 or may be stored in advance in the terminal device 16. For example, when a plurality of devices are linked to execute the linkage function, the plurality of devices are photographed by the camera 42, and an image representing each device is displayed on the display unit of the UI unit 46. By linking each image linked to each device to be cooperated by the user, the cooperation function to be executed is specified by the device group to be cooperated, and the information on the cooperation function is the UI unit 46 according to the priority order according to the connection order. Is displayed on the display unit of

  The terminal device 12 as an apparatus has, for example, the same configuration as that of the terminal device 16 described above. The terminal device 12 may not have the camera 42.

  The device function information 32 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the device function information 32 may not be stored in the storage unit 30 of the server 14. Similarly, the cooperation function information 34 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the cooperation function information 34 may not be stored in the storage unit 30 of the server 14. In addition, the control unit 48 of the terminal device 16 may include the above-described identification unit 38, identify the device based on the device identification information, and identify the function possessed by the device, or by linking a plurality of functions. The cooperation function to be executed may be specified. In this case, the server 14 may not include the identifying unit 38. When the function purchase history information is created, the function purchase history information may be stored in the storage unit 44 of the terminal device 16. In this case, the function purchase history information may not be stored in the storage unit 30 of the server 14. Further, the control unit 48 of the terminal device 16 may manage the history of purchase of the function by the user. In this case, the control unit 36 of the server 14 may not have the management function. In addition, the control unit 48 of the terminal device 16 may specify a function available to the user based on the user identification information. Further, as another example, the device function information 32 and the cooperation function information 34 may be stored in the device such as the image forming device 10 or the terminal device 12 or the device such as the image forming device 10 or the terminal device 12 is specified. The part 38 may be provided. That is, the processing by the specifying unit 38 of the server 14 may be performed in the server 14, may be performed in the terminal device 16, or may be performed in an apparatus such as the image forming apparatus 10 or the terminal device 16. .

  In the first embodiment, as an example, by applying an AR (Augmented Reality) technology (augmented reality technology), device identification information is acquired and a device to be used is identified. For example, by applying the AR technology, the device identification information of the device to be cooperated is acquired, and the device to be cooperated is identified. AR technology known as AR technology is used. For example, a marker type AR technology using a marker such as a two-dimensional barcode, a markerless type AR technology using an image recognition technology, a position information AR technology using position information, or the like is used. Of course, the device identification information may be acquired and the device to be used may be identified without applying the AR technology. For example, as long as the device is connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device.

  Hereinafter, with reference to FIG. 5, the process of acquiring the device identification information of the device will be described in detail. A case where device identification information of the image forming apparatus 10 is acquired will be described as an example. The appearance of the image forming apparatus 10 is schematically shown in FIG. Here, processing for acquiring device identification information by applying the marker type AR technology will be described. The case of the image forming apparatus 10 is provided with a marker 50 such as a two-dimensional bar code. The marker 50 is information in which the device identification information of the image forming apparatus 10 is coded. The user activates the camera 42 of the terminal device 16 and photographs the marker 50 provided on the image forming apparatus 10 to be used by the camera 42. Thereby, image data representing the marker 50 is generated. The image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 extracts the device identification information by applying the decoding process to the marker image represented in the image data. As a result, the image forming apparatus 10 to be used (the image forming apparatus 10 having the photographed marker 50) is identified. The specifying unit 38 of the server 14 specifies the function identification information of the function associated with the extracted device identification information in the device function information 32. Thereby, the function of the image forming apparatus 10 to be used is specified.

  The control unit 48 of the terminal device 16 may extract the device identification information by applying the decoding process to the image data representing the marker 50. In this case, the extracted device identification information is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies, in the device function information 32, the function identification information of the function associated with the device identification information transmitted from the terminal device 16. When the device function information 32 is stored in the storage unit 44 of the terminal device 16, the control unit 48 of the terminal device 16 determines the function of the function associated with the extracted device identification information in the device function information 32. Identification information may be specified.

  The marker 50 may include function identification information of the function of the image forming apparatus 10 in a coded form. In this case, by applying the decoding process to the image data representing the marker 50, the function identification information of the image forming apparatus 10 is extracted, and the function identification information of the function of the image forming apparatus 10 is also extracted. Thus, the image forming apparatus 10 is specified, and the function of the image forming apparatus 10 is specified. This decoding process may be performed by the server 14 or may be performed by the terminal device 16.

  When the cooperation function is executed by linking a plurality of devices, the device identification information of each device is acquired by photographing the markers of the respective devices to be cooperated, whereby the cooperation function is specified.

  When the device identification information is acquired by applying the markerless AR technology, for example, the user captures all or part of the appearance of the device to be used (for example, the image forming apparatus 10) by the camera 42 of the terminal device 16. Of course, it is useful to obtain information for identifying the device, such as the name of the device to be used (for example, a product name) and a model number from the appearance. By photographing, appearance image data representing all or part of the appearance of the device to be used is generated. The appearance image data is transmitted from, for example, the terminal device 16 to the server 14. In the server 14, the control unit 36 identifies the device to be used based on the appearance image data. For example, the storage unit 30 of the server 14 stores, for each device, appearance image correspondence information indicating correspondence between appearance image data representing all or part of the appearance of the device and device identification information of the device. It is done. The control unit 36 compares, for example, the appearance image data transmitted from the terminal device 16 with the appearance image data included in the appearance image correspondence information, and based on the comparison result, device identification information of the device to be used. Identify For example, the control unit 36 extracts the feature of the appearance of the device to be used from the appearance image data transmitted from the terminal device 16, and the appearance image data group included in the appearance image association information is the same as the feature of the appearance Alternatively, appearance image data representing similar features is identified, and device identification information associated with the appearance image data is identified. Thus, the device to be used (the device photographed by the camera 42) is recognized. As another example, if the device name (for example, a product name) or a model number is captured and appearance image data representing the name or model number is generated, the name or model number represented in the appearance image data is used. The device to be used may be identified. The specifying unit 38 of the server 14 specifies the function identification information of each function associated with the specified device identification information in the device function information 32. Thereby, the function of the device to be used (for example, the image forming apparatus 10) is specified.

  The control unit 48 of the terminal device 16 compares the appearance image data representing all or part of the appearance of the device to be used (for example, the image forming apparatus 10) with the appearance image data included in the appearance image correspondence information. The device identification information of the device to be used may be specified based on the comparison result. The appearance image correspondence information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the appearance image correspondence information stored in the storage unit 44 of the terminal device 16. Alternatively, the control unit 48 of the terminal device 16 may specify the device identification information of the device to be used by acquiring the appearance image association information from the server 14 and referring to the appearance image association information.

  When the cooperation function is executed by linking a plurality of devices, the device identification information of each device is acquired by photographing all or part of the appearance of each device to be cooperated, whereby the cooperation function is specified. Ru.

  When acquiring device identification information by applying position information AR technology, for example, position information indicating a position where an apparatus (for example, the image forming apparatus 10) is installed is available by using a GPS (Global Positioning System) function. It is acquired. For example, each device has a GPS function, and acquires device position information indicating the position of the device itself. The terminal device 16 outputs information indicating an acquisition request of the device position information to the device to be used, and receives the device position information of the device from the device as a response to the acquisition request. The device position information is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 identifies the device to be used based on the device position information. For example, the storage unit 30 of the server 14 stores, for each device, position association information indicating correspondence between device position information indicating the position where the device is installed and device identification information of the device. There is. The control unit 36 identifies the device identification information associated with the device location information transmitted from the terminal device 16 in the location association information. Thereby, the device to be used is recognized. The specifying unit 38 of the server 14 specifies the function identification information of each function associated with the specified device identification information in the device function information 32. Thereby, the function of the device to be used (for example, the image forming apparatus 10) is recognized.

  The control unit 48 of the terminal device 16 may specify, in the position association information, the device identification information associated with the position information of the device to be used. The position association information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the position association information stored in the storage unit 44 of the terminal device 16. As another example, the control unit 48 of the terminal device 16 may obtain the position association information from the server 14 and specify the device identification information of the device to be used by referring to the position association information. .

  When the cooperation function is executed by linking a plurality of devices, device position information of each device to be cooperated is acquired, and device identification information of each device is specified based on the device position information. Thereby, the cooperation function is identified.

  The image forming system according to the first embodiment will be described in detail below.

  First, the device function information 32 will be described in detail with reference to FIG. An example of the device function management table as the device function information 32 is shown in FIG. In the device function management table, as an example, a device ID, information indicating a device name (for example, type of device), information indicating a function of the device (function identification information), and an image ID are associated with each other. There is. The device ID and the device name correspond to an example of device identification information. The image ID is an example of image identification information for identifying an image (for example, an icon) that schematically represents the device. The device function management table may not include the image ID. For example, the device whose device ID is “B” is a multifunction peripheral (image forming apparatus having a plurality of image forming functions) and has functions such as a print function and a scan function. The device is associated with an image ID for identifying an image that schematically represents the device. Data of an image schematically representing the device is stored, for example, in the storage unit 30 of the server 14.

  For example, by applying the AR technology, a device ID for identifying a device to be used is acquired. The specifying unit 38 of the server 14 specifies the device name, the function, and the image ID associated with the device ID by referring to the device function management table. Thus, the device to be used is identified. For example, information indicating the device name and data of an image schematically representing the device are transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. An image schematically representing the device is displayed as an image linked to the device. Of course, the image itself captured by the camera 42 may be displayed on the UI unit 46 of the terminal device 16 without displaying an image schematically representing the device. In addition, when the image (for example, an image photographed by the camera 42 or an image schematically representing the device) associated with the device is designated by the user in the UI unit 46 of the terminal device 16, the function of the device is related. Information (for example, function identification information, function explanation information, etc.) may be transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

  Next, the cooperation function information 34 will be described in detail with reference to FIG. An example of the cooperation function management table as the cooperation function information 34 is shown in FIG. In the linked function management table, as an example, information indicating a combination of device IDs, information indicating a linked device name (for example, type of each device), information indicating a linked function (linked function identification information), and connection order And the information indicating the priority are associated with each other. The connection order corresponds to the order in which the images linked to the device are connected. The priority is a display priority of information on the cooperation function. For example, the device with the device ID “A” is a PC (personal computer), and the device with the device ID “B” is a multifunction peripheral. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” and a “print function” are realized as linked functions. The “scan transfer function” is a function of transferring image data generated by scanning by the multifunction peripheral (B) to the PC (A). The “printing function” is a function of transmitting data (for example, image data and document data) stored in the PC (A) to the multifunction peripheral (B) and printing the data in the multifunction peripheral (B). For example, when a device is connected from the multifunction device (B) to the PC (A), that is, when an image is linked from an image linked to the multifunction device (B) to an image linked to the PC (A) The priority of the “transfer function” is “first place”, and the priority of the “print function” is “second place”. In this case, the information on the "scan transfer function" is displayed prior to the information on the "print function". Conversely, when a device is connected from the PC (A) to the multifunction machine (B), that is, the image is linked from the image linked to the PC (A) to the image linked to the multifunction machine (B) In this case, the priority of the “print function” is “first place”, and the priority of the “scan transfer function” is “second place”. In this case, the information on the "print function" is displayed prior to the information on the "scan transfer function".

  Next, the screen displayed on the UI unit 46 of the terminal device 16 will be described in detail with reference to FIG. An example of the function display screen is shown in FIG. The function display screen is a screen displayed on the UI unit 46 of the terminal device 16 when using a single device.

  For example, by using any of the marker type AR technology, the markerless type AR technology, or the position information AR technology described above, the device identification information of the device to be used (for example, the image forming apparatus 10) is acquired and A device is identified, and function identification information indicating each function associated with the device identification information, that is, function identification information indicating each function possessed by the device to be used (identified device) is identified. In addition, function identification information indicating each function associated with user identification information of a user who uses a device to be used, that is, function identification information indicating each function available by the user may be specified. Those pieces of information are displayed on the UI unit 46 of the terminal device 16 as a function display screen. Further, since the function possessed by the device to be used is identified, the function not possessed by the device to be utilized may be identified among the functional groups to be sold. Function identification information indicating each function that the device to be used does not have may be displayed on the function display screen.

  On the function display screen 52 shown in FIG. 8, a button image 54 representing the function A, a button image 56 representing the function B, and a button image 58 representing the function C are displayed as an example of the function identification information. ing. The function A is a function of a device to be used (a device identified by AR technology or the like) and is a function available to the user, that is, a function purchased by the user. The function B is a function that the device to be used has, and is a function that can not be used by the user, that is, a function that has not been purchased by the user. When the user purchases the function B, the user can use the function B. The function C is a function that the device to be used does not have, that is, a function to which the device to be used does not correspond. The control unit 48 of the terminal device 16 may change the display mode of the button image in accordance with whether or not the function represented by the button image is the function possessed by the device to be used. In addition, the control unit 48 may change the display mode of the button image depending on whether the function represented by the button image is a function available to the user. For example, the control unit 48 changes the color or shape of the button image. In the example shown in FIG. 8, the button images 54, 56, 58 are displayed in different colors. As one example, a button image (for example, a button image 54 representing function A) which is a function possessed by the device to be used and which represents a function available to the user is displayed in blue. A button image (button image 56 representing function B) representing a function possessed by the device to be used that can not be used by the user is displayed in yellow. The button image (for example, the button image 58 representing the function C) representing the function that the device to be used does not have is displayed in gray. As another example, the control unit 48 may change the shapes of the button images 54, 56, 58, or may change the font of the function display name. Of course, the display mode may be changed by another method. Thereby, the user can be notified of availability of each function with high visibility.

  For example, when the user designates the button image 54 representing the function A using the terminal device 16 and gives an instruction to execute the function A, execution instruction information indicating an instruction to execute the function A is to be used from the terminal device 16 Sent to the device. The execution instruction information includes control data for executing the function A, data such as image data to which the process by the function A is applied, and the like. When receiving the execution instruction information, the device to be used executes the function A in accordance with the execution instruction information. For example, when the device to be used is the image forming apparatus 10 and the function A is a scan transfer function, the image forming unit 20 of the image forming apparatus 10 executes a scan function, whereby scan data (image data) is generated. It is generated. The scan data is transmitted from the image forming apparatus 10 to the set destination (for example, the terminal device 16).

  When the user designates the button image 56 representing the function B using the terminal device 16 and instructs the purchase of the function B, the purchase process is executed. When the purchase process is performed by the server 14, the terminal device 16 accesses the server 14. As a result, a screen (for example, a website) for purchasing the function B is displayed on the UI unit 46 of the terminal device 16 as information for enabling the user to use the function B. By performing the purchase procedure on this screen, the use of the function B by the user is permitted. For example, a program of a web browser is stored in the terminal device 16, and access from the terminal device 16 to the server 14 is realized by using the web browser. When the user accesses the server 14 using a web browser, a function purchase screen (for example, a web site) is displayed on the UI unit 46 of the terminal device 16, and the function is purchased via the web site. Of course, the purchase process may be performed by a server other than the server 14 or the like. As another example, in the UI unit 46 of the terminal device 16, a usage permission request screen (for example, the web) for requesting the administrator etc. to use the function B as information for enabling the user to use the function B. The site may be displayed. If the administrator or the like is requested to permit the use of the function B via the use permission request screen, and the permission is obtained, the user can use the function B.

  Hereinafter, a method of linking a plurality of functions by applying AR technology will be described in detail.

  A method of identifying a plurality of devices by applying a marker type AR technology or a markerless type AR technology and linking a plurality of functions of the plurality of devices will be described with reference to FIG. FIG. 9 shows an example of the device to be cooperated. As an example, the image forming apparatus 10 and the terminal device 12 are used as devices to be cooperated. The image forming apparatus 10 is, for example, a multifunction peripheral, and the terminal device 12 is, for example, a PC (personal computer). The image forming apparatus 10 and the terminal device 12 are devices existing in the real space. The terminal device 16 is a device that exists in the real space and is used when linking a plurality of functions. The terminal device 16 is, for example, a mobile terminal device such as a smartphone or a mobile phone.

  For example, a marker 50 such as a two-dimensional bar code is provided in the case of the image forming apparatus 10, and a marker 60 such as a two-dimensional bar code is provided in the case of the terminal device 12 (PC). The marker 60 is information in which the device identification information of the terminal device 12 is coded. When the marker type AR technology or the markerless type AR technology is used, the user captures an image of the cooperation target image forming apparatus 10 and the terminal apparatus 12 with the camera 42 of the terminal apparatus 16 (for example, a smartphone). In the example illustrated in FIG. 9, the image forming apparatus 10 and the terminal device 12 are photographed together in a state in which both the image forming apparatus 10 and the terminal device 12 are included in the imaging area of the camera 42. Thereby, image data representing each of the markers 50 and 60 is generated and transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 applies the decoding process to the image data to extract the device identification information of the image forming device 10 and the device identification information of the terminal device 12, thereby the image forming device 10 and the terminal device Identify 12 As another example, appearance image data representing the appearance of both the image forming apparatus 10 and the terminal device 12 may be generated and transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 specifies the device identification information of the image forming apparatus 10 and the device identification information of the terminal device 12 by referring to the appearance image correspondence information. Thereby, the image forming apparatus 10 and the terminal device 12 are identified. Of course, the process for identifying the device to be cooperated may be performed by the terminal device 16.

  The device display screen 62 is displayed on the display unit of the UI unit 46 of the terminal device 16, and the device image 64 linked to the image forming apparatus 10 and the device image 66 linked to the terminal device 12 are displayed on the device display screen 62. Ru. The device images 64 and 66 may be images of the devices (the image forming apparatus 10 and the terminal device 12) taken by the camera 42 (the images at the time of shooting, or may be enlarged or reduced images) The image data may be the image data prepared in advance (not the image obtained by photographing itself but a schematic image (for example, an icon)). As described above, an image representing a device captured by an external camera may be used.

  For example, when using image data obtained by photographing the device, the appearance of the current device itself (for example, the appearance in which a scratch, a note, a sticker attached to the device, or the like is reflected) is reflected in the image As a result, it is possible for the user to visually recognize the difference from other similar devices.

  When a schematic image is used, data of the schematic image is transmitted from the server 14 to the terminal device 16, for example. For example, when the image forming apparatus 10 is identified, the identifying unit 38 of the server 14 refers to the device function management table (device function information 32) illustrated in FIG. Specific images. The data of the schematic image is transmitted from the server 14 to the terminal device 16, and the schematic image is displayed on the device display screen 62 as the device image 64. The same applies to the device image 66 linked to the terminal device 12. Data of the device images 64 and 66 as schematic images may be stored in the terminal device 16 in advance. In this case, when the image forming apparatus 10 is identified, the device image 64 stored in the terminal device 16 is displayed on the device display screen 62. The same applies to the terminal device 12. The data of the schematic image may be stored in a device other than the server 14 or the terminal device 16.

  In addition, when the device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the terminal device 16 may display the name of the device on the device display screen 62. In the example shown in FIG. 9, the image forming apparatus 10 is a multifunction peripheral, and its name is "multifunctional peripheral (B)". The terminal device 12 is a PC (personal computer), and its name is "PC (A)".

  When the device to be imaged is not identified even if the AR technology or the like is applied, the device image representing the device to be imaged does not have to be displayed on the device display screen 62. This improves the visibility of the identified device. For example, when there are a mixture of identified devices and unidentified devices, and each of them is photographed by the camera 42, the device image representing the unidentified devices is not displayed. As a result, since the device image representing the identified device is displayed without being buried in the device image representing the non-identified device, the visibility of the identified device is improved. Also, an instrument image (for example, instrument images 64 and 66) representing the identified instrument may be displayed in an emphasized manner. For example, the device image representing the identified device may be displayed in a specific color, the edge of the device image may be highlighted, or the device image may be enlarged and displayed. The device image may be displayed in three dimensions, or may be displayed by vibrating the device image. This improves the visibility of the identified device.

  When the user operates the device images 64 and 66 on the device display screen 62, the image forming apparatus 10 and the terminal device 12 are designated as the devices to be cooperated, and by causing the image forming device 10 and the terminal device 12 to cooperate. The linkage function to be executed is identified. Information on the cooperation function is displayed on the UI unit 46 of the terminal device 16 according to the priority order according to the order in which the device image is operated.

  Hereinafter, with reference to FIG. 10 and FIG. 11, the operation for specifying the cooperation function will be described in detail.

  10 and 11 show an example of a screen displayed on the UI unit 46 of the terminal device 16. For example, it is assumed that the image forming apparatus 10 and the terminal device 12 are identified as devices to be cooperated with each other. As shown in FIG. 10A, a device display screen 62 is displayed on the UI unit 46 of the terminal device 16, and device images 64 and 66 are displayed on the device display screen 62. In this state, the user uses an indicator (for example, a user's finger, a pen, a stylus, or the like) to connect the device image representing the device to be cooperated. The control unit 48 of the terminal device 16 detects the contact of the indicator on the device display screen 62 and detects the movement of the indicator on the device display screen 62. For example, as indicated by the arrow 68, the user may use the operator to touch the device image 64 on the device display screen 62 to operate the device image 66 (for example, to trace the operation member on the device display screen 62) Thus, the device image 64 and the device image 66 are connected. As a result, the image forming apparatus 10 (the multifunction machine (B)) linked to the device image 64 and the terminal device 12 (PC (A)) linked to the device image 66 are designated as linked devices, and connected. The order is specified. The order of connecting device images corresponds to the order of connection. In the example shown in FIG. 10A, since the image is connected from the device image 64 to the device image 66, the device is connected from the image forming apparatus 10 (MFP (B)) to the terminal device 12 (PC (A)). It is done. The information indicating the connection order of the devices is transmitted from the terminal device 16 to the server 14. The control unit 48 of the terminal device 16 may display an image representing the locus traced by the user on the device display screen 62, and after the devices are connected, the locus is previously set as a straight line, etc. And may be displayed on the device display screen 62.

  As described above, when devices to be cooperated (for example, the image forming apparatus 10 and the terminal device 12) are designated, the specifying unit 38 of the server 14 performs the cooperation function management table (cooperation function information 34 shown in FIG. 7). , Identify the cooperation function associated with the combination of the PC (A) and the multifunction machine (B). Thus, the cooperation function to be executed by causing the PC (A) and the multifunction machine (B) to cooperate is specified. In addition, when the connection order of devices is specified by the user, the specification unit 38 specifies the priority order associated with the connection order in the cooperation function management table. A specific example will be described with reference to FIG. 7. Since the PC (A) and the multi-functional peripheral (B) are designated as devices to be cooperated with each other, the cooperation function executed by them is “scan transfer function”. It is a "printing function". Also, since the device is connected from the multi-function device (B) to the PC (A) (B → A), the priority of the “scan transfer function” is “first place” and the priority of the “print function” is It is "second place".

  The information on the cooperation function identified as described above and the information indicating the priority are transmitted from the server 14 to the terminal device 16. The control unit 48 of the terminal device 16 causes the UI unit 46 to display the information on the cooperation function as the information on the cooperation function candidate according to the priority.

  For example, as shown in FIG. 10B, the control unit 48 of the terminal device 16 causes the UI unit 46 to display the cooperation function display screen 70, and causes the cooperation function display screen 70 to display information on cooperation function candidates. Since the priority of "scan transfer function" is "first place" and the priority of "print function" is "second place", the information on "scan transfer function" has priority over the information on "print function" (For example, in the upper rank) is displayed. For example, as the information related to the “scan transfer function”, an explanatory note of “scan transfer function” of “transfer data scanned by the MFP (B) to the PC (A)” is displayed. Further, as the information regarding the "printing function", an explanatory text of "printing function" of "printing data in the PC (A)" is displayed.

  When the collaboration function is designated by the user and given an execution instruction, the designated collaboration function is executed. For example, when the "YES" button is pressed by the user, a cooperation function tied to the "YES" button is executed. Further, a "return" button is displayed on the cooperation function display screen 70, and when the user presses the "return" button, the device connection process is canceled.

  The identification processing of the cooperation function and the identification processing of the priority may be performed by the terminal device 16.

  As an operation other than tracing between device images with an operator, a device to be cooperated may be designated by a drawing operation such as putting a circle, and a connection order may be designated. For example, the order of drawing operations corresponds to the connection order. As another example, the device to be cooperated and the connection order may be designated according to the user's voice instruction.

  An example of another operation is shown in FIG. For example, as shown in FIG. 11A, the user touches the device image 66 on the device display screen 62 using the operation element, and operates the device image 64 in the direction indicated by the arrow 72. And the device image 64 are connected. As a result, the terminal device 12 (PC (A)) linked to the device image 66 and the image forming apparatus 10 (multifunction machine (B) linked to the device image 64 are designated as devices to be cooperated, and The order is specified. In this example, since the image is connected from the device image 66 to the device image 64, the device is connected from the terminal device 12 (PC (A)) to the image forming apparatus 10 (multifunctional peripheral (B)). Referring to the cooperation function management table shown in FIG. 7, the priority of the “print function” is “first place”, and the priority of the “scan transfer function” is “second place”. In this case, as shown in FIG. 11B, on the cooperation function display screen 74, the information on the "print function" is displayed preferentially (for example, above) to the information on the "scan transfer function".

  As described above, according to the first embodiment, by linking the device images linked to the device, a cooperation function that uses the function of the device is identified. Further, the display order of the information related to the cooperative function is changed according to the order of connecting the images, that is, the order of connecting the devices. The connection order of the devices is the same as the order of the functions used by each device and the movement order of the data moving between the linked devices, and the operation of connecting the devices (that is, the operation of connecting the images) is the order of the functions And the operation to specify the data transfer order. Therefore, by changing the display order of the information on the cooperation function according to the connection order, the information on the cooperation function predicted to be used by the user is preferentially displayed. That is, the information regarding the cooperation function highly likely to be used by the user is preferentially displayed. For example, when an image is connected from the multi-function device (B) to the PC (A), “Use the function of the multi-function device (B) before the PC (A), and the multi-function device (B) to PC (A) It is predicted that the cooperation function of “transfer data to” is used by the user. Also, when an image is connected from the PC (A) to the multi-function device (B), “Use the function of the PC (A) before the multi-function device (B) and use the PC (A) to multi-function device (B) It is predicted that the cooperation function of “transfer data to” is used by the user. Therefore, by changing the display order of the information on the cooperation function according to the order of connecting the images, the information on the cooperation function highly likely to be used by the user is preferentially displayed. In addition, the order of functions to be used and the order of movement of data are specified without performing a special operation other than the operation of connecting device images, and information on a cooperation function predicted to be used by the user is displayed. .

  Hereinafter, modifications of the first embodiment will be described.

(Modification 1)
Hereinafter, the first modification will be described. In the first modification, the function of the device assigned to the cooperative function is different according to the position in the device image linked to the device. When a specific position is designated by the user in the device image, information on a cooperation function using a function corresponding to the specific position is preferentially displayed. Hereinafter, the first modification will be described in detail.

  FIG. 12 illustrates an example of the device function management table according to the first modification. Data of the device function management table according to the first modification is stored in the server 14 as the device function information 32. In the device function management table according to the first modification, as an example, the device ID, information indicating the device name (for example, type of device), information indicating the position in the image, and the function corresponding to the position in the image Information (function identification information) and an image ID are associated with each other. The position in the image is a specific position (specific part) in the device image linked to the device, and for example, a specific position in the device image schematically representing the device or in the device image captured by the camera It is a specific position. Different functions are associated with specific positions in the image.

  An example of the screen displayed on the UI unit 46 of the terminal device 16 is shown in FIG. For example, it is assumed that the image forming apparatus 10 (multifunctional peripheral (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated with each other. As shown in FIG. 13A, the device display screen 62 is displayed on the UI unit 46 of the terminal device 16, and the device images 64 and 66 are displayed on the device display screen 62. For example, “print function” is associated with a specific position (partial image 76) corresponding to the main body portion of the multifunction machine (B) in the device image 64. A “scan function” is associated with a specific position (partial image 78) corresponding to the document cover or document glass of the multifunction machine (B) or the automatic document feeder in the device image 64. The “stapling function” is associated with a specific position (partial image 80) corresponding to the post-processing device in the device image 64. The "stapling function" is a function of stapling the output sheet. In addition, a “data storage function” is associated with a specific position (partial image 82) corresponding to the main body of the PC (A) in the device image 66. The “screen display function” is associated with the specific position (partial image 84) corresponding to the display unit of the PC (A) in the device image 66. The "data storage function" is a function for storing data sent from another device in the PC (A). The “screen display function” is a function to display data sent from another device on the PC (A).

  The control unit 48 of the terminal device 16 may cause the device display screen 62 to display the name (for example, printing, scanning, etc.) of the function associated with the specific position in the device image. This provides the user with easy-to-understand information on which function corresponds to a specific position. Of course, the name of the function may not be displayed.

  In the first modification, when the position where the function is associated in the device image is designated by the user, the function associated with the designated position is designated as the function to be cooperated. The user uses the indicator to connect a specific position (partial image) to which the function is associated in the device image representing the device to be cooperated. For example, as indicated by the arrow 86, the user touches the partial image 78 on the device display screen 62 using the operator to operate the partial image 84, thereby connecting the partial image 78 and the partial image 84. As a result, the multifunction machine (B) linked to the device image 64 including the partial image 78 and the PC (A) linked to the device image 66 including the partial image 84 are designated as devices to be cooperated with each other. The "scan function" corresponding to the "screen display function" corresponding to the partial image 84 is specified, and the order of connection is further specified. The order of connecting the partial images corresponds to the order of connection. In the example shown in FIG. 13A, since the image is connected from the partial image 78 to the partial image 84, a device is connected from the multi-functional peripheral (B) to the PC (A). In addition, "scan function" and "screen display function" are specified as functions to be used for the cooperation function. Information indicating the connection order of the devices and information indicating a specific position designated by the user in the device image are transmitted from the terminal device 16 to the server 14.

  When devices to be cooperated (for example, PC (A) and multifunction peripheral (B)) are identified, the identifying unit 38 of the server 14 combines with PC (A) in the cooperation function management table shown in FIG. 7. Identify the collaboration function that is realized by linking the machine (B). Further, the specifying unit 38 specifies the function associated with the position specified by the user in the device image by referring to the device function management table shown in FIG. 12. Then, the specifying unit 38 uses the function associated with the position designated by the user in the cooperation function group realized by causing the PC (A) and the multifunction machine (B) to cooperate with each other. Raise the priority of the function and lower the priority of the linked function that does not use that function.

  The information on the cooperation function identified as described above and the information indicating the priority are transmitted from the server 14 to the terminal device 16. The control unit 48 of the terminal device 16 causes the UI unit 46 to display the information on the cooperation function as the information on the cooperation function candidate according to the priority.

  For example, as shown in FIG. 13B, the control unit 48 of the terminal device 16 causes the display unit of the UI unit 46 to display the cooperation function display screen 88, and the cooperation function display screen 88 displays information on cooperation function candidates. Let Since the "scan function" and the "screen display function" are designated by the user in that order, the information on the "scan transfer display function" linked function executed by linking the "scan function" and the "image display function" Is displayed preferentially (for example, at a higher rank) than information on other cooperation functions. For example, information related to the “scan transfer display function” is displayed with priority over information related to the linked function “scan transfer storage function” executed by linking the “scan function” and the “data storage function”. The scan transfer display function is a function of transferring data generated by scanning by the multi-functional peripheral (B) to the PC (A) and displaying it on the screen of the PC (A). The scan transfer storage function is a function of transferring data generated by scanning by the multi-functional peripheral (B) to the PC (A) and storing the data in the PC (A). In the example illustrated in FIG. 13B, a descriptive text of each cooperation function is displayed as information on each cooperation function.

  According to the first modification, when the device to be cooperated has a plurality of functions, the function is individually designated, and information on the cooperation function using the designated function is preferentially displayed. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

  Another example of the first modification is shown in FIG. FIG. 14 is a view showing an example of a screen displayed on the UI unit 46 of the terminal device 16. The control unit 48 of the terminal device 16 superimposes and displays an identifiable touch area image at a specific position (partial image) to which a specific function is assigned in the functional image. In the example shown in FIG. 14A, the touch area image 90 is displayed superimposed on the partial image 76, the touch area image 92 is displayed superimposed on the partial image 78, and the touch area is displayed on the partial image 80. An image 94 is displayed superimposed. Further, the contact area image 96 is displayed superimposed on the partial image 82, and the contact area image 98 is displayed superimposed on the partial image 84. Each contact area image is, for example, a translucent or opaque image, and has a shape such as a circle, an ellipse, or a rectangle. By displaying the contact area image at the position to which the specific function is assigned, the position is emphasized, so that it is easy for the user to understand to which position the function is assigned. That is, it becomes easy to understand which position is the position that responds as the function used for the cooperation function.

  In addition, the control unit 48 of the terminal device 16 detects the touch area image such that the touch area image corresponding to the function used as the cooperation function or the touch area image designated by the user is distinguished from the other touch area images. The display mode (for example, at least one of color and shape) may be changed. For example, as shown in FIG. 14B, when the partial image 78 and the partial image 84 are connected by the operation shown by the arrow 100, the contact area image 92 corresponding to the partial image 78 and the contact area corresponding to the partial image 84 The color of the image 98 is displayed in a color different from the other contact area images.

  FIG. 15 shows still another example of the first modification. FIG. 15 is a view showing an example of a screen displayed on the UI unit 46 of the terminal device 16. When the identified device has a plurality of functions, by designating a plurality of functions possessed by the same device, a collaboration function to be executed by linking the plurality of functions is designated. For example, as illustrated in FIG. 15A, it is assumed that the user connects images in the directions indicated by the arrows 102, 104, and 106. In this case, the partial images 78, 84, 82, 76 are connected in that order. The specifying unit 38 of the server 14 specifies the functions (scan function, screen display function, data storage function, and print function) assigned to the partial images 78, 84, 82, 76, and each function in accordance with the order of connection Identify the collaboration function that uses the and prioritize the collaboration function. Thereby, the information on the cooperation function is displayed with priority over the information on other cooperation functions. The linked function display screen 108 is shown in FIG. 15 (b), and the linked function display screen 108 preferentially displays information on the linked function. For example, “a data transfer generated by scanning with the MFP (B) is transferred to the PC (A) and displayed on the screen of the PC (A), and the color adjustment is performed by the PC (A). After that, the data is stored in the PC (A) and transferred to the multi-function device (B) for printing on the multi-function device (B).

(Modification 2)
Hereinafter, the second modification will be described with reference to FIGS. 16 and 17. An example of the screen displayed on the terminal device 16 is shown in FIG. 16 and FIG. In the second modification, when the device image is continuously pressed for a predetermined time or more, the mode is shifted to a multiple function selection mode (a mode for specifying a plurality of functions).

  Also in the second modification, the image forming apparatus 10 (the multifunction machine (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated with each other, as shown in FIG. In the device 16, device images 64 and 66 are displayed on the device display screen 62. For example, it is assumed that the user continues to designate the device image 64 for a predetermined time or more using the operation elements. That is, it is assumed that the device image 64 has been long pressed. In the example shown in FIG. 16A, the portion indicated by reference numeral 110 is long pressed. In this case, as shown in FIG. 16B, the function selection screen 112 is displayed on the UI unit 46 of the terminal device 16, and the function selection screen 112 displays a list of functions possessed by the multifunction peripheral (B). . In the example shown in FIG. 16B, the names of the functions (scan, print, stapler stop, fax transmission) are displayed. For example, when the device image 64 is designated by the user, the device identification information of the multifunction peripheral (B) linked to the device image 64 is transmitted from the terminal device 16 to the server 14. In the server 14, the function possessed by the multifunction machine (B) is specified by referring to the device function management table. Information on the function (for example, device identification information) is transmitted from the server 14 to the terminal device 16 and displayed. Of course, the terminal device 16 may specify the function of the multifunction peripheral (B). In addition, when the user interface of the terminal device 16 can detect the contact force at the time of a user's operation in multiple steps and detect it, it detects that the contact force became strong (for example, detects the contact force set beforehand) Function selection screen 112 may be displayed.

  In the function selection screen 112 shown in FIG. 16B, the user selects a plurality of functions used for the cooperation function. For example, it is assumed that (1) scan function and (4) fax transmission function are selected. In this case, as shown in FIG. 16C, the confirmation screen 114 is displayed on the UI unit 46 of the terminal device 16. The confirmation screen 114 displays information indicating the function selected by the user. Then, when the user designates the order of cooperation (order of use) in the confirmation screen 114, another confirmation screen 116 is displayed on the UI unit 46 of the terminal device 16, as shown in FIG. 17A. . In the example shown in FIG. 17A, (1) the order of the scan function is "No. 1", and (4) the order of the FAX transmission function is "No. 2". In the function selection screen 112 shown in FIG. 16B, the user may select a plurality of functions used for the cooperation function in accordance with the order of cooperation. In this case, the display of the confirmation screen 114 illustrated in FIG. 16C is omitted, and the confirmation screen 116 illustrated in FIG. 17A is displayed on the UI unit 46 of the terminal device 16.

  Next, in the confirmation screen 116 shown in FIG. 17A, when the user presses the “YES” button, the function of the multifunction peripheral (B) used for the cooperation function is determined, and FIG. As shown in), the screen transitions to the device display screen 62. On the device display screen 62, as indicated by an arrow 118, the user connects the device image 64 and the device image 66. As a result, the PC (A) is specified as a device to be used for the cooperation function, and the order of cooperation (order of use) is specified. The specifying unit 38 of the server 14 specifies a cooperation function that uses each function in accordance with the order of cooperation, and raises the priority of the cooperation function. Thereby, the information on the cooperation function is displayed with priority over the information on other cooperation functions. The information regarding the cooperation function is transmitted from the server 14 to the terminal device 16 and displayed. For example, as shown in FIG. 17C, the cooperation function display screen 120 is displayed on the UI unit 46 of the terminal device 16, and information on the cooperation function is displayed on the cooperation function display screen 120. Information on cooperation functions that use each function according to the order of cooperation is preferentially displayed. The preferential cooperation function is, for example, a function “fax-transmits an image generated by scanning by the multifunction machine (B) and transfers a copy of the fax-transmitted data to the PC (A)”.

  In the confirmation screen 116 shown in FIG. 17A, when the user presses the “NO” button, the selection of the function of the multifunction machine (B) used for the cooperation function is cancelled, and the screen is a function It returns to the selection screen 112. Further, a "cancel" button is displayed on the function selection screen 112, and when the "cancel" button is pressed by the user, the screen returns to the device display screen 62.

  According to the second modification, since a list of functions possessed by the specified device is displayed, it is easy for the user to understand what kind of function the device has.

  Another example of the modification 2 is shown in FIG. In the second modification, the device display screen 62 displays a button image 122 for selecting a plurality of functions. When the user presses the button image 122 while pressing the functional image, the mode shifts to the multiple function selection mode. In the example shown in FIG. 18, as indicated by reference numeral 124, the button image 122 is pressed in a state where the device image 64 linked to the multi-functional peripheral (B) is pressed by the user. In this case, the screen changes to the function selection screen 112 shown in FIG. Thereafter, the processing described with reference to FIGS. 16 and 17 is performed.

(Modification 3)
Hereinafter, the third modification will be described with reference to FIG. An example of a screen displayed on the UI unit 46 of the terminal device 16 is shown in FIG.

  Also in the third modification, the image forming apparatus 10 (the multifunction machine (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated, as shown in FIG. 19A, the terminal In the device 16, device images 64 and 66 are displayed on the device display screen 62. Further, as shown by arrows 126, 128 and 130, partial images 78, 84, 82 and 76 are connected in that order. That is, the scan function, the screen display function, the data storage function, and the print function are designated in that order.

  In the third modification, when the image connected one before the cancellation target image is designated, the function associated with the cancellation target image is canceled from the cooperation function. That is, the function to be canceled is excluded from the function to be cooperated. For example, when an image linked to one before the image to be canceled is designated, information indicating the designation is transmitted from the terminal device 16 to the server 14, and the identifying unit 38 of the server 14 Exclude linked functions from linked functions.

  In the example shown in FIG. 19A, as shown by the broken arrow 132, an image is connected from the partial image 76 to the partial image 82. That is, the partial image 82 connected to the front of the partial image 76 is designated. In this case, the function (print function) linked to the partial image 76 is excluded from the functions to be linked.

  Thereafter, as shown in FIG. 19B, when the image is connected from the partial image 82 to the partial image 80 by the user in the direction indicated by the arrow 134, the function (stapler stop function) linked to the partial image 80 has The cooperation function which is selected as the cooperation target function and uses the function associated with the partial image 80 is preferentially selected.

  According to the third modification, the function to be canceled is excluded from the function to be cooperated without performing a special operation other than the operation to connect the images, and the cooperation function to use the function to be canceled is cancelled. For example, if a function to be linked is specified incorrectly, by specifying an image linked one before the image linked to that function, it is specified incorrectly without performing any other special operation. The function is released. As a result, the function to be canceled is excluded with a simple operation as compared with the case where the function is excluded from the functions to be cooperated using the release button or the like.

  The trajectory of the release operation (for example, the trajectory of the arrow 132) may be displayed on the device display screen 62. In this case, the trajectory of the release operation may be displayed in a display mode (for example, different colors or different types of lines) different from the trajectory of the connection operation (for example, the trajectory of the arrows 126, 128, 130).

  As another example, when releasing the cooperation, the user may designate on the device display screen 62 the device image or the partial image linked to the device to be released, or may push the cooperation release button. Also, when a device that is not a collaboration target is reflected on the device display screen 62, the user designates the device image linked to the device on the device display screen 62, thereby excluding the device from the devices of the collaboration target. May be The device may be excluded from the devices to be cooperated when a preset operation is performed such as putting a cross mark on the device image linked to the device to be canceled.

(Modification 4)
Hereinafter, the fourth modification will be described with reference to FIG. An example of a screen displayed on the UI unit 46 of the terminal device 16 is shown in FIG.

  Also in the fourth modification, the image forming apparatus 10 (the multi-function device (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated, and as shown in FIG. In the device 16, device images 64 and 66 are displayed on the device display screen 62. Further, as shown by an arrow 136, partial images 78 and 84 (contact area images 92 and 98) are assumed to be connected in that order. That is, the scan function and the screen display function are designated in that order.

  In the fourth modification, the control unit 48 of the terminal device 16 changes the imaging area of the camera 42 and displays it on the device display screen 62 while displaying the device image representing the device that cooperated last on the device display screen 62. Display. In this case, the identification information of the devices (for example, the multifunction machine (B) and the PC (A)) that have already cooperated with each other is stored in the storage unit of the terminal device 16 or the server 14.

  In the example shown in FIG. 20A, since the terminal device 12 (PC (A)) is the last device to cooperate with, the device image 66 remains on the device display screen 62 and is displayed. When the user changes the shooting area of the camera 42 of the terminal device 16, the control unit 48 of the terminal device 16 displays the image captured by the camera 42 in a state where the device image 66 is displayed on the device display screen 62. It is displayed on the screen 62. For example, when using the projector (C) which can not be photographed in the environment shown in FIG. 20A as the device to be cooperated, the device image 66 is displayed on the device display screen 62 as shown in FIG. 20B. In this state, the user shoots the projector (C) by changing the direction of the camera 42 so that the projector (C) is included in the shooting area of the camera 42. Thus, the projector (C) is photographed by the camera 42, and the device image 138 linked to the projector (C) is displayed on the device display screen 62. As described above, the projector (C) is identified, and, for example, the name (for example, “project”) of the function of the projector (C) is displayed on the device display screen 62. Further, the contact area image 140 is displayed superimposed on the device image 138 representing the projector (C). When an image is connected from the partial image 84 (contact area image 98) to the apparatus image 138 (contact area image 140) by the user as indicated by the arrow 142, the function (for example, projection) of the apparatus linked to the apparatus image 138 A function is selected as the function to be cooperated, and the cooperation function using the function linked to the device image 138 is displayed preferentially.

  According to the fourth modification, even when the plurality of devices to be cooperated can not be photographed at one time, the plurality of devices can be photographed separately and cooperated. That is, the devices to be cooperated are not necessarily arranged at positions close to each other. According to the fourth modification, even in such a case, a plurality of devices can be separately photographed and cooperated. Of course, it is also possible to respond by changing the angle of the shooting area of the camera 42 or enlarging or reducing the shooting area, but if such an operation can not cope with the situation, the shooting may be divided into multiple times. Each device is identified.

  In the above example, the device image linked to the device at the end of cooperation is displayed on the device display screen 62, but as another example, the device image designated by the user or the first image taken A device image linked to a device or a device image linked to a preset basic device may be displayed on the device display screen 62 and displayed. Also in this case, it becomes possible to capture a plurality of devices separately and to make them cooperate.

  In the first embodiment and the modified example 1-4 described above, an apparatus image representing the identified apparatus, a contact area image, a trajectory of an operation for connection (for example, a trajectory of a traced line), and an operation for releasing The trajectory may be displayed in three dimensions so as not to be buried in the background image. That is, those images may be displayed as a three-dimensional image. For example, the background image is displayed in two dimensions, and the device image, the touch area image, and the trajectory of the operation are displayed in three dimensions. This improves the visibility of the device image, the contact area image, and the trajectory of the operation. As another example, the device image or the background image linked to an unidentified device may not be displayed on the device display screen 62, and only the device image linked to the identified device may be displayed on the device display screen 62.

  As described above, when the image linked to the device is connected and the cooperation function is specified, a connection request is transmitted from the terminal device 16 to the device to be cooperated, and the terminal device 16 and the device to be cooperated are connected Be done. The connection process will be described below with reference to FIG. FIG. 21 is a sequence diagram showing the process.

  First, the user uses the terminal device 16 to instruct activation of an application (program) for executing the cooperation function. The control unit 48 of the terminal device 16 activates the application according to the instruction (S01). The application may be stored in advance in the storage unit 44 of the terminal device 16 or may be downloaded from the server 14 or the like.

  Next, by applying the marker type AR technology, the markerless type AR technology, or the position information AR technology, the device to be cooperated is identified (S02). Of course, devices to be cooperated may be identified by techniques other than AR technology. When the marker type AR technology or the markerless type AR technology is applied, the user shoots a device to be cooperated with the camera 42 of the terminal device 16. As an example, when the image forming apparatus 10 (multifunctional peripheral (B)) and the terminal device 12 (PC (A)) are used as linked devices, the user captures an image of the image forming apparatus 10 and the terminal device 12 with the camera 42. The device identification information of each of the image forming device 10 and the terminal device 12 is acquired, and the image forming device 10 and the terminal device 12 are identified as devices to be cooperated with each other. When the position information AR technology is applied, position information of each of the image forming apparatus 10 and the terminal device 12 is acquired, and device identification information of each of the image forming apparatus 10 and the terminal device 12 is specified based on the position information. , The image forming apparatus 10 and the terminal device 12 are identified.

  In the terminal device 16, when the user mutually connects a plurality of device images linked to a plurality of devices to be cooperated, the cooperation function is specified, and the priority of each cooperation function is specified. Information on the cooperation function is displayed on the UI unit 46 of the terminal device 16 according to the priority (S03). The specifying process of the cooperation function and the specifying process of the priority may be performed by the server 14 or may be performed by the terminal device 16.

  Next, in the terminal device 16, when the cooperation function to be executed is designated by the user, the terminal device 16 requests connection to the devices to be cooperated (for example, the image forming apparatus 10 and the terminal device 12) that executes the cooperation function. Is transmitted (S04). For example, when the address information indicating the address of the device to be cooperated is stored in the server 14, the terminal device 16 acquires the address information of the device to be cooperated from the server 14. When the device identification information includes the address information, the terminal device 16 may obtain the respective address information from the device identification information of the device to be cooperated. As another example, the terminal device 16 may store address information of a device to be cooperated. Of course, the terminal device 16 may acquire the address information of the collaboration target device by another method. The terminal device 16 transmits information indicating a connection request to the devices to be cooperated (eg, the image forming device 10 and the terminal device 12) using the address information of the devices to be cooperated (eg, the image forming device 10 and the terminal device 12) Do.

  The image forming apparatus 10 and the terminal device 12 having received the information indicating the connection request permit or do not permit connection with the terminal device 16 (S05). For example, if the image forming apparatus 10 and the terminal apparatus 12 correspond to an apparatus for which connection is not permitted or if the number of apparatuses requesting connection exceeds an upper limit, the connection is not permitted. . When the connection from the terminal device 16 is permitted, the change operation may be prohibited so that the setting information unique to the image forming apparatus 10 and the terminal device 16 is not changed from the terminal device 16. For example, the change of the color parameter of the image forming apparatus 10, the setting time when shifting to the power saving, or the like may be prohibited. This improves the security of the device to be linked. As another example, when devices are linked, the change in setting information may be restricted as compared to the case where the devices are used alone without linking. For example, as compared with the case of using the device alone, it may be permitted to change less setting items. Further, browsing of personal information of other users such as operation history may be prohibited. This improves the security of the user's personal information.

  Result information indicating permission or non-permission of connection is transmitted from the image forming apparatus 10 and the terminal device 12 to the terminal device 16 (S06). When the connection to the image forming apparatus 10 and the terminal device 12 is permitted, communication is established between the terminal device 16 and the image forming apparatus 10 and the terminal device 12.

  Next, the user uses the terminal device 16 to instruct execution of the cooperation function (S07). In response to this instruction, execution instruction information indicating a cooperative function execution instruction is transmitted from the terminal device 16 to the image forming apparatus 10 and the terminal device 12 (S08). The execution instruction information transmitted to the image forming apparatus 10 includes information (for example, job information) indicating a process to be executed by the image forming apparatus 10, and the execution instruction information transmitted to the terminal device 12 includes a terminal The information (for example, job information) indicating the process to be executed by the device 12 is included.

  The image forming apparatus 10 and the terminal device 12 having received the execution instruction information execute the function according to the execution instruction information (S09). For example, as in the scan transfer function of transferring scan data from the image forming apparatus 10 (multifunctional peripheral (B)) to the terminal apparatus 12 (PC (A)), the cooperation function includes the image forming apparatus 10 and the terminal apparatus 12 When the process of transmitting and receiving data is included, communication is established between the image forming apparatus 10 and the terminal device 12. In this case, for example, the execution instruction information transmitted to the image forming apparatus 10 includes the address information of the terminal apparatus 12, and the execution instruction information transmitted to the terminal apparatus 12 includes the address information of the image forming apparatus 10. Communication is established between the image forming apparatus 10 and the terminal device 12 using the address information.

  When the execution of the cooperation function ends, information indicating the completion of the execution of the cooperation function is transmitted from the image forming apparatus 10 and the terminal device 12 to the terminal device 16 (S10). Information indicating that the execution of the cooperation function has been completed is displayed on the UI unit 46 of the terminal device 16 (S11). If the information indicating the completion of execution is not displayed even if the time set in advance has passed since the time of giving the execution instruction, the control unit 48 of the terminal device 16 causes the UI unit 46 to display the information indicating an error, The execution instruction information or the information indicating the connection request may be transmitted to the image forming apparatus 10 and the terminal device 12 again.

  Next, the user confirms whether or not to cancel the cooperation state of the image forming apparatus 10 and the terminal device 12 (S12), and a process according to the presence or absence of the cancellation is executed (S13). When canceling the cooperation state, the user uses the terminal device 16 to give a cancellation instruction. Thereby, the communication between the terminal device 16 and the image forming apparatus 10 and the terminal device 12 is released. Similarly, the communication between the image forming apparatus 10 and the terminal device 12 is released. If the cooperation state is not canceled, an execution instruction may be continuously given.

  Moreover, you may increase the apparatus of cooperation object. For example, the device identification information of the third device may be acquired, and the cooperation function to be executed by causing the three devices including the image forming apparatus 10 and the terminal device 12 to cooperate may be specified. The information indicating that the image forming apparatus 10 and the terminal device 12 have already been identified as the cooperation function is stored in the server 14 or the terminal device 16.

  The device identification information indicating the device to be cooperated and the cooperation function identification information indicating the executed cooperation function may be stored in the terminal device 16 or the server 14 or the like. For example, user account information (user identification information) of a user who uses the terminal device 16 is acquired, and the user account information, device identification information indicating a device to be cooperated, and cooperation function identification information indicating an executed cooperation function The historical information which shows the correspondence of and may be created, and may be memorized by terminal unit 16 or server 14 grade. The history information may be created by the terminal device 16 or may be created by the server 14. By referring to the history information, it is specified what kind of cooperation function has been executed using what kind of device group.

  The devices to be cooperated (for example, the image forming apparatus 10 and the terminal device 12) store, as history information, user account information of the user who requested connection, and terminal identification information indicating the terminal device 16 of the connection request source. It is also good. By referring to this history information, the user who has used the device is identified. For example, when identifying a user who has used the device when the device is broken, or when performing a charging process for consumables or the like, the user may be identified using history information. The history information may be stored in the server 14 or the terminal device 16 or may be stored in another device.

  The user account information is stored in advance in, for example, the storage unit 44 of the terminal device 16, and the control unit 48 of the terminal device 16 functions as an example of a user identification unit, and reads user account information of the user from the storage unit 44. , Identify the user who uses the terminal device 16. When user account information of a plurality of users is stored in the storage unit 44, the user uses the terminal device 16 to specify his own user account information. Thereby, the user account information of the user is read, and the user is identified. As another example, the control unit 48 of the terminal device 16 may identify the user by reading user account information of the user logged in to the terminal device 16. As still another example, when only one piece of user account information is stored in the same terminal device 16, the control unit 48 of the terminal device 16 may identify the user by reading the user account information. . When the user account is not set and the user account information is not created, the initial setting is performed, whereby the user account information is created.

  The use history of the collaboration function may be managed for each user, and information indicating the collaboration function used in the past by the user indicated by the read user account information may be displayed on the UI unit 46 of the terminal device 16. The information indicating the usage history may be stored in the terminal device 16 or may be stored in the server 14. In addition, information indicating a collaboration function being used may be displayed at a frequency higher than a preset usage frequency. By providing such a shortcut function, the time and effort of the user's operation regarding the cooperation function can be reduced.

Second Embodiment
Hereinafter, an image forming system as an information processing system according to the second embodiment of the present invention will be described. FIG. 22 shows a server 14A according to the second embodiment. The image forming system according to the second embodiment includes a server 14A instead of the server 14 according to the first embodiment. The configuration other than the server 14A is the same as the configuration of the image forming system according to the first embodiment shown in FIG.

  Similar to the server 14 according to the first embodiment, the server 14A manages a cooperation function that is executed by causing a plurality of functions to cooperate. Hereinafter, the configuration of the server 14A will be described in detail with reference to FIG.

  The server 14A includes a communication unit 28, a storage unit 30A, and a control unit 36A. The communication unit 28 has the same configuration as the communication unit 28 according to the first embodiment.

  The storage unit 30A is a storage device such as a hard disk or an SSD. The storage unit 30A stores, for example, device function information 32, cooperation function information 144, various data, various programs, device address information indicating the address of each device, and the like. Of course, they may be stored in separate storage devices, or may be stored in one storage device. The device function information 32 and the cooperation function information 144 stored in the storage unit 30A are provided to the terminal device 16 periodically or at a designated timing, whereby the information stored in the terminal device 16 is updated. May be

  The device function information 32 is the same information as the device function information 32 according to the first embodiment.

  The cooperation function information 144 is information indicating a cooperation function executed by causing a plurality of functions to cooperate, and indicates, for example, a combination of function identification information for identifying a function to be cooperated in order to execute the cooperation function. This is information indicating correspondence between information and cooperation function identification information for identifying the cooperation function. In the case of linking a plurality of devices, the linking function information 144 is information indicating a linking function to be executed by linking a plurality of functions possessed by the plurality of devices, and the devices linked to execute the linking function are selected. This is information indicating the correspondence between information indicating a combination of device identification information for identification and cooperative function identification information.

  The control unit 36A controls the operation of each unit of the server 14A. Further, the control unit 36A includes a specifying unit 38 and a cooperation function registration unit 146.

  The identifying unit 38 identifies the device based on the device identification information and identifies the function of the device, as the identifying unit 38 according to the first embodiment. Further, the specifying unit 38 specifies a cooperation function to be executed by causing the function of the device to be cooperated to cooperate. Also in the second embodiment, by applying the AR technology, the device identification information is acquired and the device is identified. Of course, the device identification information may be acquired by a technique other than the AR technique to identify the device.

  In the second embodiment, the device image linked to the device to be cooperated is displayed on the UI unit 46 of the terminal device 16, the cooperation image is specified by superimposing the device images, and the information on the cooperation function is the terminal It is displayed on the UI unit 46 of the device 16.

  The cooperation function registration unit 146 has a function of registering the cooperation function, and, for example, information indicating the registration content of the cooperation function to be registered and an image displayed when executing the cooperation function (for example, an image for a shortcut) ) And correspond. The image is, for example, an image generated by superimposing device images or an image schematically representing a device to be superimposed.

  Hereinafter, the image forming system according to the second embodiment will be described in detail.

  First, the cooperation function information 144 will be described in detail with reference to FIG. An example of the cooperation function management table as the cooperation function information 144 is shown in FIG. In the linked function management table, as an example, information indicating a combination of device IDs, information indicating a linked device name (for example, type of each device), and information indicating a linked function (linked function identification information) It is associated. Unlike the cooperation function management table according to the first embodiment shown in FIG. 7, the cooperation function management table according to the second embodiment does not include information indicating the connection order and information indicating the priority. .

  Hereinafter, with reference to FIG. 24 and FIG. 25, an operation for specifying the cooperation function will be described in detail. An example of a screen displayed on the UI unit 46 of the terminal device 16 is shown in FIGS. 24 and 25.

  For example, as in the first embodiment, the image forming apparatus 10 (multifunctional peripheral (B)) and the terminal device 12 (PC (A)) are identified as cooperation target devices by applying the UR technology or the like. It shall be. As shown in FIG. 24A, the device display screen 62 is displayed on the UI unit 46 of the terminal device 16, and the identified devices (image forming apparatus 10 and terminal device 12) are displayed on the device display screen 62. The device image 64, 66 linked to the is displayed. In this state, the user uses an indicator (for example, a user's finger, a pen, a stylus, or the like) to superimpose the device image linked to the device to be cooperated with the device image linked to another device to be cooperated. For example, as shown in FIG. 24B, the user designates the device image 64 using the operation element, and superimposes the device image 64 on the device image 66 as shown by the arrow 148. For example, device images are superimposed by a drag and drop operation. That is, the user drags the device image 64 and drops the device image 64 at a position overlapping the device image 66. This drag and drop operation itself is, for example, a known technique. A device image to be superimposed (an image linked to a device to be linked) may be designated according to the user's voice instruction. For example, the device images 64 and 66 may be designated as device images to be superimposed according to the user's voice instruction, and they may be superimposed.

  The image forming apparatus 10 (the multifunction machine (B)) linked to the device image 64 and the terminal device 12 (PC (A)) linked to the device image 66 are linked with each other by overlapping the device images 64 and 66 with each other. Designated as a device. Although the device image 64 is superimposed on the device image 66 in the example shown in FIG. 24B, the device image 66 may be superimposed on the device image 64.

  The control unit 48 of the terminal device 16 may cause the UI unit 46 to display the device image during the drag operation in a distinguishable display mode. For example, the device image during the drag operation may be displayed semitransparently or in a specific color.

  When the device image 64 is displayed superimposed on the device image 66, a confirmation screen 150 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. 24C. The confirmation screen 150 is a screen for confirming whether the device group designated as the device to be cooperated is to be cooperated. In the confirmation screen 150, when the collaboration instruction is given by the user (for example, when the “YES” button is pressed by the user), information on the collaboration function is displayed on the UI unit 46 of the terminal device 16. Specifically, when a device to be cooperated (for example, the image forming apparatus 10 and the terminal device 12) is specified and a cooperation instruction is given, the specifying unit 38 of the server 14A performs the cooperation function management table shown in FIG. In (collaboration function information 144), the cooperation function associated with the combination of the PC (A) and the multifunction machine (B) is specified. Thus, the cooperation function to be executed by causing the PC (A) and the multifunction machine (B) to cooperate is specified. The information on the cooperation function specified in this manner is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16 as the information on cooperation function candidates.

  For example, as shown in FIG. 25A, the control unit 48 of the terminal device 16 causes the UI unit 46 to display the cooperation function display screen 152, and causes the cooperation function display screen 152 to display information on cooperation function candidates. Since the “scan transfer function” and the “print function” are realized, for example, by linking the PC (A) and the multi-functional peripheral (B), the cooperative function display screen 152 displays information on the “scan transfer function” and the like. Information on "Print Function" is displayed.

  When the cooperation function is designated by the user and an execution instruction is given, a connection request is issued from the terminal device 16 to the device to be cooperated. As shown in FIG. 25B, a standby screen 154 is displayed on the UI unit 46 of the terminal device 16 during the connection request. When the connection between the terminal device 16 and the device to be cooperated is successful, the designated cooperation function is executed.

  As described above, according to the second embodiment, by superimposing the device images linked to the device, the cooperation function that uses the function of the device is identified. Therefore, the functions can be linked without performing any special operation other than the image operation, and the functions can be linked by a simple operation.

  Hereinafter, a modification of the second embodiment will be described.

(Modification 5)
The fifth modification will be described below with reference to FIGS. 26 and 27. 26 and 27 show an example of a screen displayed on the UI unit 46 of the terminal device 16. In the fifth modification, after a plurality of device images are superimposed, a new image showing the superimposed state is generated and displayed. Hereinafter, the fifth modification will be described in detail.

  Also in the fifth modification, the image forming apparatus 10 (the multi-function device (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated, and as shown in FIG. In the device 16, it is assumed that the device images 64 and 66 are displayed on the device display screen 62. For example, it is assumed that the device image 64 is superimposed on the device image 66 as indicated by the arrow 156 using the operator. The image forming apparatus 10 (multifunction machine (B)) linked to the device image 64 and the terminal device 12 (PC (A)) linked to the device image 66 by superimposing the device images 64 and 66 are devices targeted for cooperation. Designated as

  When the device image 64 is displayed superimposed on the device image 66, a confirmation screen 158 is displayed on the UI unit 46 of the terminal device 16, as shown in FIG. 26 (b). The confirmation screen 158 is a screen for confirming whether or not the device group designated as the device to be cooperated is to be cooperated, and a screen for confirming whether or not there is a further device to be cooperated. When the user gives an instruction to cause the multifunction device (B) to cooperate with the PC (A) (for example, when the user presses the “YES” button for causing the multifunction device (B) to cooperate with the PC (A)), The multifunction peripheral (B) and the PC (A) are linked. When there is another device to be linked (for example, when the user presses the “YES” button for linking another device), the device display screen 62 is displayed as shown in FIG. 26C. . On the device display screen 62, a new device image 160 generated by grouping the overlapped device images 64 and 66 is displayed. The new device image 160 may be generated by, for example, the control unit 48 of the terminal device 16 or may be generated by the control unit 36A of the server 14A. The device image 160 is an image representing a state in which the device image 66 is superimposed on the device image 64.

  In the fifth modification, the control unit 48 of the terminal device 16 displays the newly generated device image 160 on the device display screen 62. When the user changes the shooting area of the camera 42 of the terminal device 16, the control unit 48 of the terminal device 16 displays the image captured by the camera 42 in a state where the device image 160 is displayed on the device display screen 62. It is displayed on the screen 62. In this case, the identification information of the devices (for example, the multifunction machine (B) and the PC (A)) that have already cooperated with each other is stored in the storage unit of the terminal device 16 or the server 14A.

  For example, when using a projector (C) which can not be photographed in the environment shown in FIG. 26A as a device to be cooperated, the device image 160 is displayed on the device display screen 62 as shown in FIG. 26C. In this state, the user shoots the projector (C) by changing the direction of the camera 42 so that the projector (C) is included in the shooting area of the camera 42. Thus, the projector (C) is photographed by the camera 42, the device image 138 linked to the projector (C) is displayed on the device display screen 62, and the projector (C) is identified. As indicated by the arrow 162, the user superimposes the new device image 160 on the device image 138, whereby the device linked to the device image 138 is selected as the device to be cooperated. Furthermore, when there is no device to be cooperated, the cooperation function executed by causing the already-specified cooperation target equipment group to cooperate is specified in the server 14, and the information on the cooperation function is transmitted from the server 14 to the terminal device 16. It is displayed on the UI unit 46 of the terminal device 16. For example, in the confirmation screen 158 shown in FIG. 26B, when the user presses the “NO” button for instructing that there is no device to be cooperated, the information on the cooperation function is displayed on the terminal device 16. It is displayed on the UI unit 46.

  As described above, according to the fifth modification, a new device image generated by overlapping device images is superimposed on the device image linked to the next cooperation candidate device, thereby increasing the number of devices to be linked. be able to. Further, since the new device image represents a state in which the device image linked to the device to be linked is superimposed, by displaying the new device image, any device is designated as the device to be linked. It becomes easy to understand visually whether it is.

  Note that the display of the confirmation screen 158 may be omitted. In this case, after the device images are superimposed, when the display instruction of the cooperation function is given by the user, the information on the cooperation function to be executed by the cooperation target device designated by the superposition is the UI of the terminal device 16 It is displayed on the part 46.

  In the fifth modification, a new device image is generated by grouping device images overlapping one another, but the device images overlapping one another are displayed on the device display screen 62 instead of another image. It is also good. For example, as shown in FIG. 27, a new device image 164 is displayed on the device display screen 62 instead of the device image 160 generated by grouping, and the new device image 164 is shown by an arrow 166. It has been manipulated as. The device image 164 is an image schematically representing the multi-function device (B) and the PC (A), an image representing the contents of the functions of the multi-function device (B) and the functions of the PC (A). The device image 164 may be generated by the control unit 48 of the terminal device 16 or may be generated by the control unit 36A of the server 14A. By displaying an image representing the content of the function as a new device image, it becomes easy to visually understand what function is designated as the function to be cooperated.

  In the example shown in FIG. 27, the original device image 160 is displayed by being automatically replaced with a new device image 164, but a confirmation screen is displayed on the UI unit 46 of the terminal device 16 when this replacement is performed. It is also good. An example of the confirmation screen is shown in FIG. After a plurality of functional images are superimposed, a confirmation screen 168 is displayed on the UI unit 46 of the terminal device 16. On the confirmation screen 168, an original device image 160 generated by grouping a plurality of device images and a schematic device image 164 are displayed. When the user instructs to replace the image (for example, when the “YES” button is pressed by the user), the original device image 160 is replaced with a schematic device image 164, as shown in FIG. The device image 164 is displayed on the device display screen 62.

(Modification 6)
Hereinafter, the sixth modification will be described with reference to FIGS. 29 to 31. 29 to 31 show an example of a screen displayed on the UI unit 46 of the terminal device 16. In the sixth modification, the device image generated at the time of the last cooperation is used as an image for a shortcut button when the cooperation function is executed next time or later. Hereinafter, the sixth modification will be described in detail.

  In the sixth modification, as in the fifth modification, the device images 64 and 66 are superimposed, and the image forming apparatus 10 (the multi-function device (B)) and the terminal device 12 (PC (A)) are devices to be cooperated with each other. It is assumed that a new device image 160 is generated by grouping the device images 64 and 66.

  When the user gives an instruction to register a new device image 160 as an image for a shortcut button using the terminal device 16, as shown in FIG. 29A, the cooperation function registration screen is displayed on the UI unit 46 of the terminal device 16. 170 is displayed. On the linked function registration screen 170, a new device image 160 is displayed, and information indicating the registered content of the linked function to be executed by linking the MFP (B) and the PC (A) is displayed. Ru. For example, the cooperation function name, the outline of the cooperation function, and the information indicating the device necessary for cooperation are displayed as the information indicating the registered content. The user may edit the registered content (for example, the cooperation function name, the outline of the cooperation function, etc.) using the terminal device 16.

  In the cooperation function registration screen 170, when the user gives an instruction not to register the shortcut function of the cooperation function (for example, when the “NO” button is pressed by the user), as shown in FIG. A confirmation screen 172 is displayed on the sixteen UI units 46. In the confirmation screen 172, when the end instruction is given by the user (for example, when the “YES” button related to the end of registration is pressed by the user), the cooperation function registration process is ended. When a part of the editing instruction is given by the user (for example, when the "YES" button for partial editing is pressed by the user), the screen returns to the cooperation function registration screen 170.

  On the other hand, when an instruction to register the shortcut function of the cooperation function is given by the user on the cooperation function registration screen 170 (for example, when the “YES” button is pressed by the user), as shown in FIG. A confirmation screen 174 is displayed on the UI unit 46 of the terminal device 16. When an instruction to create a shortcut button is given by the user on the confirmation screen 174 (for example, when the “YES” button is pressed by the user), an image (shortcut button image) representing the shortcut button for executing the cooperation function Is generated. The shortcut button image may be created by the terminal device 16 or may be created by the server 14A. For example, as shown in FIG. 30B, the shortcut button image 178 is displayed on the screen 176 (for example, a desktop screen) of the UI unit 46 of the terminal device 16. The shortcut button image 178 includes the device image 160. For example, the cooperation function registration unit 146 of the server 14A associates the information indicating the registration content of the cooperation function to be registered with the shortcut button image 178. Of course, the association may be performed by the terminal device 16. When a plurality of shortcut button images are generated and registered, a list of shortcut button images is displayed on the screen 176. The information indicating the registered content of the cooperation function and the shortcut button image are stored, for example, in the terminal device 16. As another example, the information indicating the registered content of the cooperation function and the shortcut button image may be stored in the server 14 in association with the user account information. In this case, when the user account information is transmitted from the terminal device 16 to the server 14, the information (the information indicating the registered content of the cooperation function and the shortcut button image) associated with the user account information is transmitted from the server 14 to the terminal It is sent to the device 16 and displayed.

  The shortcut button image 178 includes status images 180 and 182. The status image is an image showing the status of the device to be cooperated, that is, an image showing whether the device to be cooperated is in a state where the cooperation function can be executed. For example, the status image 180 is an image representing the status of the image forming apparatus 10 (the multifunction machine (B)), and the status image 182 is an image representing the status of the terminal device 12 (PC (A)). When the device to be cooperated can execute the cooperation function, the status image is displayed in a specific color (for example, green, blue, etc.) symbolizing that effect, and the device to be cooperated is not in the state to be able to execute the cooperation function The status image is displayed in another specific color (for example, white, red, etc.) symbolizing that. In the example shown in FIG. 30B, the status image 180 is displayed in green and the status image 182 is displayed in white. Therefore, although the image forming apparatus 10 (the multifunction machine (B)) can execute the cooperation function, the terminal apparatus 12 (PC (A)) is not in the state capable of executing the cooperation function. Of course, information other than color may indicate the status of the device to be linked. For example, a status image having a shape corresponding to the status may be displayed, or a character string indicating the status may be displayed.

  The state in which the cooperation function can be executed is, for example, a state not currently being used by another user, a state in which no failure has occurred, or the like. The state in which the cooperation function can not be executed is, for example, a state currently being used by another user, a state in which a failure has occurred, or the like. For example, the terminal device 16 acquires information indicating the status of the cooperation target device from the cooperation target device, and displays the status image in a specific color based on the information. As another example, the server 14 may acquire the information indicating the status from the cooperation target device, and the terminal device 16 may acquire the information from the server 14.

  As described above, by linking and managing the collaboration function and the shortcut button image, the operation for executing the collaboration function is reduced from the next time on.

  FIG. 31 shows an example of the browsing edit screen of the cooperation function. When the user uses the terminal device 16 to give an instruction to browse the registration information of the cooperation function, as shown in FIG. 31A, the browsing editing screen 184 is displayed on the UI unit 46 of the terminal device 16. The browsing editing screen 184 displays a shortcut image as a display image, registered contents of a cooperation function (for example, a cooperation function name, a function outline, a connected device name) and the like. For example, when the user designates an item (for example, a display image, a linked function name, a function summary, a connected device name) about a linked function to be edited on the browsing and editing screen 184, as shown in FIG. A confirmation screen 186 is displayed on the UI unit 46 of the terminal device 16. For example, when the cooperation function name of a certain cooperation function is designated by the user, a confirmation screen 186 for confirming editing of the cooperation function name is displayed. When the user presses the “YES” button on the confirmation screen 186, as shown in FIG. 31C, the editing screen 188 is displayed on the UI unit 46 of the terminal device 16. In the edit screen 188, the linked function name is edited by the user, and the edited screen 188 displays the linked function name before editing and the linked function name after editing. When an instruction to reflect the editing content is given by the user (for example, when the “YES” button is pressed by the user), the editing content is reflected. For example, the linked function name after editing is displayed on the desktop screen of the UI unit 46 of the terminal device 16 together with the shortcut image. Items other than linked function names are edited in the same manner.

(Modification 7)
Hereinafter, the seventh modification will be described with reference to FIG. An example of a screen displayed on the UI unit 46 of the terminal device 16 is shown in FIG. In the seventh modification, after the device image to be superimposed is selected, if the selection is canceled in the area where the device image is not displayed on the device display screen 62, the cooperation function is canceled.

  For example, as shown in FIG. 32A, the device image 66 is superimposed on the device image 64, and a new device image 160 is generated and displayed. At this time, when the user selects the device image 160 and cancels the selection of the device image 160 in the area where the device image is not displayed on the device display screen 62 as shown by the arrow 190, the multifunction device (B) and the PC The cooperation function executed by the cooperation with (A) is cancelled. That is, by dragging the device image 160 to a region where the device image is not displayed and performing a drop operation in that region, the cooperation function is canceled and the device returns to the state before the device cooperation. For example, information indicating the drag operation and the drop operation is transmitted from the terminal device 16 to the server 14, and the specifying unit 38 of the server 14 releases the cooperation function. Of course, the terminal device 16 may release the cooperation function.

  FIG. 32 (b) shows the device display screen 62 after the cooperation function is released. Before the devices are linked, the device images 64, 66 and 138 are displayed separately without being superimposed.

  As described above, according to the seventh modification, since the cooperation function is canceled without performing any other special operation by canceling the selection of the device image to be superimposed, the cancellation of the cooperation function is performed. The cooperation function is easily released as compared to the case where a special operation other than the selection cancellation is required.

  As another example, when releasing the cooperation, the user may designate the device image linked to the release target device on the device display screen 62, or may press the cooperation release button. Also, when a device that is not a collaboration target is reflected on the device display screen 62, the user designates the device image linked to the device on the device display screen 62, thereby excluding the device from the devices of the collaboration target. May be The device may be excluded from the devices to be cooperated when a preset operation is performed such as putting a cross mark on the device image linked to the device to be canceled.

(Modification 8)
The eighth modification will be described below with reference to FIG. An example of a screen displayed on the UI unit 46 of the terminal device 16 is shown in FIG.

  Also in the modified example 8, the image forming apparatus 10 (the multi-function device (B)) and the terminal device 12 (PC (A)) are identified as devices to be cooperated, and as shown in FIG. In the device 16, device images 64 and 66 are displayed on the device display screen 62. Further, as indicated by an arrow 192, the device image 64 is superimposed on the device image 66, whereby a new device image 160 is generated and displayed. The image forming apparatus 10 (multifunctional peripheral (B)) and the terminal device 12 (PC (A)) are designated as devices to be cooperated.

  In the eighth modification, the control unit 48 of the terminal device 16 changes the imaging area of the camera 42 while displaying the device image generated by overlapping at the end on the device display screen 62 and displaying the device display screen 62. Display on. In this case, the identification information of the linked device (the device linked to the overlapped device image) is stored in the terminal device 16 or the server 14.

  In the example shown in FIG. 33A, since the device image 64 is superimposed on the device image 66 and a new device image 160 is generated and displayed, the device image 160 remains displayed on the device display screen 62. Ru. For example, when using the projector (C) which can not be photographed in the environment shown in FIG. 33A as a device to be cooperated, the device image 160 is displayed on the device display screen 62 as shown in FIG. 33B. In this state, the user shoots the projector (C) by changing the direction of the camera 42 so that the projector (C) is included in the shooting area of the camera 42. Thus, the projector (C) is photographed by the camera 42, and the device image 138 linked to the projector (C) is displayed on the device display screen 62. By superimposing the device image 160 on the device image 138, the projector (C) is designated as a device to be cooperated.

  According to the eighth modification, even when the plurality of devices to be cooperated can not be photographed at one time, the plurality of devices can be photographed separately and cooperated.

  In the above example, the device image generated by superimposing at the end is displayed on the device display screen 62, but as another example, the device image designated by the user or the first image taken A device image linked to a device or a device image linked to a preset basic device may be displayed on the device display screen 62 and displayed. Also in this case, it becomes possible to capture a plurality of devices separately and to make them cooperate.

(Modification 9)
Hereinafter, modification 9 will be described. In the ninth modification, the process according to the first embodiment is applied. That is, according to the order in which the device images are superimposed, the display priority of the information related to the cooperative function is determined. For example, as shown in FIG. 24B, when the device image 64 is selected and the device image 64 is manipulated and superimposed on the device image 66, the image is connected from the device image 64 to the device image 66. become. As a result, the image forming apparatus 10 (the multifunction machine (B)) linked to the device image 64 and the terminal device 12 (PC (A)) linked to the device image 66 are designated as linked devices, and connected. The order is specified. The order of overlapping the device images corresponds to the order of connection. In the example shown in FIG. 24B, a device is connected from the image forming apparatus 10 (MFP (B)) to the terminal device 12 (PC (A)).

  In the ninth modification, the server 14A stores cooperation function information 34 (cooperation function management table shown in FIG. 7) according to the first embodiment. As described above, when devices to be cooperated (for example, the image forming apparatus 10 and the terminal device 12) are designated, the specifying unit 38 of the server 14A performs the cooperation function management table (cooperation function information 34 shown in FIG. 7). , Identify the cooperation function associated with the combination of the PC (A) and the multifunction machine (B). As a result, the cooperation function realized by causing the PC (A) and the multifunction peripheral (B) to cooperate is specified. In addition, when the connection order of devices is specified by the user, the specification unit 38 specifies the priority order associated with the connection order in the cooperation function management table.

  The information on the cooperation function identified as described above and the information indicating the priority are transmitted from the server 14A to the terminal device 16. The control unit 48 of the terminal device 16 causes the UI unit 46 to display the information on the cooperation function as the information on the cooperation function candidate according to the priority.

  As described above, according to the ninth modification, the cooperation function that uses the function of the device is identified by superimposing the device image. Moreover, the display order of the information regarding a cooperation function is changed by changing the order which overlaps an apparatus image, ie, the order which connects an apparatus. Therefore, the information related to the cooperation function predicted to be used by the user is preferentially displayed without performing any special operation other than the operation of overlapping the device images.

(Modification 10)
Hereinafter, with reference to FIG. 34, the modification 10 will be described. FIG. 34 shows an example of a screen displayed on the UI unit 46 of the terminal device 16.

  In the tenth modification, as in the first modification of the first embodiment, the function of the device assigned to the cooperative function is different according to the position in the device image linked to the device. By superimposing a specific partial image in a device image on a specific partial image in another device image, a cooperative function using the function associated with the specific partial image is specified. Hereinafter, modification 10 will be described in detail.

  Also in the modified example 10, it is assumed that the image forming apparatus 10 (multifunctional peripheral (B)) and the terminal device 12 (PC (A)) are identified. As shown in FIG. 34A, the device display screen 62 is displayed on the UI unit 46 of the terminal device 16, and the device images 64 and 66 are displayed on the device display screen 62. As in the first modification of the first embodiment, the "print function" is associated with the partial image 76 in the device image 64, and the "scan function" is associated with the partial image 78 in the device image 64. The "stapling function" is associated with the partial image 80 in the device image 64. Further, the “data storage function” is associated with the partial image 82 in the device image 66, and the “screen display function” is associated with the partial image 84 in the device image 66. In the tenth modification, the partial image is displayed as an image that can be moved separately from the other partial images.

  As in the first modification of the first embodiment, the names of the respective functions (for example, printing, scanning, etc.) may be displayed on the device display screen 62, and the contact area images 90 to 98 are displayed on the device display screen 62. It may be displayed. In this case, the color of the touch area image corresponding to the partial image designated by the user or the color of the touch area image corresponding to the superimposed partial image may be changed. Of course, the name of the function and the contact area image may not be displayed.

  In the modification 10, when a partial image is designated by the user and the partial image is superimposed on another partial image, a cooperation function that uses the function associated with the superimposed partial image is identified, and the cooperation function is related to the cooperation function. The information is displayed on the UI unit 46 of the terminal device 16. This specifying process may be performed by the specifying unit 38 of the server 14 or may be performed by the terminal device 16.

  For example, as shown by an arrow 194 in FIG. 34B, when the user drags the partial image 78 using the operation element to overlap the partial image 84 and perform a drop operation, the partial image 78 is included. The MFP (B) linked to the device image 64 and the PC (A) linked to the device image 66 including the partial image 84 are designated as the devices to be cooperated, and “scan function” corresponding to the partial image 78 The “screen display function” corresponding to the partial image 82 is designated as the function to be cooperated.

  In the server 14A, functions corresponding to partial images are managed. For example, identification information for identifying a partial image, function identification information indicating a function associated with the partial image, and linked function identification information indicating a linked function to be executed by linking the functions correspond to each other. And stored in the server 14A. When a partial image is selected on the device display screen 62 and superimposed on another partial image, identification information indicating the superimposed partial image is transmitted from the terminal device 16 to the server 14A. In the example shown in FIG. 34 (b), identification information respectively indicating the partial images 78 and 84 is transmitted from the terminal device 16 to the server 14A. The specifying unit 38 of the server 14A specifies a function corresponding to each of the superimposed partial images 78 and 84 based on the identification information, and specifies a cooperation function using the function. The information on the cooperation function is transmitted from the server 14A to the terminal device 16 and displayed.

  As described above, according to the modification 10, when the device to be cooperated has a plurality of functions, the function is individually designated, and the information on the cooperation function using the designated function is prioritized Is displayed on. Thereby, the cooperation function predicted to be used by the user is preferentially displayed.

  Also in the modification 10, as in the first embodiment, the display priority of the cooperative function may be changed according to the order in which the partial images are superimposed. In this case, information relating to the cooperation function using the function linked to the superimposed partial image is displayed preferentially.

(Modification 11)
Hereinafter, modification 11 will be described. In the eleventh modification, as in the tenth modification, depending on the position in the device image linked to the device, the function of the device assigned to the cooperation function is different, but unlike the tenth modification, the partial image is different It is not separately movable from other partial images separately. In the eleventh modification, as in the example shown in FIG. 24B, the user moves the entire device image and superimposes the other device images. At this time, the cooperation function is changed in accordance with the overlapped position. This point will be described with reference to a specific example with reference to FIG. For example, it is assumed that the entire device image 64 linked to the multifunction machine (B) is operated by the user and superimposed on the device image 66 linked to the PC (A). At this time, if the partial image 78 (partial image linked to the scan function) in the device image 64 and the partial image 84 (partial image linked to the screen display function) in the device image 66 overlap, the scan function The information regarding the cooperation function using the and the screen display function is displayed preferentially.

  Also in the second embodiment and the modified example 5-11 described above, the devices to be cooperated are connected according to the sequence shown in FIG. 21 as in the first embodiment.

  Further, in the second embodiment and the modified example 5-11 described above, the device image associated with the identified device and the device image superimposed may be displayed in three dimensions so as not to be buried in the background image. . That is, those images may be displayed as a three-dimensional image. For example, the background image is displayed in two dimensions, and the device image is displayed in three dimensions. This improves the visibility of the device image. Further, the color of the designated device image may be changed or the designated device image may be vibrated so that the device image designated by the user is identified from other device images.

  In the first embodiment, the second embodiment, and the modification 1-11 described above, the hardware such as the device is identified to realize the cooperation function, but the software (application program) is identified and each software is identified. The functions of may be linked. For example, by linking or overlapping images (corresponding to an example of an image linked to a function) such as icons relating to functions displayed on a home screen of a smartphone or a desktop screen of a personal computer, a cooperation function is specified, Information on the cooperation function may be displayed. In this case, the display priority of the cooperation function may be changed according to the order of connecting the images and the order of overlapping.

  In addition, according to the first embodiment, the second embodiment, and the modification 1-11 described above, by applying the AR technology, a cooperation function that uses a function of a device to be cooperated is identified, and the cooperation function is related. Information is displayed. Thereby, even if it is not clear from the appearance of what kind of cooperation function can be executed by the device to be cooperated, information on the cooperation function is provided to the user. In addition, it is convenient because functions that can not be performed individually by individual devices can be performed by linking a plurality of devices. In addition, it is possible to use the cooperation function simply by applying AR technology and identifying the device to be cooperated, which is simpler than when the user performs settings etc. for executing the cooperation function manually. The cooperation function can be used by the operation, and the time and effort of the user can be reduced.

  Further, in an environment where a plurality of devices are used by a plurality of users, information on the cooperation function is displayed on the terminal device 16 of each user. For example, even if a user interface such as a touch panel is removed from the device, the terminal device 16 functions as a user interface, and information on the cooperation function is displayed on the terminal device 16 of each user. As another scene, for example, when a plurality of devices are temporarily used at the place where the user goes out, a user interface suitable for the user, that is, the function of the device to be cooperated designated by the user is used. The user interface representing the information on the linked function is displayed.

  Device identification information and an image of the device obtained by imaging the device in advance in the process of the linked function execution operation of the first embodiment, the second embodiment and the modified example 1-11 described above, and another user identifying and acquiring the device If the device has already been identified, the device identification information and the image of the device are stored in the storage unit such as the server 14 or the terminal device 16 and edited to execute the cooperation function later. The screen is started, and the stored device identification information and the image of the device are read out from the storage unit, and the image of the device is connected or superimposed on the editing screen as described above to execute the cooperation function. May be Here, with reference to FIG. 35, a specific example will be described. An editing screen 196 displayed on the terminal device 16 is shown in FIG. In this editing screen, as an example, the device image 66 linked to the PC (A), the device image 64 linked to the multi-function machine (B), the device image 138 linked to the projector (C), and the camera (D) A linked device image 198 is displayed. As described above, these images are images already acquired in the past, and are stored in the storage unit such as the server 14 or the terminal device 16. When executing the cooperation function, these images are read from the storage unit and displayed on the editing screen 196. As described above, the linking function is executed on the editing screen 196 by connecting or overlapping the images. By performing such processing, it is not necessary to take a picture of a device that has been identified in the past every time the linked function is executed, and the linked function can be efficiently executed. In addition, since there is no image other than the device reflected in the photographed image of the camera, the visibility of the device to be operated is also improved.

  Each of the image forming apparatus 10, the terminal devices 12 and 16 and the servers 14 and 14A is realized by cooperation of hardware resources and software as an example. Specifically, each of the image forming apparatus 10, the terminal devices 12 and 16, and the servers 14 and 14A includes one or more processors such as a CPU (not shown). The one or more processors read and execute a program stored in a storage device (not shown) to realize the functions of the respective units of the image forming apparatus 10, the terminal devices 12 and 16, and the servers 14 and 14A. The program is stored in the storage device via a recording medium such as a CD or a DVD or via a communication path such as a network. As another example, each unit of the image forming apparatus 10, the terminal devices 12 and 16, and the servers 14 and 14A may be realized by hardware resources such as a processor and an electronic circuit. A device such as a memory may be used in the implementation. As still another example, each unit of the image forming apparatus 10, the terminal devices 12 and 16, and the servers 14 and 14A may be realized by a digital signal processor (DSP), a field programmable gate array (FPGA), or the like.

DESCRIPTION OF SYMBOLS 10 Image forming apparatus, 12, 16 terminal device, 14, 14A server, 18, 28, 40 Communication part, 20 Image forming part, 22, 30, 30A, 44 Storage part 24, 46 UI part, 26, 36, 36A , 48 control unit, 32 device function information, 34, 144 linked function information, 38 identification unit, 42 camera, 146 linked function registration unit.

Claims (5)

Control means for changing the function according to a portion where the first image linked to the first function and the second image linked to the second function overlap each other,
An information processing apparatus having The function uses a function corresponding to the overlapping portion.
An information processing apparatus according to claim 1, characterized in that. The first image and the second image are displayed on the display means, and the whole or the part of the first image is superimposed on the whole or the part of the second image by the user on the display means.
The information processing apparatus according to claim 1 or 2, characterized in that: The control means changes the display color of the overlapped portion.
The information processing apparatus according to any one of claims 1 to 3, characterized in that: Computer,
Control means for changing the function according to a portion where the first image linked to the first function and the second image linked to the second function overlap each other,
A program to function as