JP2018067291A - Information processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide information on executable cooperation functions.SOLUTION: An information processing apparatus combines a first function of a first apparatus and a second function of a second apparatus according to display positions of a first image associated with the first apparatus and a second image associated with the second apparatus, so as to control display of executable cooperation functions.SELECTED DRAWING: Figure 19

Description

  The present invention relates to an information processing apparatus and a program.

  Patent Document 1 discloses a technique for displaying an image indicating operation guidance on an operation panel that can be attached to and detached from the apparatus main body and wirelessly communicated.

  Patent Document 2 discloses a relay device that relays data between a client device and a service providing device that provides a service.

  Patent Document 3 discloses a portable terminal that generates an augmented reality image in which additional information including state information of an image forming apparatus is superimposed on a captured image captured by an imaging unit.

  Patent Document 4 discloses a technique for displaying a usage status of each cloud server by a user on a display of a device operated by the user during login in a communication system communicating with a plurality of cloud servers.

Patent No. 5737906 JP 2014-10769 A JP 2014-48672 A JP 2014-238786 A

  By the way, there are cases where it is not easy for the user to know what cooperation function can be executed using the device or function.

  An object of the present invention is to provide information on executable cooperation functions.

  According to the first aspect of the present invention, the first function of the first device and the second device correspond to the display positions of the first image associated with the first device and the second image associated with the second device. The information processing apparatus includes a control unit that switches display of a cooperative function that can be executed by combining the second function.

  According to a second aspect of the present invention, the first image is an image obtained by photographing the first device, and the second image is an image obtained by photographing the second device. The information processing apparatus according to claim 1.

  The invention according to claim 3 further includes identification means for identifying the first device based on the first image and identifying the second device based on the second image, and the control means includes: The information processing apparatus according to claim 2, wherein when the first device and the second device are identified by the identification unit, the display of the cooperation function is switched according to the display position.

  The invention according to claim 4 is executed by combining the first function and the second function in accordance with the display position of the first image associated with the first function and the second image associated with the second function. This is an information processing apparatus having control means for controlling display of possible cooperative functions.

  The invention according to claim 5 is a function in which the control means is associated with the partial image in the first image according to the display position of the partial image in the first image and the partial image in the second image. 5. The information processing apparatus according to claim 1, wherein display of the cooperative function that can be executed is switched by combining a function associated with a partial image in the second image. is there.

  The invention according to claim 6 is the information processing apparatus according to claim 5, wherein different functions are associated with each partial image.

  The invention according to claim 7 is the information processing apparatus according to any one of claims 1 to 6, wherein the cooperation function is a function that cannot be used before cooperation.

  According to an eighth aspect of the present invention, the first function of the first device and the first function of the computer according to the display positions of the first image associated with the first device and the second image associated with the second device. This is a program that functions as a switching unit that switches display of a cooperative function that can be executed by combining the second function of two devices.

  The invention according to claim 9 combines the first function and the second function in accordance with the display position of the first image associated with the first function and the second image associated with the second function. This is a program that functions as a control unit that controls display of an executable cooperative function.

  According to the first to fourth aspects of the present invention, information on the cooperation function that can be executed is provided.

  According to the fifth and sixth aspects of the present invention, information on a cooperative function that can be executed using a function associated with a partial image is provided.

  According to the seventh aspect of the invention, a function that could not be executed before cooperation can be executed.

1 is a block diagram illustrating an image forming system according to an embodiment of the present invention. 1 is a block diagram illustrating an image forming apparatus according to an exemplary embodiment. It is a block diagram which shows the server which concerns on this embodiment. It is a block diagram which shows the terminal device which concerns on this embodiment. 1 is a schematic diagram illustrating an appearance of an image forming apparatus. It is a figure which shows an example of an apparatus function management table. It is a figure which shows an example of a cooperation function management table. It is a figure which shows the apparatus utilized independently. It is a figure which shows an example of a function display screen. It is a figure which shows an example of a function display screen. It is a figure which shows the apparatus of cooperation object. It is a figure which shows an example of a function display screen. It is a figure which shows an example of an apparatus display screen. It is a figure which shows an example of an apparatus display screen. It is a figure which shows an example of the apparatus display screen which concerns on the modification 1. It is a figure which shows an example of the function display screen which concerns on the modification 1. FIG. It is a figure which shows an example of the apparatus display screen which concerns on the modification 1. It is a figure which shows an example of the function display screen which concerns on the modification 1. FIG. It is a figure which shows an example of the apparatus display screen which concerns on the modification 1. It is a figure which shows an example of the apparatus display screen which concerns on the modification 2. It is a figure which shows an example of the apparatus display screen which concerns on the modification 2. It is a figure which shows an example of the apparatus display screen which concerns on the modification 3. It is a figure which shows an example of the apparatus display screen which concerns on the modification 4. It is a figure which shows an example of the apparatus display screen which concerns on the modification 4. It is a figure which shows an example of the apparatus function management table which concerns on the modification 5. FIG. It is a figure which shows an example of the cooperation function management table which concerns on the modification 5. FIG. It is a figure which shows an example of the apparatus display screen which concerns on the modification 5. It is a figure which shows an example of the apparatus display screen which concerns on the modification 5. It is a figure which shows an example of the apparatus display screen which concerns on the modification 6. FIG. It is a figure which shows an example of the apparatus display screen which concerns on the modification 6. FIG. It is a figure which shows an example of the apparatus display screen which concerns on the modification 7. It is a figure which shows an example of the apparatus display screen which concerns on the modification 7. It is a sequence diagram which shows the connection process in an image forming system.

  With reference to FIG. 1, an image forming system as an information processing system according to an embodiment of the present invention will be described. FIG. 1 shows an example of an image forming system according to the present embodiment.

  The image forming system according to the present embodiment includes a plurality of devices (for example, devices 10 and 12), a server 14 as an example of an external device, and a terminal device 16 as an example of an information processing device. The devices 10 and 12, the server 14, and the terminal device 16 have a function of communicating with each other via a communication path N such as a network. Of course, the devices 10, 12, the server 14, and the terminal device 16 may communicate with other devices via different communication paths. In the example shown in FIG. 1, two devices (devices 10 and 12) are included in the image forming system, but three or more devices may be included in the image forming system. A plurality of servers 14 and a plurality of terminal devices 16 may be included in the image forming system.

  The devices 10 and 12 are devices having specific functions. For example, an image forming device having an image forming function, a personal computer (PC), a projector, a display device such as a liquid crystal display or a projector, a telephone, a clock, and a surveillance camera. , Etc. The devices 10 and 12 have a function of transmitting and receiving data to and from other devices. In the present embodiment, the device 10 is an image forming apparatus as an example. The image forming apparatus (device 10) is an apparatus having at least one of a scan function, a print function, a copy function, and a facsimile function, for example.

  The server 14 is a device that manages the functions of each device. For example, the server 14 manages a function that each device has and a cooperation function that uses a function that a plurality of apparatuses have. The server 14 has a function of transmitting and receiving data to and from other devices.

  For each user, the server 14 may manage functions that can be used by the user. The functions that can be used by the user are, for example, functions that are provided to the user free of charge, functions that are provided to the user for a fee and purchased by the user, and the like. The server 14 may manage usable function information (for example, function purchase history information) indicating functions available to the user for each user. Of course, since there are functions that can be used free of charge, additional update functions, and functions that are specially managed by the administrator, the server 14 does not need to manage the functions depending on whether or not there is a purchase. The function purchase process is performed by the server 14, for example. Of course, the purchase process may be performed by another apparatus.

  The terminal device 16 is a device such as a personal computer (PC), a tablet PC, a smartphone, or a mobile phone, and has a function of transmitting / receiving data to / from other devices. For example, the terminal device 16 functions as a user interface unit (UI unit) when the device is used.

  In the image forming system according to the present embodiment, for example, when one device is identified, display of the function of the device is controlled, and when a plurality of devices are identified, the functions of the plurality of devices are included. The display of the linkage function to be used is controlled. For example, when one device is photographed, display of a function of the device is controlled, and when a plurality of devices are photographed, display of a cooperation function that uses a function of the plurality of devices is controlled.

  Hereinafter, each device included in the image forming system according to the present embodiment will be described in detail.

  With reference to FIG. 2, the configuration of the device 10 as the image forming apparatus will be described in detail. Hereinafter, the device 10 is referred to as an image forming apparatus 10. FIG. 2 shows the configuration of the image forming apparatus 10.

  The communication unit 18 is a communication interface and has a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 18 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The image forming unit 20 has an image forming function. Specifically, the image forming unit 20 has at least one of a scan function, a print function, a copy function, and a facsimile function. By executing the scan function, the document is read and scan data (image data) is generated. By executing the print function, an image is printed on a recording medium such as paper. By executing the copy function, the document is read and printed on the recording medium. By executing the facsimile function, the image data is transmitted or received by facsimile. In addition, a cooperation function that links a plurality of functions may be executed. For example, a scan transfer function that combines a scan function and a transmission function (transfer function) may be executed. By executing the scan transfer function, the original is read to generate scan data (image data), and the scan data is transmitted to a transmission destination (for example, an external device such as the terminal device 16). Of course, this cooperation function is only an example, and another cooperation function may be executed.

  The storage unit 22 is a storage device such as a hard disk or a memory (for example, SSD). The storage unit 22 indicates, for example, information indicating an image formation command (for example, job information), image data to be printed, scan data generated by executing a scan function, and addresses of other devices. Device address information, server address information indicating the address of the server 14, various control data, various programs, and the like are stored. Of course, they may be stored in separate storage devices or may be stored in one storage device.

  The UI unit 24 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is an input device such as a touch panel or a keyboard. Of course, it may be a user interface that serves both as a display unit and an operation unit (for example, including a touch-type display or a device that electronically displays a keyboard or the like on the display). The image forming apparatus 10 may not include the UI unit 24, or may include a hardware user interface unit (hardware UI unit) as hardware without including a display unit. The hardware UI unit is, for example, a hardware key (for example, a numeric keypad) specialized for numerical input, a hardware key (for example, a direction indication key) specialized for direction indication, and the like.

  The control unit 26 controls the operation of each unit of the image forming apparatus 10.

  Hereinafter, the configuration of the server 14 will be described in detail with reference to FIG. FIG. 3 shows the configuration of the server 14.

  The communication unit 28 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 28 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The storage unit 30 is a storage device such as a hard disk or a memory (for example, SSD). The storage unit 30 stores, for example, device function information 32, linkage function information 34, various data, various programs, device address information indicating the address of each device, server address information indicating the address of the server 14, and the like. ing. Of course, they may be stored in separate storage devices or may be stored in one storage device. The device function information 32 and the cooperation function information 34 stored in the storage unit 30 are provided to the terminal device 16 regularly or at a designated timing, whereby the information stored in the terminal device 16 is updated. May be. Hereinafter, the device function information 32 and the link function information 34 will be described.

  The device function information 32 is information indicating a function of each device (for example, the image forming apparatus 10 or the device 12) included in the image forming system. For example, device identification information for identifying the device and the device have the device function information 32. It is information which shows matching with the function identification information for identifying a function. The device identification information is, for example, a device ID, a device name, information indicating the type of device, a model number, position information, an appearance image representing the appearance of the device, and the like. The function identification information is, for example, a function ID or a function name. For example, when the image forming apparatus 10 has a scan function, a print function, a copy function, and a scan transfer function, the device identification information of the image forming apparatus 10 includes function identification information indicating the scan function and function identification information indicating the print function. The function identification information indicating the copy function and the function identification information indicating the scan transfer function are associated with each other. By referring to the device function information 32, the function of each device is specified (identified).

  The cooperation function information 34 is information indicating a cooperation function executed by linking a plurality of functions. By coordinating a plurality of functions, one or a plurality of cooperation functions are executed. The cooperation function may be executed by linking a plurality of functions of one device (for example, the image forming apparatus 10 or the device 12), or a plurality of devices (for example, the image forming apparatus 10 and the device 12). This function may be executed by linking these functions. In addition, a terminal device that issues an operation instruction (in the present embodiment, the terminal device 16) may be included in the device to be identified, and the functions of the terminal device may be used as part of the linkage function.

  The cooperation function information 34 is information indicating a correspondence between a combination of function identification information for identifying each function used for the cooperation function and cooperation function identification information for identifying the cooperation function, for example. . The cooperation function identification information is, for example, a cooperation function ID or a cooperation function name.

  When a plurality of devices are to be linked, the linkage function information 34 is information indicating a linkage function that uses a plurality of functions of the plurality of devices, and device identification information for identifying each device used for the linkage function. This is information indicating the association between the combination and the cooperative function identification information.

  The cooperation function may be a function executed by linking a plurality of different functions to each other, or may be a function executed by linking the same function. The cooperation function may be a function that could not be used before the cooperation. The functions that could not be used before cooperation may be functions that can be used by using the same functions among the functions of the devices to be linked, or functions that can be used by combining different functions. May be. For example, by coordinating a device having a print function (printer) and a device having a scan function (scanner), a copy function as a cooperation function is realized. That is, the copy function is realized by linking the print function and the scan function. In this case, a copy function as a linkage function is associated with a combination of a print function and a scan function. In the cooperation function information 34, for example, cooperation function identification information for identifying a copy function as a cooperation function, apparatus identification information for identifying a device having a print function, and a device having a scan function are identified. A combination of device identification information is associated with the combination.

  The storage unit 30 may store usable function information. The usable function information is information indicating a function that can be used by each user. For example, user identification information for identifying a user, and function identification information indicating a function that can be used by the user (cooperation function identification information). Information). As described above, the functions that can be used by the user are, for example, functions provided free of charge to the user, functions purchased by the user, and the like. It may be. The user identification information is user account information such as a user ID and a name, for example. By referring to the usable function information, a function that can be used by each user is specified. The available function information is updated, for example, every time a function is provided to the user (for example, every time a function is provided to the user free of charge or for a fee).

  The control unit 36 controls the operation of each unit of the server 14. The control unit 36 includes a specifying unit 38.

  The specifying unit 38 receives device identification information for identifying a device, and specifies function identification information of a function associated with the device identification information in the device function information 32 stored in the storage unit 30. As a result, the function of the device is specified (identified). For example, the device identification information is transmitted from the terminal device 16 to the server 14, and the function identification information of the function associated with the device identification information is specified by the specifying unit 38. Information related to the function (for example, function identification information and function description information) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. Thereby, information related to the function of the device specified by the device identification information is displayed on the terminal device 16.

  The specifying unit 38 receives device identification information for identifying each device to be linked, and the linkage function associated with the combination of each piece of device identification information in the linkage function information 34 stored in the storage unit 30. Identify the linked function identification information. Thereby, the cooperation function performed by cooperating the function which each apparatus of cooperation object has is identified (identified). For example, a plurality of pieces of device identification information are transmitted from the terminal device 16 to the server 14, and the specifying unit 38 specifies the link function identification information of the link function associated with the plurality of device identification information. Information relating to the cooperation function (for example, cooperation function identification information and explanation information of the cooperation function) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. Thereby, information related to the cooperation function executed by the plurality of devices specified by the plurality of device identification information is displayed on the terminal device 16.

  For example, when one device is identified (for example, when one device is photographed), the specifying unit 38 receives device identification information for identifying the device, and includes the device identification information in the device function information 32. The function identification information of the associated function is specified. Thereby, when one device is identified (for example, when one device is photographed), the function of the device is specified (identified). When a plurality of devices are identified (for example, when a plurality of devices are photographed), the specifying unit 38 receives device identification information for identifying each device included in the plurality of devices, and in the cooperation function information 34 The linkage function identification information of the linkage function associated with each combination of device identification information is specified. Thereby, when a plurality of devices are identified (for example, when a plurality of devices are photographed), a cooperation function that uses a function of the plurality of devices is specified (identified). As described above, the specifying unit 38 switches the function specifying process according to the number of identified devices (for example, according to the number of devices photographed). In the terminal device 16, information related to a function of a single device or information related to a cooperation function using a function of a plurality of devices is displayed according to the number of devices identified (for example, photographed). Of course, even when a plurality of devices are photographed, the specifying unit 38 receives the device identification information of each device included in the plurality of devices and refers to the device function information 32 to thereby have each device. The function may be specified (identified).

  When the functions that can be used by the user are managed, the specifying unit 38 receives user identification information for identifying the user, and the user identification information is stored in the usable function information stored in the storage unit 30. You may identify the function identification information of each function matched. Thereby, the function group which can be used by the user is specified (identified). For example, user identification information is transmitted from the terminal device 16 to the server 14, and the function identification information of each function associated with the user identification information is specified by the specifying unit 38. Information regarding each function available to the user (for example, information indicating the name of each function) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16, for example. As a result, information related to each function available to the user specified by the user identification information is displayed on the terminal device 16. For example, the specifying unit 38 receives the device identification information and the user identification information, specifies the function identification information of the function associated with the device identification information in the device function information 32, and also includes the user in the available function information. The function identification information of the function associated with the identification information is specified. As a result, a function that is included in the device specified by the device identification information and that can be used by the user specified by the user identification information is specified.

  The control unit 36 may execute a function purchase process and manage the purchase history. For example, when a paid function is purchased by a user, the control unit 36 may apply a charging process to the user.

  Further, the control unit 36 may execute functions related to image processing such as a character recognition function, a translation function, an image processing function, and an image forming function. Of course, the control unit 36 may execute a function related to processing other than image processing. By executing the character recognition function, a character in the image is recognized and character data indicating the character is generated. By executing the translation function, the characters in the image are translated into characters expressed in a specific language, and character data indicating the translated characters is generated. The image is processed by executing the image processing function. The control unit 36 receives, for example, scan data generated by executing a scan function from the image forming apparatus 10, and performs image processing such as a character recognition function, a translation function, an image processing function, and the like for the scan data. Execute functions related to. The control unit 36 may receive image data from the terminal device 16 and execute each function for the image data. The character data and image data generated by the control unit 36 are transmitted from the server 14 to the terminal device 16, for example. The server 14 may be used as an external device, and the cooperation function may be a function that uses functions of a plurality of devices including the server 14.

  Hereinafter, the configuration of the terminal device 16 will be described in detail with reference to FIG. FIG. 4 shows the configuration of the terminal device 16.

  The communication unit 40 is a communication interface, and has a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 40 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

  The camera 42 as an imaging unit generates image data (for example, still image data or moving image data) by imaging an imaging target. In addition to using the camera of the terminal device 16, the communication unit 40 receives image data captured by an external camera connected to a communication path such as a network, and the UI unit 46 displays the image data. The image data may be manipulated.

  The storage unit 44 is a storage device such as a hard disk or a memory (for example, SSD), and various programs, various data, address information of the server 14, address information of each device (for example, address information of the image forming device 10 and the device 12). ), Information regarding the identified device, information regarding the identified device to be linked, information regarding the function of the identified device, information regarding the linkage function, and the like.

  The UI unit 46 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel, a keyboard, and a mouse. Of course, a user interface that serves as both a display unit and an operation unit (for example, a touch-type display or a device that electronically displays a keyboard or the like on the display) may be used.

  The control unit 48 controls the operation of each unit of the terminal device 16. The control unit 48 functions as, for example, a display control unit (control unit), and displays various types of information on the display unit of the UI unit 46.

  On the display unit of the UI unit 46, for example, an image taken by the camera 42, an image associated with a device identified as a device to be used (for example, a device used alone or a device to be linked), etc. Is displayed. The image associated with the device may be an image (still image or moving image) representing the device taken by the camera 42, or may be an image (for example, an icon) schematically representing the device. The image data schematically represented may be stored in the server 14 and provided from the server 14 to the terminal device 16, or may be stored in advance in the terminal device 16, or stored in another device. The terminal device 16 may be provided from the other device.

  The device function information 32 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the device function information 32 may not be stored in the storage unit 30 of the server 14. Similarly, the cooperation function information 34 described above may be stored in the storage unit 44 of the terminal device 16. In this case, the cooperation function information 34 may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 includes the specifying unit 38 described above, identifies the device based on the device identification information, may specify the function of the device, and has a linkage function that uses a plurality of functions. You may specify. In this case, the server 14 may not have the specifying unit 38.

  When the usable function information is created, the usable function information may be stored in the storage unit 44 of the terminal device 16. In this case, the usable function information may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 may manage the purchase history of the function by the user. In this case, the control unit 36 of the server 14 may not have the management function. The control unit 48 of the terminal device 16 may specify a function that can be used by the user based on the user identification information.

  As yet another example, device function information 32 and linkage function information 34 may be stored in devices such as the image forming apparatus 10 and the device 12, and devices such as the image forming device 10 and the device 12 may be specified by the specifying unit 38. You may have. That is, the processing by the specifying unit 38 of the server 14 may be performed in the server 14, may be performed in the terminal device 16, or may be performed in devices such as the image forming apparatus 10 and the device 12.

  In this embodiment, as an example, by applying AR (Augmented Reality) technology (augmented reality technology), device identification information is acquired and a device to be used is identified. For example, by applying AR technology, device identification information of a device used alone is acquired and the device is identified, and device identification information of a device to be linked is acquired to identify a device to be linked. Is done. A known AR technique is used as the AR technique. For example, a marker type AR technique using a marker such as a two-dimensional barcode, a markerless type AR technique using an image recognition technique, a position information AR technique using position information, and the like are used. Of course, the device identification information may be acquired without applying the AR technology to identify the device to be used. For example, if the device is connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device. Furthermore, if the device or terminal device has various wireless communication functions such as infrared communication, visible light communication, Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (Bluetooth (registered trademark)), these wireless communication functions are provided. The device may be identified by acquiring the device ID of the device to be used and linked to execute the linkage function.

  Hereinafter, the device identification information acquisition process will be described in detail with reference to FIG. As an example, a case where the device identification information of the image forming apparatus 10 is acquired will be described. FIG. 5 schematically shows the appearance of the image forming apparatus 10. Here, a process for acquiring the device identification information by applying the marker AR technology will be described. The casing of the image forming apparatus 10 is provided with a marker 50 such as a two-dimensional barcode. The marker 50 is information in which device identification information of the image forming apparatus 10 is encoded. The user activates the camera 42 of the terminal device 16 and photographs the marker 50 provided in the image forming apparatus 10 to be used by the camera 42. Thereby, image data representing the marker 50 is generated. The image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control part 36 extracts apparatus identification information by applying a decoding process with respect to the marker image represented by the said image data. Thereby, the image forming apparatus 10 to be used (the image forming apparatus 10 having the photographed marker 50) is identified. The specifying unit 38 of the server 14 specifies function identification information of a function associated with the extracted device identification information in the device function information 32. Thereby, the function of the image forming apparatus 10 to be used is specified (identified).

  Note that the control unit 48 of the terminal device 16 may extract the device identification information by applying a decoding process to the image data representing the marker 50. In this case, the extracted device identification information is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies function identification information of a function associated with the device identification information transmitted from the terminal device 16 in the device function information 32. When the device function information 32 is stored in the storage unit 44 of the terminal device 16, the control unit 48 of the terminal device 16 functions in the function associated with the extracted device identification information in the device function information 32. Identification information may be specified.

  The marker 50 may include encoded function identification information of the function of the image forming apparatus 10. In this case, by applying the decoding process to the image data representing the marker 50, the device identification information of the image forming apparatus 10 is extracted, and the function identification information of the function of the image forming apparatus 10 is also extracted. Thereby, the image forming apparatus 10 is specified (identified), and the function of the image forming apparatus 10 is specified (identified). This decoding process may be performed by the server 14 or the terminal device 16.

  When executing a linkage function that uses a function of a plurality of devices, device identification information of each device is acquired by photographing a marker of each device to be linked, and thereby the linkage function is specified (identified). .

  When acquiring the device identification information by applying the markerless AR technology, for example, the user captures all or part of the appearance of the device to be used (for example, the image forming apparatus 10) with the camera 42 of the terminal device 16. Of course, it is useful to obtain information for identifying the device such as the name of the device to be used (for example, product name) and the model number from the appearance. Appearance image data representing all or a part of the appearance of the device to be used is generated by photographing. The appearance image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 identifies a device to be used based on the appearance image data. For example, the storage unit 30 of the server 14 stores, for each device, appearance image association information indicating association between appearance image data representing all or part of the appearance of the device and device identification information of the device. Has been. For example, the control unit 36 compares the appearance image data transmitted from the terminal device 16 with each appearance image data included in the appearance image association information, and based on the comparison result, device identification information of the device to be used. Is identified. For example, the control unit 36 extracts the appearance feature of the device to be used from the appearance image data transmitted from the terminal device 16, and is the same as the appearance feature in the appearance image data group included in the appearance image association information. Alternatively, appearance image data representing similar features is specified, and device identification information associated with the appearance image data is specified. Thereby, a device to be used (device photographed by the camera 42) is identified. As another example, when a device name (for example, a product name) or model number is photographed and appearance image data representing the name or model number is generated, it is based on the name or model number represented in the appearance image data. Thus, the device to be used may be identified. The specifying unit 38 of the server 14 specifies function identification information of each function associated with the specified device identification information in the device function information 32. Thereby, the function which the apparatus (for example, image forming apparatus 10) of utilization object has is specified.

  Note that the control unit 48 of the terminal device 16 compares the appearance image data representing all or part of the appearance of the device to be used (for example, the image forming apparatus 10) with each appearance image data included in the appearance image association information. And based on the comparison result, you may identify the apparatus identification information of the apparatus of utilization object. The appearance image association information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the appearance image association information stored in the storage unit 44 of the terminal device 16. Alternatively, the control unit 48 of the terminal device 16 may specify the device identification information of the device to be used by acquiring the appearance image association information from the server 14 and referring to the appearance image association information.

  When executing a linkage function that uses the functions of multiple devices, the device identification information of each device is acquired by photographing all or part of the appearance of each device to be linked. (Identified).

  When device identification information is acquired by applying the position information AR technology, for example, by using a GPS (Global Positioning System) function, position information indicating a position where a device (for example, the image forming apparatus 10) is installed is obtained. To be acquired. For example, each device has a GPS function, and device location information indicating the location of the device itself is acquired. The terminal device 16 outputs information indicating an acquisition request for device position information to the device to be used, and receives device position information of the device from the device as a response to the acquisition request. The device position information is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 identifies a device to be used based on the device position information. For example, the storage unit 30 of the server 14 stores, for each device, position association information indicating association between device position information indicating a position where the device is installed and device identification information of the device. Yes. The controller 36 specifies device identification information associated with the device position information transmitted from the terminal device 16 in the position association information. Thereby, the device to be used is specified (identified). The specifying unit 38 of the server 14 specifies function identification information of each function associated with the specified device identification information in the device function information 32. Thereby, the function of the device to be used (for example, the image forming apparatus 10) is specified (identified).

  Note that the control unit 48 of the terminal device 16 may specify device identification information associated with the position information of the device to be used in the position association information. The position association information may be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device to be used by referring to the position association information stored in the storage unit 44 of the terminal device 16. As another example, the control unit 48 of the terminal device 16 may acquire the position association information from the server 14 and specify the device identification information of the device to be used by referring to the position association information. .

  When executing a linkage function using a plurality of devices, device position information of each device to be linked is acquired, and device identification information of each device is specified based on each device position information. Thereby, a cooperation function is specified (identified).

  Hereinafter, the image forming system according to the present embodiment will be described in detail.

  The device function information 32 will be described in detail with reference to FIG. FIG. 6 shows an example of a device function management table as the device function information 32. In the device function management table, for example, a device ID, information indicating a device name (for example, device type), information indicating a function of the device (function identification information), and an image ID are associated with each other. ing. The device ID and the device name correspond to an example of device identification information. The image ID is an example of image identification information for identifying an image representing a device (for example, an image representing an external appearance of the device, an image schematically representing the device (for example, an icon), or the like). The device function management table may not include the image ID. For example, a device whose device ID is “B” is a multifunction peripheral (an image forming apparatus having a plurality of image forming functions), and has functions such as a print function and a scan function. The device is associated with an image ID for identifying an image representing the device. Image data representing the device is stored in, for example, the storage unit 30 of the server 14.

  For example, by applying the AR technology, a device ID for identifying a device to be used is acquired. The specifying unit 38 of the server 14 specifies the device name, function, and image ID associated with the device ID by referring to the device function management table. Thereby, the device to be used is identified. For example, information indicating the device name and image data representing the device are transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. An image representing a device is displayed as an image associated with the device. Of course, the image itself captured by the camera 42 may be displayed on the UI unit 46 of the terminal device 16. Further, in the UI unit 46 of the terminal device 16, when an image associated with the device (for example, an image photographed by the camera 42 or an image schematically representing the device) is designated by the user, the function of the device is provided. Information (for example, function identification information and function description information) may be transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

  Hereinafter, the cooperation function information 34 will be described in detail with reference to FIG. FIG. 7 shows an example of a cooperation function management table as the cooperation function information 34. In the cooperation function management table, as an example, a combination of device IDs, information indicating a device name to be linked (for example, the type of each device), and information indicating a cooperation function (cooperation function identification information) are associated with each other. It has been. For example, a device with a device ID “A” is a PC (personal computer), and a device with a device ID “B” is a multifunction device. By linking the PC (A) and the multifunction peripheral (B), for example, a “scan transfer function” or a “print function” is realized as a link function. The “scan transfer function” is a function for transferring image data generated by scanning by the multifunction peripheral (B) to the PC (A). The “print function” is a function for transmitting data (for example, image data or document data) stored in the PC (A) to the multifunction device (B) and printing the data on the multifunction device (B).

  Hereinafter, with reference to FIG. 8, processing when the device is used alone will be described. FIG. 8 shows an example of a device used alone. As an example, it is assumed that the image forming apparatus 10 is a device used alone. The image forming apparatus 10 is, for example, a multifunction machine. The image forming apparatus 10 is a device that exists in a real space. The terminal device 16 illustrated in FIG. 8 is a device that exists in the real space, and is a mobile terminal device such as a smartphone or a mobile phone.

  For example, a marker 50 such as a two-dimensional barcode is provided on the housing of the image forming apparatus 10. When the marker type AR technology or the markerless type AR technology is used, the user photographs the image forming apparatus 10 to be used by the camera 42 of the terminal device 16 (for example, a smartphone). As a result, image data representing the marker 50 or appearance image data representing the appearance of the image forming apparatus 10 is generated. A device display screen 52 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 54 associated with the image forming apparatus 10 is displayed on the device display screen 52. The device image 54 is, for example, an image generated by photographing with the camera 42 (an image having a size at the time of photographing may be used, or an enlarged or reduced image may be used).

  Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 applies the decoding process to the image data to extract the device identification information of the image forming apparatus 10, thereby identifying the image forming apparatus 10. As another example, appearance image data representing the appearance of the image forming apparatus 10 may be generated, and the appearance image data may be transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 specifies the device identification information of the image forming apparatus 10 by referring to the appearance image association information. Thereby, the image forming apparatus 10 is identified.

  Further, the specifying unit 38 of the server 14 specifies (identifies) the function of the image forming apparatus 10 by referring to the device function information 32 (for example, the device function management table shown in FIG. 6). This point will be described in detail with reference to FIG. The image forming apparatus 10 is assumed to be a “multifunction machine (B)” as an example. The specifying unit 38 specifies the function associated with the multifunction peripheral (B) in the device function management table shown in FIG. Thereby, the function which a multifunctional machine (B) has is specified. Information regarding the identified function is transmitted from the server 14 to the terminal device 16. Of course, the process for identifying a device or a function may be performed by the terminal device 16.

  It should be noted that the device display screen 52 is a schematic image (not an image itself obtained by photographing) prepared in advance associated with the identified image forming apparatus 10 instead of an image generated by photographing by the camera 42. An image (for example, an icon) or an image generated by photographing with an external camera may be displayed as the device image 54.

  For example, when using image data obtained by shooting a device, the appearance of the current device itself (for example, an appearance that reflects scratches, notes, stickers attached to the device, etc.) is reflected in the image. As a result, the user can visually understand the difference from other similar devices.

  When using a schematic image, the data of the schematic image is transmitted from the server 14 to the terminal device 16, for example. For example, when the image forming apparatus 10 is identified, the specifying unit 38 of the server 14 refers to the device function management table (device function information 32) illustrated in FIG. Specific images. The schematic image data is transmitted from the server 14 to the terminal device 16, and the schematic image is displayed on the device display screen 52 as the device image 54. Schematic image data may be stored in the terminal device 16 in advance. In this case, when the image forming apparatus 10 is identified, the device image 54 stored in the terminal device 16 is displayed on the device display screen 52. Schematic image data may be stored in a device other than the server 14 or the terminal device 16.

  When a device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the device name may be displayed on the device display screen 52 in the terminal device 16. In the example illustrated in FIG. 8, the image forming apparatus 10 is a multifunction peripheral, and the name “multifunction peripheral (B)” is displayed.

  When the function of the image forming apparatus 10 is specified, for example, as illustrated in FIG. 9, the control unit 48 of the terminal device 16 displays a function display screen 56 on the UI unit 46 of the terminal device 16, and the function display screen is displayed. 56 displays information about the function. As information about a function, for example, a button image for instructing execution of the function is displayed. Since the multifunction peripheral (B) as the image forming apparatus 10 has, for example, a print function, a scan function, a copy function, and a facsimile function, button images for instructing execution of these functions are displayed as functions. It is displayed on the screen 56. For example, when a user designates a button image representing a print function using the terminal device 16 and instructs execution of the print function, execution instruction information indicating an instruction to execute the print function is transmitted from the terminal device 16 to the image forming apparatus 10. Sent. The execution instruction information includes control data for executing the print function, data such as image data to which the print function is applied, and the like. Upon receiving the execution instruction information, the image forming apparatus 10 executes printing according to the execution instruction information.

  FIG. 10 shows another example of the function display screen. As shown in FIG. 8, the function display screen 58 is a screen displayed on the UI unit 46 of the terminal device 16 when using a single device. As described above, the device to be used (for example, the image forming apparatus 10) is specified, and the function of the device to be used is specified. Moreover, the function identification information which shows the function with which the user using the apparatus of utilization object is matched with user identification information, ie, the function which the said user can utilize, may be specified. Furthermore, since the function of the device to be used is specified, the function that the device to be used does not have may be specified in the function group to be provided. Such information may be displayed on the function display screen 58.

  On the function display screen 58 shown in FIG. 10, a button image 60 representing the function A, a button image 62 representing the function B, and a button image 64 representing the function C are displayed as an example of the function identification information. ing. The function A is a function that a device to be used (for example, the identified image forming apparatus 10) has, and is a function that can be used by the user. The function B is a function possessed by the device to be used and cannot be used by the user. When the user receives the provision of the function B, the user can use the function B. If the function B is a paid function, the user can use the function B by purchasing the function B. If the function B is a free function, the user receives the function B free of charge. The function B can be used. Function C is a function that the target device does not have, that is, a function that the target device does not support. The control unit 48 of the terminal device 16 may change the display mode of the button image according to whether or not the function represented by the button image is a function of the device to be used. The control unit 48 may change the display mode of the button image depending on whether or not the function represented by the button image is a function that can be used by the user. For example, the control unit 48 may change the color and shape of the button image. In the example shown in FIG. 10, the button images 60, 62, and 64 are displayed in different colors. As an example, a button image (for example, a button image 60 representing the function A) that represents a function that the target device has and that can be used by the user is displayed in blue. A button image (a button image 62 representing the function B) that represents a function that the target device has and cannot be used by the user is displayed in yellow. A button image representing a function that the device to be used does not have (for example, a button image 64 representing the function C) is displayed in gray. As another example, the control unit 48 may change the shape of the button images 60, 62, and 64, or may change the font of the function display name. Of course, the display mode may be changed by another method. This informs the user whether or not each function can be used with high visibility.

  For example, when the user designates the button image 60 representing the function A using the terminal device 16 and gives an instruction to execute the function A, the execution instruction information indicating the execution instruction of the function A is used from the terminal device 16 as the usage target. Sent to other devices. The execution instruction information includes control data for executing function A, data such as image data to which processing by function A is applied, and the like. When the target device receives the execution instruction information, it executes the function A according to the execution instruction information. For example, when the device to be used is the image forming apparatus 10 and the function A is the scan transfer function, the scan function is executed by the image forming unit 20 of the image forming apparatus 10, and thereby the scan data (image data) is obtained. Generated. The scan data is transmitted from the image forming apparatus 10 to a transmission destination (for example, the terminal device 16) set.

  When the user designates the button image 62 representing the function B using the terminal device 16 and instructs the provision of the function B, the providing process is executed. When the providing process is performed by the server 14, the terminal device 16 accesses the server 14. As a result, a screen (for example, a website) for receiving the provision of the function B is displayed on the UI unit 46 of the terminal device 16 as information for enabling the user to use the function B. By performing the provision procedure on this screen, the user can use the function B. For example, a Web browser program is stored in the terminal device 16, and access from the terminal device 16 to the server 14 is realized by using the Web browser. When the user accesses the server 14 using a Web browser, a function providing screen (for example, a Web site) is displayed on the UI unit 46 of the terminal device 16 and functions are provided via the Web site. Of course, the providing process may be performed by a server other than the server 14. As another example, the UI unit 46 of the terminal device 16 has a use permission request screen (for example, Web) for requesting the administrator or the like to use the function B as information for enabling the user to use the function B. Site) may be displayed. If the administrator or the like is requested to permit the use of the function B via the use permission request screen and the permission is obtained, the user can use the function B.

  Hereinafter, with reference to FIG. 11, processing when the cooperation function is used will be described. FIG. 11 shows an example of a cooperation target device. As an example, it is assumed that the image forming apparatus 10 and a projector as the device 12 (hereinafter referred to as a projector 12) are used as devices to be linked. The image forming apparatus 10, the projector 12, and the terminal device 16 are devices that exist in the real space.

  For example, a marker 50 such as a two-dimensional barcode is provided on the casing of the image forming apparatus 10, and a marker 66 such as a two-dimensional barcode is provided on the casing of the projector 12. The marker 66 is information obtained by coding the device identification information of the projector 12. When the marker type AR technology or the markerless type AR technology is used, the user captures the image forming apparatus 10 and the projector 12 to be linked by the camera 42 of the terminal device 16 (for example, a smartphone). In the example illustrated in FIG. 11, the image forming apparatus 10 and the projector 12 are photographed together in a state where both the image forming apparatus 10 and the projector 12 are included in the photographing region of the camera 42. Thereby, image data representing each of the markers 50 and 66 is generated. A device display screen 68 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 70 associated with the image forming apparatus 10 and a device image 72 associated with the projector 12 are displayed on the device display screen 68. . The device images 70 and 72 are, for example, images generated by photographing with the camera 42 (an image having a size at the time of photographing may be used, or an enlarged or reduced image may be used).

  Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 applies the decoding process to the image data to extract the device identification information of the image forming device 10 and the device identification information of the projector 12, and thereby the image forming device 10 and the projector 12 are connected. Identify. As another example, appearance image data representing the appearances of the image forming apparatus 10 and the projector 12 may be generated and transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 specifies the device identification information of the image forming apparatus 10 and the device identification information of the projector 12 by referring to the appearance image association information. Thereby, the image forming apparatus 10 and the projector 12 are identified.

  Further, the specifying unit 38 of the server 14 uses the function of the image forming apparatus 10 and the function of the projector 12 by referring to the cooperation function information 34 (for example, the cooperation function management table shown in FIG. 7). Identify (identify) the linkage function to be performed. This point will be described in detail with reference to FIG. The image forming apparatus 10 is assumed to be a “multifunction machine (B)” as an example, and the projector 12 is assumed to be a “projector (C)” as an example. The specifying unit 38 specifies the link function associated with the combination of the multifunction peripheral (B) and the projector (C) in the link function management table shown in FIG. Thereby, the cooperation function performed by cooperating the multifunction peripheral (B) and the projector (C) is specified. Information regarding the identified linkage function is transmitted from the server 14 to the terminal device 16. Of course, the processing for identifying the device and the linkage function may be performed by the terminal device 16.

  The device display screen 68 includes an image prepared in advance (for example, a schematic image (for example, an icon)) associated with the identified image forming apparatus 10 instead of an image generated by photographing with the camera 42, An image generated by photographing with an external camera may be displayed as the device image 70. Similarly, an image prepared in advance associated with the identified projector 12 or an image generated by photographing with an external camera may be displayed as the device image 72. As described above, the schematic image data may be transmitted from the server 14 to the terminal device 16, stored in the terminal device 16 in advance, or stored in other devices. Also good.

  When a device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the device name may be displayed on the device display screen 68 in the terminal device 16. In the example shown in FIG. 11, the name “MFP (B)” of the image forming apparatus 10 and the name “Projector (C)” of the projector 12 are displayed.

  Even when a plurality of devices are photographed, the specifying unit 38 of the server 14 may specify the function of each device by referring to the device function information 32. In the example illustrated in FIG. 11, the specifying unit 38 may specify the function of the image forming apparatus 10 and the function of the projector 12. Information regarding the identified function may be transmitted from the server 14 to the terminal device 16.

  When the cooperation function is specified, for example, as illustrated in FIG. 12, the control unit 48 of the terminal device 16 displays the function display screen 74 on the UI unit 46 of the terminal device 16, and the function display screen 56 relates to the cooperation function. Display information. For example, a button image for instructing execution of the cooperation function is displayed as information on the cooperation function. By linking the multifunction machine (B) and the projector (C), for example, a cooperation function for projecting an image generated by scanning by the multifunction machine (B) by the projector (C), or being projected by the projector (C). The cooperative function of printing the existing image by the multifunction peripheral (B) can be executed. A button image for instructing execution of these linkage functions is displayed on the function display screen 74. For example, when the user designates a button image using the terminal device 16 and instructs execution of the cooperation function, execution instruction information indicating the execution instruction of the cooperation function is transmitted from the terminal device 16 to the image forming apparatus 10 and the projector 12. Sent. Upon receiving the execution instruction information, the image forming apparatus 10 and the projector 12 execute the cooperation function designated by the user.

  A device to be linked may be designated by a user operation. For example, when the image forming apparatus 10 and the projector 12 are photographed by the camera 42, as shown in FIG. 11, the UI unit 46 of the terminal device 16 has a device image 70 associated with the image forming apparatus 10 and the projector 12. An attached device image 72 is displayed. The image associated with the device may be an image obtained by photographing with the camera 42, or may be an image prepared in advance (for example, a schematic image (eg, icon)) associated with the identified device. May be. When the user designates device images 70 and 72 on the device display screen 68, the image forming apparatus 10 and the projector 12 are designated as devices to be linked. For example, when the user designates the device image 70, the marker type AR technology or the markerless type AR technology is applied to the device image 70, and thereby the image forming apparatus 10 is specified (identified). Similarly, when the user designates the device image 72, the marker type AR technology or the markerless type AR technology is applied to the device image 72, and thereby the projector 12 is specified (identified). Thereby, the cooperation function executed by the image forming apparatus 10 and the projector 12 is specified, and information related to the cooperation function is displayed on the UI unit 46 of the terminal device 16.

  As another example, when the user touches the device image 70 on the device display screen 68 and operates the device image 72 with a finger or the like (for example, tracing with the finger or the like), the device images 70 and 72 are designated. The image forming apparatus 10 and the projector 12 may be designated as devices to be linked. The order of touching the device images 70 and 72 and the direction of tracing may be the reverse of the above example. Of course, screen contact means other than a finger such as a pen for tracing the device display screen 68 may be used. The device to be linked may be specified by a drawing operation such as adding a circle, or the device to be linked can be specified by specifying the device image associated with the device to be linked within a preset time. Good. When canceling the cooperation, the user may specify a device to be canceled on the device display screen 68 or may press a cooperation cancellation button. When a device that is not a collaboration target is reflected in the image, the device may be excluded from the collaboration target device group by designating the device on the device display screen 68. The device to be released may be designated by a preset operation such as adding a cross mark.

  Each device to be linked may be shot separately. For example, each device to be linked is identified by dividing the shooting by the camera 42 into a plurality of times. When the shooting by the camera 42 is divided into a plurality of times, the device identification information of the device identified at each shooting time is stored in the storage unit of the server 14 or the terminal device 16. For example, the image forming apparatus 10 is photographed with the image forming apparatus 10 included in the photographing area of the camera 42, and then the projector 12 is photographed with the projector 12 included in the photographing area of the camera 42. Thereby, image data for the image forming apparatus 10 and image data for the projector 12 are generated, and the marker-type AR technique or the markerless-type AR technique is applied to each image data. Identification (identification) is performed, and a cooperation function that uses the functions of the image forming apparatus 10 and the projector 12 is identified (identification). For example, the devices to be linked are arranged at positions close to each other, and both the image forming apparatus 10 and the projector 12 are not necessarily included together in the shooting area of the camera 42. In some cases, it is possible to respond by changing the angle of the shooting area of the camera 42, or by expanding or reducing the shooting area. It is also possible to identify other devices.

  As another example, a device to be linked may be set in advance as a basic linked device. For example, it is assumed that the image forming apparatus 10 is preset as a basic cooperation device. The device identification information of the basic cooperation device may be stored in advance in the storage unit of the server 14 or the terminal device 16. The user may specify the basic cooperation device using the terminal device 16. When the basic cooperation device is set, the user uses the camera 42 of the terminal device 16 to photograph a device to be linked other than the basic cooperation device. As a result, the device to be linked is specified (identified), and the linked function that uses the functions of the basic linked device and the captured device is specified (identified).

  Note that if the device to be imaged is not identified even when the AR technology or the like is applied, the device image representing the device to be imaged may not be displayed on the device display screen. This improves the visibility of the identified device. For example, when an identified device and an unidentified device are mixed and each is photographed by the camera 42, the device image representing the unidentified device is hidden. Thereby, since the device image representing the identified device is displayed without being buried in the device image representing the unidentified device, the visibility of the identified device is improved. In addition, a device image representing the identified device may be highlighted and displayed. For example, the device image representing the identified device may be displayed in a specific color, the edge of the device image may be displayed highlighted, the device image may be displayed enlarged, The device image may be displayed in three dimensions, or the device image may be displayed with vibration. This improves the visibility of the identified device.

  Hereinafter, switching control between the display of a function that is used independently by a device (hereinafter referred to as a “single use function”) and the display of a linkage function will be described in detail.

  For example, when only one device is identified within a preset identification time, the control unit 48 of the terminal device 16 independently uses information on the function of the one device (for example, the image forming apparatus 10). Information on the function is displayed on the UI unit 46 of the terminal device 16. The length of the identification period may be changed by the user. The device may be identified by applying the AR technology, or the device may be identified by applying a technology other than the AR technology. The device identification process may be performed by the server 14 or the terminal device 16. The start point (starting point) of the identification time may be, for example, a time point when one device is identified, or may be a time point specified by the user (for example, the identification process start time point).

  For example, when another device is not identified within the identification time from the time when one device is identified, the control unit 48 of the terminal device 16 uses information regarding the function of the one device as information regarding the single use function. And displayed on the UI unit 46 of the terminal device 16. In this case, the one device is treated as a device identified within the identification period. The information related to the device may be information transmitted from the server 14 to the terminal device 16 or information stored in the terminal device 16 in advance.

  As another example, when only one device is identified within the identification time from the start time when the time specified by the user is the start time of the identification time, the control unit 48 of the terminal device 16 Information related to the functions possessed is displayed on the UI unit 46 of the terminal device 16 as information related to the single use device.

  As another example, when a display instruction of a single use function is given after one device is identified, the control unit 48 of the terminal device 16 displays information on the function of the one device regarding the single use function. Information may be displayed on the UI unit 46 of the terminal device 16. The control unit 48 of the terminal device 16 displays a button image for instructing the display of the single use function, for example, at all times or when one device is photographed (or when one device is identified). 8 is displayed on the device display screen 52 shown in FIG. When the button image is pressed by the user, the control unit 48 causes the UI unit 46 to display information related to the function of the one device.

  When the identification time has elapsed, the control unit 48 of the terminal device 16 may display a confirmation screen on the UI unit 46 of the terminal device 16. The confirmation screen is, for example, a screen for the user to instruct extension of the identification time. When the user instructs the extension of the identification time via the confirmation screen, when another device is not photographed within the extension time, the control unit 48 of the terminal device 16 displays information on the function of the identified device. It is displayed on the UI unit 46 of the terminal device 16.

  The display control of information related to the single use function will be described in more detail. For example, it is assumed that the device is identified by the marker type AR technology or the markerless type AR technology. For example, when only one device is photographed within a preset photographing time, the control unit 48 of the terminal device 16 uses the information regarding the function of the one device as information regarding the single use function. 16 UI parts 46 are displayed. The start point (starting point) of the shooting time may be the time when one device is shot, or may be the time point designated by the user (for example, the shooting start time). The length of the shooting time may be changed by the user. When the one device is photographed, the one device is identified by the marker type AR technology or the markerless type AR technology. This identification process may be performed by the server 14 or the terminal device 16.

  For example, when another device is not shot within the shooting time from the time when one device is shot, the control unit 48 of the terminal device 16 uses information related to the function of the one device as information related to the single use function. And displayed on the UI unit 46 of the terminal device 16. In this case, the one device is treated as a device photographed within the photographing time.

  As another example, when the time point specified by the user is set as the start time of the shooting time, and only one device is shot within the shooting time from the start time, the control unit 48 of the terminal device 16 causes the one device to Information related to the function that the user has is displayed on the UI unit 46 of the terminal device 16 as information related to the single use function.

  As another example, when a display instruction of a single use function is given after one device is photographed, the control unit 48 of the terminal device 16 displays information on the function of the one device regarding the single use function. Information may be displayed on the UI unit 46 of the terminal device 16.

  For example, when only one device is photographed within the photographing time (for example, when another device is not photographed within the photographing time from the time when the first device was photographed, or from the start time specified by the user) When only one device is photographed in time), the control unit 48 of the terminal device 16 transmits the image data generated by the photographing to the server 14. The shooting time may be measured by the control unit 48 or may be measured by a timer. The specifying unit 38 of the server 14 specifies (identifies) a device based on the image data, and specifies the function of the device. Information regarding the function is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. Of course, the terminal device 16 may not manage the time, but the server 14 may manage the time and transmit information about the functions of the identified device to the terminal device 16.

  When the shooting time has elapsed, the control unit 48 of the terminal device 16 may display a confirmation screen on the UI unit 46 of the terminal device 16. The confirmation screen is, for example, a screen for the user to instruct extension of the shooting time. When the user instructs to extend the shooting time via the confirmation screen, if another device is not shot within the extended time, the control unit 48 of the terminal device 16 sends the image data obtained by shooting to the server 14. In response, the function of the one device is displayed on the UI unit 46 of the terminal device 16. The length of the extension time may be changed by the user.

  As another example, the control unit 48 of the terminal device 16 transmits the image data generated by the shooting to the server 14 when a display instruction of the single use function is given after one device is shot. In response, information regarding the functions of the photographed device may be received from the server 14.

  As yet another example, the control unit 48 of the terminal device 16 transmits the image data to the server 14 each time the device is photographed and image data is generated, and the function of the photographed device accordingly. Information regarding the server 14 may be received. In this case, when only one device is photographed within the photographing time, the control unit 48 of the terminal device 16 displays information regarding the one device on the UI unit 46 of the terminal device 16 as information regarding the single use function. Let

  On the other hand, when a plurality of devices are identified within the identification time, the control unit 48 of the terminal device 16 displays information related to the cooperation function that uses the functions of the plurality of devices identified within the identification time. Are displayed on the UI part 46 of FIG. When the identification time is extended, the control unit 48 of the terminal device 16 displays information related to the cooperation function using the functions of a plurality of devices identified within the identification time including the extended time, as the UI unit of the terminal device 16. 46 is displayed. A plurality of devices identified within the identification time are designated as devices to be linked. Information related to the cooperation function may be information transmitted from the server 14 to the terminal device 16, or may be information stored in advance in the terminal device 16.

  For example, when another device is identified within the identification time from when the first device is identified, the control unit 48 of the terminal device 16 includes the first device and the other device. Information related to a cooperative function that uses functions of a plurality of devices is displayed on the UI unit 46 of the terminal device 16. In this case, the first device is also treated as a device identified within the identification time.

  In addition, when another device is identified within the identification time from the time when the first device is identified, the control unit 48 of the terminal device 16 newly starts from the time when the other device is identified. A different identification time may be set. The same applies to the subsequent processes. When another device is identified within the new identification time, a new identification time is set. In this case, the control unit 48 of the terminal device 16 uses the UI unit of the terminal device 16 to transmit information related to a cooperation function that uses functions of a plurality of devices (including the first device) identified within the entire identification time. 46 is displayed.

  As another example, when the time point specified by the user is set as the start time of the identification time, and a plurality of devices are photographed within the identification time from the start time, the control unit 48 of the terminal device 16 has the devices. Information related to the cooperation function that uses the function is displayed on the UI unit 46 of the terminal device 16.

  As another example, when a display instruction of a cooperation function is given after a plurality of devices are identified, the control unit 48 of the terminal device 16 displays information on the cooperation function that uses the functions of the plurality of devices. The information may be displayed on the UI unit 46 of the terminal device 16. The control unit 48 of the terminal device 16 displays, for example, a button image for instructing display of the cooperation function when a plurality of devices are photographed at all times (or when a plurality of devices are identified). 68. When the button image is pressed by the user, the control unit 48 causes the UI unit 46 to display information related to a cooperative function that uses the functions of the plurality of devices.

  As another example, when another device is identified after the first device is identified and an instruction to execute the function of the first device is not given, the control unit of the terminal device 16 48 may cause the UI unit 46 of the terminal device 16 to display information related to a cooperation function that uses the functions of the plurality of devices. For example, as shown in FIG. 9 or FIG. 10, in a state where the function display screens 56 and 58 are displayed on the UI unit 46 of the terminal device 16, another device may be connected before the execution of the function is instructed by the user. When the image is taken, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information related to the first identified device and a cooperation function that uses a function of the other device.

  The display control of information related to the cooperation function will be described in more detail. For example, it is assumed that a plurality of devices are identified by the marker type AR technology or the markerless type AR technology. For example, when a plurality of devices are photographed within a preset photographing time, the control unit 48 of the terminal device 16 obtains information on a cooperation function that uses a function of the plurality of devices photographed within the photographing time. And displayed on the UI unit 46 of the terminal device 16. When the shooting time is extended, the control unit 48 of the terminal device 16 displays information related to the cooperation function using the functions of a plurality of devices shot within the shooting time including the extended time, as the UI unit of the terminal device 16. 46 is displayed. A plurality of devices photographed within the photographing time are designated as cooperation target devices. When a plurality of devices are photographed, the plurality of devices are identified by the marker type AR technology or the markerless type AR technology. This identification process may be performed by the server 14 or the terminal device 16.

  For example, when another device is photographed within the photographing time from the time when the first device is photographed, the control unit 48 of the terminal device 16 includes the first device and the another device. Information related to a cooperative function that uses functions of a plurality of devices is displayed on the UI unit 46 of the terminal device 16. In this case, the first device is also treated as a device that has been photographed within the photographing time.

  In addition, when another device is photographed within the photographing time from the time when the first device is photographed, the control unit 48 of the terminal device 16 newly starts from the time when the other device is photographed. An appropriate shooting time may be set. The same applies to the following, and when another device is photographed within the new photographing time, a new photographing time is set. In this case, the control unit 48 of the terminal device 16 uses the UI unit of the terminal device 16 to transmit information related to the cooperation function that uses the functions of a plurality of devices (including the first device) captured within the entire imaging time 46 is displayed.

  As another example, when a plurality of devices are photographed within the photographing time from the start time when the time point specified by the user is the starting point of the photographing time, the control device 48 of the terminal device 16 has the plurality of devices. Information related to the cooperation function that uses the function is displayed on the UI unit 46 of the terminal device 16.

  As another example, when a display instruction of a cooperation function is given after a plurality of devices are photographed, the control unit 48 of the terminal device 16 displays information on the cooperation function that uses the functions of the plurality of devices. The information may be displayed on the UI unit 46 of the terminal device 16.

  As another example, when another device is photographed after the first device is photographed and an instruction to execute the function of the first device is not given, the control unit of the terminal device 16 48 may cause the UI unit 46 of the terminal device 16 to display information related to a cooperation function that uses the functions of the plurality of devices. For example, as shown in FIG. 9 or FIG. 10, in a state where the function display screens 56 and 58 are displayed on the UI unit 46 of the terminal device 16, before the user instructs execution of the function, another device is connected. When the image is photographed, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information related to the cooperation function that uses the function of the first imaged device and the other device.

  For example, when a plurality of devices are photographed within the photographing time (for example, when another device is photographed within the photographing time from the time when the first device is photographed, or from the start time specified by the user) When a plurality of devices are photographed in time), the control unit 48 of the terminal device 16 transmits the plurality of image data generated by the photographing to the server 14. The specifying unit 38 of the server 14 specifies (identifies) each device based on each image data, and specifies a cooperation function that uses functions of the plurality of identified devices. Information regarding the cooperation function is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. Of course, a function that each device has independently may be specified, and information on a function that each device has may be transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

  As another example, the control unit 48 of the terminal device 16 transmits a plurality of pieces of image data generated by the shooting to the server 14 when a display instruction for the cooperation function is given after a plurality of devices are shot. Accordingly, information related to a cooperation function that uses functions of a plurality of photographed devices may be received from the server 14.

  As another example, when another device is photographed after the first device is photographed and before an instruction to execute a function of the first device is given, the control unit 48 of the terminal device 16. May transmit a plurality of pieces of image data generated by the shooting to the server 14 and, in response thereto, may receive information from the server 14 regarding a cooperation function that uses functions of the plurality of devices that have been shot.

  As another example, the control unit 48 of the terminal device 16 transmits the image data to the server 14 each time the device is photographed and image data is generated, and the plurality of devices photographed accordingly have the image data. Information regarding the cooperative function that uses the function may be received from the server 14. In this case, when a plurality of devices are photographed within the photographing time, the control unit 48 of the terminal device 16 displays information related to the cooperation function that uses the functions of the plurality of devices on the UI unit 46 of the terminal device 16. Let

  Hereinafter, another display example of information regarding functions will be described.

  FIG. 13 shows a display example of the single use function. The control unit 48 of the terminal device 16 displays the device display screen 52 on the UI unit 46 of the terminal device 16, and the device display screen 52 displays a device image 54 representing a photographed device (for example, the image forming apparatus 10) Information on the functions of the device is displayed. In the example illustrated in FIG. 13, information (for example, the name of each function) related to the function of the image forming apparatus 10 is displayed in the display area 76 having a balloon shape. The shape of the display area 76 is merely an example, and a display area 76 having another shape may be displayed. When information related to the function displayed in the display area 76 is designated by the user and a function execution instruction is given, the designated function is executed.

  FIG. 14 shows a display example of the cooperation function. The control unit 48 of the terminal device 16 displays a device display screen 68 on the UI unit 46 of the terminal device 16, and the device display screen 68 represents a plurality of photographed devices (for example, the image forming apparatus 10 and the projector 12). Device images 70 and 72 are displayed. Furthermore, the control unit 48 of the terminal device 16 causes the device display screen 68 to display information related to the cooperation function that uses the functions of the plurality of devices. In the example illustrated in FIG. 14, information related to the cooperation function (for example, the name and description of the cooperation function) is displayed in the display area 78 having a balloon shape. The display area 78 is only an example, and a display area 78 having another shape may be displayed. When information related to the cooperation function displayed in the display area 78 is designated by the user and an instruction to execute the cooperation function is given, the designated cooperation function is executed.

  Further, the control unit 48 of the terminal device 16 may cause the device display screen 68 to display information related to the single use function of each cooperation target device. In the example shown in FIG. 14, information related to the function of the image forming apparatus 10 is displayed in the display area 80 having a balloon shape, and information related to the function of the projector 12 is displayed in the display area 82 having a balloon shape. It is displayed. The control unit 48 of the terminal device 16 may display information on the cooperation function and information on the single use function on the device display screen 68 separately. For example, the display area 78 may be displayed in a different color from the display areas 80 and 82, and the shape of the display area 78 may be different from the shape of the display areas 80 and 82. As another example, the display positions of the display areas 78, 80, and 82 may be different between the single use function and the cooperation function. For example, the display region 80 is displayed near the corresponding device image 70, the display region 82 is displayed near the corresponding device image 72, and the display region 78 is more device image 70, than the display regions 80, 82. It is displayed at a position away from 72. Thus, by displaying the display area 78 and the display areas 80 and 82 separately, the user can easily distinguish between the single use function and the cooperation function.

  The control unit 48 of the terminal device 16 may control the display of information related to the single use function and the display of information related to the cooperation function in accordance with a user instruction. For example, when the user designates a device image (for example, device image 70), the control unit 48 of the terminal device 16 obtains information on the functions of the device (for example, the image forming apparatus 10) associated with the designated device image. You may display on the apparatus display screen 68. FIG. When the user designates a device image and gives a display instruction for a single use function, the control unit 48 of the terminal device 16 displays information related to the function of the device associated with the designated device image on the device display screen 68. It may be displayed. When the user designates a plurality of device images (for example, device images 70 and 72), the control unit 48 of the terminal device 16 causes a plurality of devices (for example, the image forming apparatus 10 and the projector) associated with the designated plurality of device images. Information related to the linkage function that uses the function of 12) may be displayed on the device display screen 68. For example, after the first device image is designated by the user, when another device image is designated by the user within a preset time, the control unit 48 of the terminal device 16 associates the plurality of device images with the device image. Information related to a cooperation function that uses functions of a plurality of attached devices is displayed on the device display screen 68. As another example, when the user designates a plurality of device images and gives a display instruction for the cooperation function, the control unit 48 of the terminal device 16 has a plurality of devices associated with the designated plurality of device images. Information related to the cooperation function that uses the function may be displayed on the device display screen 68.

  As described above, according to the present embodiment, when one device is identified (for example, photographed), information on the function of the one device is displayed, and when a plurality of devices are identified (for example, photographed), Information related to a cooperative function that uses the functions of the plurality of devices is displayed. Accordingly, information on functions that can be executed using the identified (eg, photographed) device is provided to the user, which is convenient for the user.

  Also, by executing the linkage function, a function that cannot be executed by each individual device is executed, which is convenient.

  In addition, since the single use function and the cooperation function can be used only by identifying the device by applying the AR technology, it is simpler than the case where the user performs the setting for executing the function manually. Each function can be used in the work, reducing the user's trouble.

  Further, according to the present embodiment, for example, in an environment where a plurality of users use a plurality of devices, information related to the single use function and information related to the link function are appropriately displayed on the terminal device 16. For example, even when a user interface such as a touch panel is removed from the device, the terminal device 16 is used as the user interface. As another scene, for example, when a device is temporarily used at a user's destination, etc., a user interface suitable for the user, that is, a function of the device specified by the user and cooperation using the device A user interface on which the function is displayed is realized by the terminal device 16.

  In the above-described embodiment, the cooperation function using two devices has been described. Of course, the cooperation function may be a function using three or more devices.

  Further, even when a plurality of devices are identified (for example, photographed), the device specified by the user may be excluded from the devices to be linked.

(Modification 1)
Hereinafter, Modification 1 will be described. In the first modification, it is assumed that a plurality of devices are identified and information regarding the cooperation function is displayed. The device may be identified by using the AR technology as in the above-described embodiment, or may be identified by using a technology other than the AR technology. The device identification process may be performed by the server 14 or the terminal device 16.

  In the first modification, the control unit 48 of the terminal device 16 changes the cooperation function according to the display position of the device image on the device display screen 68 and causes the UI unit 46 of the terminal device 16 to display information related to the cooperation function. . For example, the order of use of each device used in the cooperation function is changed according to the display position of the device image, and the control unit 48 of the terminal device 16 provides information on the cooperation function according to the use order to the terminal device 16. Are displayed on the UI part 46 of FIG. In other words, according to the display position of the device image, the movement order of the data moving between the devices used in the cooperation function is changed, and the control unit 48 of the terminal device 16 performs the cooperation function according to the movement order. Information on the terminal device 16 is displayed on the UI unit 46. Hereinafter, Modification 1 will be described in detail.

  FIG. 15 shows a device display screen 68 according to the first modification. In the device display screen 68, for example, a start point display area 86 in which a device image associated with the device as a starting point is displayed by a boundary line 84, and a cooperation in which a device image associated with the next device to be linked is displayed. A pre-display area 88 is preset. As an example, the device display screen 68 is divided into right and left, a starting point display area 86 is set on the left side, and a cooperation destination display area 88 is set on the right side. Of course, the boundary line 84 and the character string indicating the display area (for example, “left” and “right”) may not be displayed. In the example shown in FIG. 15, the device image 70 is displayed in the starting point display area 86, and the device image 72 is displayed in the cooperation destination display area 88. Therefore, the image forming apparatus 10 (multifunction device (B)) associated with the device image 70 is designated as the starting device, and the projector 12 (projector (C)) associated with the device image 72 is designated as the cooperation destination device. . In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information related to a cooperation function that uses the image forming apparatus 10 as a starting device and uses the projector 12 as a cooperation destination device. . For example, the control unit 48 of the terminal device 16 uses the image forming apparatus 10 as a starting device first, and then uses the projector 12 as a cooperation destination device to obtain information on the cooperation function of the terminal device 16. It is displayed on the UI unit 46. In other words, the control unit 48 of the terminal device 16 displays, on the UI unit 46 of the terminal device 16, information related to the cooperation function that moves data from the image forming apparatus 10 as the starting device to the projector 12 as the cooperation destination device. Let The device images 70 and 72 may be images generated by photographing with a camera or may be schematic images (for example, icons).

  FIG. 16 shows a display example of information related to the cooperation function. A function display screen 90 is displayed on the UI unit 46 of the terminal device 16, and the function display screen 90 relates to a cooperation function that uses the image forming apparatus 10 as a starting device and uses the projector 12 as a cooperation destination device. Information is displayed. In the example shown in FIG. 16, the cooperation function moves image data obtained by scanning from the image forming apparatus 10 (multifunction device (B)) to the projector 12 (projector (C)), and the projector 12 displays the image. It is a function to project. Specifically, when the cooperation function is executed, the image forming apparatus 10 is used as a starting device, and image data generated by scanning by the image forming apparatus 10 is transmitted to the projector 12 as a cooperation destination device. Then, the image is projected by the projector 12.

  FIG. 17 shows another example of the device display screen 68 according to the first modification. On the device display screen 68, a starting point display area 92 and a cooperation destination display area 94 are set by a boundary line 84. Unlike the example shown in FIG. 15, in the device display screen 68, a starting point display area 92 is set on the right side, and a cooperation destination display area 94 is set on the left side. In this case, the projector 12 (projector (C)) associated with the device image 72 is designated as the starting device, and the image forming apparatus 10 (multifunction device (B)) associated with the device image 70 is designated as the cooperation destination device. The In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information related to a cooperation function that uses the projector 12 as a starting device and uses the image forming apparatus 10 as a cooperation destination device. . For example, the control unit 48 of the terminal device 16 uses the projector 12 as the starting device first, and then uses the image forming apparatus 10 as the cooperation destination device to obtain information regarding the cooperation function of the terminal device 16. It is displayed on the UI unit 46. In other words, the control unit 48 of the terminal device 16 displays, on the UI unit 46 of the terminal device 16, information related to the cooperation function that moves data from the projector 12 as the starting device to the image forming apparatus 10 as the cooperation destination device. Let

  FIG. 18 shows a display example of information related to the cooperation function. A function display screen 96 is displayed on the UI unit 46 of the terminal device 16, and the function display screen 96 relates to a cooperation function that uses the projector 12 as a starting device and uses the image forming apparatus 10 as a cooperation destination device. Information is displayed. In the example shown in FIG. 18, the cooperation function is a function for printing a projected image. Specifically, when the cooperation function is executed, the projector 12 is used as a starting device, and image data projected on the projector 12 is transmitted to the image forming apparatus 10 as a cooperation destination device. The image data is printed by the image forming apparatus 10.

  In the first modification, the terminal device 16 transmits the device image data displayed in the starting point display area 86 (92) to the server 14 as image data of the starting point device, and displays it in the cooperation destination display area 88 (94). The displayed device image data is transmitted to the server 14 as image data of the cooperation destination device. The specifying unit 38 of the server 14 treats the device associated with the image data of the origin device as the origin device, and treats the device associated with the image data of the collaboration destination device as the collaboration destination device. The server 14 stores data of the cooperation function management table according to the first modification. In the cooperation function management table, the device identification information of the originating device, the device identification information of the cooperation destination function, and the cooperation function identification are stored. Information is associated with each other. The specifying unit 38 of the server 14 has a cooperation function associated with the device identification information of the device specified as the starting device and the device identification information of the device specified as the cooperation destination device in the cooperation function management table. Identify identification information. For example, the device identification information of the image forming apparatus 10 as a starting device, the device identification information of the projector 12 as a cooperation destination device, and the cooperation function (the cooperation function shown in FIG. 16) for projecting an image obtained by scanning. Function identification information is associated with each other. Also, the cooperation function of the device identification information of the projector 12 as the starting device, the device identification information of the image forming apparatus 10 as the cooperation destination device, and the cooperation function (cooperation function shown in FIG. 18) for printing the projected image. Identification information is associated with each other. Therefore, by referring to the cooperation function management table according to the first modification, the cooperation function that uses the functions of the originating device and the cooperation destination device is specified.

  In the first modification, the display position of the device image may be changed. In this case, the device display screen on which the device image is displayed functions as an editing screen for changing the display position of the device image. In addition, the screen may be shifted to the editing screen in the shooting state, or the device image is held because the device image shooting has been completed in advance or the device image has been delivered. In this state, an edit screen for instructing the linkage function may be launched. Of course, the screen may be transferred to the editing screen by another method. For example, as illustrated in FIG. 19, it is assumed that the device image 70 is displayed in the start point display area 86 and the device image 72 is displayed in the cooperation destination display area 88. For example, when the user moves the device image 70 from the starting point display region 86 to the cooperation destination display region 88 in the direction indicated by the arrow 98, the control unit 48 of the terminal device 16 displays the device image 70 according to the movement instruction. The device image 70 is not displayed in the starting point display region 86 and displayed in the cooperation destination display region 88. As a result, the image forming apparatus 10 (multifunction device (B)) associated with the device image 70 is specified (identified) as a cooperation destination device. When the user moves the device image 72 from the cooperation destination display region 88 to the starting point display region 86 in the direction indicated by the arrow 100, the control unit 48 of the terminal device 16 displays the device image 72 according to the movement instruction. The device image 72 is not displayed in the cooperation destination display region 88 while being displayed in the start point display region 86. Thus, the projector 12 (projector (C)) associated with the device image 72 is specified (identified) as the starting device. When the user gives a display instruction for the cooperation function after the display position is changed, information regarding the cooperation function corresponding to the display position after the change may be displayed on the UI unit 46 of the terminal device 16 or the user may cooperate. Even if the display instruction of the function is not given, information on the cooperation function corresponding to the display position after the change may be displayed on the UI unit 46 of the terminal device 16 according to the change of the display position.

  In the first modification described above, the device display screen 68 is divided in the left-right direction. However, this is only an example, and the display screen 68 is divided in the vertical direction, and the start point display area and the cooperation destination display area are aligned in the vertical direction. May be displayed. As another example, the start point display area may be displayed near the center of the screen, and the link destination display area may be displayed so as to surround the start point display area. Conversely, the link destination display area is displayed at the center of the screen. The starting point display area may be displayed so as to be displayed in the vicinity and surround the cooperation destination display area.

  In the first modification, the cooperation function that uses two devices has been described. Of course, the cooperation function may be a function that uses three or more devices. For example, the control unit 48 of the terminal device 16 displays a plurality of cooperation destination display areas on the device display screen 68. The order of use is set in advance in each cooperation destination display area, and the order of use of each device (order of cooperation) is determined according to the use order. For example, the device associated with the device image displayed in the first cooperation destination display area is used as the first cooperation destination device, and the device associated with the device image displayed in the second cooperation destination display area. Is used as the second cooperation destination device. The cooperation function in this case is a function in which the starting device is used first, then the first cooperation destination device is used, and then the second cooperation destination device is used. In other words, the cooperation function is a function in which data moves in the order of the starting device, the first cooperation destination device, and the second cooperation destination device. The use order set in each cooperation destination display area may be changed by the user.

  As described above, according to the first modification, the cooperation function is changed according to the display position of the device image, so that the operability of changing the cooperation function is improved. Moreover, a cooperation function is changed by simple operation.

(Modification 2)
Hereinafter, Modification 2 will be described. In the second modification, as in the first modification, a plurality of devices are identified. In addition, according to the display position of the device image, either information related to the single use function or information related to the cooperation function is displayed.

  For example, when a plurality of device images are arranged in a specific direction (for example, the left-right direction) on the device display screen displayed on the UI unit 46 of the terminal device 16, the control unit 48 of the terminal device 16 The UI unit 46 of the terminal device 16 displays information related to a cooperation function that uses functions of a plurality of devices associated with the device image. For example, as shown in FIG. 20, when the device display screen 68 is divided into right and left by the boundary line 102, the device image 70 is displayed in the left region 104, and the device image 72 is displayed in the right region 106, the terminal device 16 control unit 48 causes UI unit 46 of terminal device 16 to display information related to a cooperation function that uses image forming apparatus 10 associated with device image 70 and projector 12 associated with device image 72. As in the first modification, one of the areas 104 and 106 may be used as the start point display area, and the other area may be used as the cooperation destination display area.

  When a plurality of device images are arranged on the device display screen in a direction (for example, the vertical direction) intersecting the specific direction, the control unit 48 of the terminal device 16 has a function that each device associated with each device image has. Information regarding (single use function) is displayed on the UI unit 46 of the terminal device 16. For example, as shown in FIG. 21, when the device display screen 68 is vertically divided by the boundary line 108, the device image 70 is displayed in the upper region 110, and the device image 72 is displayed in the lower region 112, the terminal The control unit 48 of the device 16 includes information on the function (single use function) of the image forming apparatus 10 associated with the device image 70 and information on the function (single use function) of the projector 12 associated with the device image 72. It is displayed on the UI unit 46 of the terminal device 16.

  The boundary lines 102 and 108 and the left and right and upper and lower character strings may be displayed or may not be displayed.

  For example, when the user gives an instruction to display the display area for the cooperation function using the UI unit 46 of the terminal device 16, the control unit 48 of the terminal device 16 displays the device display screen 68 as shown in FIG. Are divided into right and left to set areas 104 and 106. On the other hand, when the user gives an instruction to display the display area for the single use function, the control unit 48 of the terminal device 16 divides the device display screen 68 vertically as shown in FIG. 112 is set. Of course, even if the display instruction for the display area is not given by the user, when a plurality of device images are arranged in the left and right direction, information about the linkage function is displayed, and the plurality of device images are arranged in the up and down direction. In this case, information on the single use function may be displayed.

  Information regarding the single use function may be displayed when a plurality of device images are arranged in the left-right direction, and information regarding the cooperation function may be displayed when a plurality of device images are arranged in the vertical direction. Of course, when a plurality of device images are arranged in directions other than these, either information regarding the single use function or information regarding the cooperation function may be displayed according to the direction.

  As described above, according to the second modification, information on a single use function or information on a linkage function is displayed according to the display positions of a plurality of device images, so that the information can be switched with a simple operation. Displayed.

(Modification 3)
Hereinafter, Modification 3 will be described. In the third modification, when a device image associated with a photographed device does not satisfy a preset area standard, the device is not identified as a device to be used. For example, as illustrated in FIG. 22, it is assumed that the image forming apparatus 10 as the multifunction peripheral (B) is photographed by the camera 42. A planted tree as an obstacle is arranged on the front side of the image forming apparatus 10 (between the image forming apparatus 10 and the terminal device 16), and a device image 70 (for example, photographed by the camera 42) associated with the image forming apparatus 10 is disposed. The image 114 is overlaid with the plant image 114. That is, the device image 70 is missing due to the obstacle image. In this case, the image forming apparatus 10 is not identified as a device to be used. For example, appearance image data obtained by photographing the device is transmitted from the terminal device 16 to the server 14, and the specifying unit 38 of the server 14 specifies (identifies) the device based on the appearance image data. In this case, when the appearance image data includes a missing portion, the specifying unit 38 of the server 14 does not specify (identify) the device based on the appearance image data. As a result, a device (for example, the image forming apparatus 10) associated with the appearance image data including the missing portion is not identified as a device to be used.

  The obstacle is, for example, an object that is not identified as a device by the specifying unit 38 of the server 14. Since plants and the like are not registered as devices, even if appearance image data representing a plant is transmitted to the server 14, the plant is not identified as a device but is identified as an obstacle.

  The specifying unit 38 of the server 14 may specify (identify) a device based on the appearance image data according to the size of the area other than the missing portion in the appearance image data. For example, when the size of the region other than the missing portion in the appearance image data is equal to or larger than a preset threshold, the specifying unit 38 of the server 14 selects the device based on the region other than the missing portion in the appearance image data. It may be specified (identified). As another example, when the area other than the missing part is larger than the missing part, the specifying unit 38 of the server 14 may specify (identify) the device based on the area other than the missing part in the appearance image data. Good. As another example, when the device can be identified by applying the markerless AR technology to the appearance image data, even if the appearance image data includes a missing portion, the specifying unit 38 of the server 14 The device may be specified (identified) based on a region other than.

  When the image forming apparatus 10 (multifunction device (B)) is not identified as a device to be used, the control unit 48 of the terminal device 16 displays information regarding functions of other devices that have been photographed, and the UI unit 46 of the terminal device 16. To display. For example, the projector 12 as the projector (C) is photographed, the device image 72 associated with the projector 12 is displayed, the camera (D) is photographed, and the device image 116 associated with the camera (D) is displayed. It shall be. In this case, the control unit 48 of the terminal device 16 performs information related to the cooperation function using the functions of the projector (C) and the camera (D) or the projector (C) and the camera ( D) Information related to the functions (single use functions) of the terminal device 16 is displayed on the UI unit 46 of the terminal device 16.

  In addition, a cooperation function may be constructed | assembled using the function linked | related with the image part other than a defect | deletion part in external appearance image data. For example, in the example illustrated in FIG. 22, when the main body of the multifunction peripheral (B) (image forming apparatus 10) is represented in the device image 70, the control unit 48 of the terminal device 16 performs processing according to the above-described embodiment. Accordingly, the information related to the cooperation function that uses the function of the main body unit or the information about the function (single use function) of the main body unit may be displayed on the UI unit 46 of the terminal device 16. For example, for each device part, an appearance partial image representing the part and a function of the part are associated with each other, and device function information 32 indicating the association is stored in the storage unit 30 of the server 14. It is remembered. By referring to the device function information 32, the function associated with the image part other than the missing part is specified (identified). In addition, cooperation function information 34 indicating a cooperation function that uses functions of a plurality of parts is stored in the storage unit 30 of the server 14. Of course, the link function information 34 may include information indicating a link function that uses the function of the device part and the function of the entire device. By referring to the cooperation function information 34, a cooperation function that uses a function associated with an image part other than a missing part is specified (identified).

  When the device image is missing, the specifying unit 38 of the server 14 specifies (identifies) the device associated with the device image by using a plurality of identification technologies such as the marker type AR technology and the position information AR technology. Thus, the function of the device and the cooperation function that uses the function of the device may be specified (identified).

  As described above, according to the third modification, a device associated with a missing device image is excluded from devices to be used. As a result, the device associated with the missing device image is prevented from being identified as a device that is not assumed by the user when the user does not want the device to be used. It is possible to prevent the information related to the function of the device and the information related to the cooperation function that uses the function of the device from being displayed.

(Modification 4)
Hereinafter, Modification 4 will be described. In the fourth modification, as in the first modification, a plurality of devices are identified. In addition, the device used as the starting point of the cooperation function is changed according to the positional relationship between the photographing device that performs photographing of the device and each device.

  For example, a device (photographed device) installed at a position closer to the photographing device (for example, the terminal device 16 including the camera 42) is preferentially used as a device for starting the cooperation function. In this case, the specifying unit 38 of the server 14 acquires information indicating the distance between the terminal device 16 and each device, and prioritizes each device based on the distance. The position information of the terminal device 16 and each device is acquired by using, for example, a GPS (Global Positioning System) function. Of course, the position information may be acquired by another technique. For example, the terminal device 16 and each device have a GPS function, and acquire their own position information. The terminal device 16 acquires position information from each device, and transmits the position information of the terminal device 16 and each device to the server 14. The specifying unit 38 of the server 14 calculates the distance between the terminal device 16 and each device based on the position information of the terminal device 16 and each device. For example, the device with the shortest distance to the terminal device 16 is used as the first (highest priority) device, and the device with the second shortest distance to the terminal device 16 is used as the second device. Used. In this case, the specifying unit 38 of the server 14 specifies (identifies) the cooperation function to be used in order from the function of the high priority device, and the control unit 48 of the terminal device 16 has the function of the high priority device. Are linked to the UI unit 46 of the terminal device 16 with priority.

  For example, as shown in FIG. 23, an image forming apparatus 10 as a multifunction peripheral (B), a projector 12 as a projector (C), a camera (D), and a display (E) are taken by a camera 42. To do. In this case, there are a device image 70 associated with the multifunction peripheral (B), a device image 72 associated with the projector (C), a device image 116 associated with the camera (D), and a device image 118 associated with the display (E). Are displayed on the UI unit 46 of the terminal device 16. For example, when it is determined that the projector (C), the camera (D), the multifunction device (B), and the display (E) are close to the terminal device 16 in this order, the projector (C), the camera (D), the composite The cooperation function to be used is identified (identified) in the order of the machine (B) and the display (E), and the cooperation function is preferentially displayed on the UI unit 46 of the terminal device 16.

  As another example, a device associated with a device image displayed at a position closer to the reference position 120 set on the device display screen 68 may be preferentially used as a device that starts the linkage function. In this case, the terminal device 16 transmits information indicating the distance (for example, the length of each arrow shown in FIG. 23) between the reference position 120 and each device image to the server 14. The specifying unit 38 of the server 14 gives priority to each device associated with each device image based on the distance between the reference position 120 and each device image. For example, the device associated with the device image having the shortest distance from the reference position 120 is used as the first (highest priority) device, and is associated with the device image with the second shortest distance from the reference position 120. The device is used as a second device. In this case, the specifying unit 38 of the server 14 specifies (identifies) the cooperation function to be used in order from the function of the high priority device, and the control unit 48 of the terminal device 16 determines the cooperation function of the terminal device 16. It is displayed on the UI unit 46. Note that the control unit 48 of the terminal device 16 may display or display a mark representing the reference position 120 (for example, a mark having a round shape) or an arrow representing the distance between images on the device display screen 68. It does not have to be.

  The control unit 48 of the terminal device 16 may display information indicating the priority order (for example, a number corresponding to the order of the positions close to the terminal device 16) on the device display screen 68 in association with each device image. For example, information (for example, a number) indicating the priority order is displayed near each device image. In the example shown in FIG. 23, a number indicating the first (highest priority) is displayed in the vicinity of the device image 72 associated with the projector (C). The same applies to other device images.

  As described above, according to the fourth modification, the cooperation function that is preferentially used as the function of the device close to the photographing device (for example, the terminal device 16) is preferentially displayed. As a result, when the user wants to use the devices in order from the device installed at a position close to the photographing device (for example, the terminal device 16), information on the linkage function that matches the user's request is displayed. Convenient.

  The positional relationship between the devices may be determined based on the missing relationship between the device images (appearance images) associated with the devices. For example, as shown in FIG. 24, a device image 70 associated with the multifunction peripheral (B) is displayed superimposed on a device image 118 associated with the display (E) (for example, an appearance image obtained by photographing with the camera 42). If it is determined that the multifunction device (B) is installed at a position closer to the terminal device 16 than the display (E). This determination may be performed by the terminal device 16 or may be performed by the server 14. In this case, the specifying unit 38 of the server 14 specifies (identifies) a cooperation function that uses each device in the order of the multifunction peripheral (B) and the display (E), and the control unit 48 of the terminal device 16 uses the cooperation function. The information regarding is preferentially displayed on the UI unit 46 of the terminal device 16.

  Since the device image 70 is superimposed on the device image 118, the device image 118 is missing. In this case, the specifying unit 38 of the server 14 specifies (identifies) the display (E) associated with the device image 118 by using a plurality of identification technologies such as the marker-type AR technology and the position information AR technology. The function that the display (E) has and the cooperation function that uses the function that the display (E) has may be specified (identified).

(Modification 5)
Hereinafter, Modification 5 will be described. In the modified example 5, functions are assigned to each part of the device, information on functions assigned to the identified (eg, photographed) part, and functions assigned to the identified (eg, photographed) part. Information related to the linkage function that uses is displayed. Hereinafter, Modification 5 will be described in detail.

  FIG. 25 shows an example of a device function management table according to Modification 5. Data of the device function management table according to Modification 5 is stored in the server 14 as device function information 32. In the device function management table according to the modified example 5, as an example, the device ID, information indicating a device name (for example, the type of device), information indicating the name of the device part (for example, the type of portion), and the portion A part ID as part identification information for identifying the part, information indicating a function assigned to the part (function of the part), and a part image ID for identifying a device part image associated with the part Are associated with each other. The device partial image is an image representing the appearance of the device portion obtained by photographing with the camera. Of course, a partial image that schematically represents the part of the device may be associated with the part. For example, a different function is assigned to each part of the device.

  A specific example will be described. A screen display function is assigned to the display unit of PC (A), and information indicating the screen display function corresponds to the partial image ID of the device partial image associated with the display unit. It is attached. The screen display function is a function for displaying information on the PC (A). A data storage function is assigned to the main body of the PC (A), and information indicating the data storage function is associated with the partial image ID of the device partial image associated with the main body. The data storage function is a function for storing data in the PC (A).

  Also, a print function is assigned to the main body of the multifunction peripheral (B), and information indicating the print function is associated with the partial image ID of the device partial image associated with the main body. A scanning function is assigned to the reading unit of the multifunction device (B) (for example, a portion corresponding to the document cover, document glass, or automatic document feeder of the multifunction device (B)), and a device associated with the reading unit. Information indicating the scan function is associated with the partial image ID of the partial image. The post-processing device of the multifunction peripheral (B) is assigned a stapling function, and information indicating the stapling function is associated with the partial image ID of the device partial image associated with the processing device thereafter. The stapling function is a function of binding the output paper by stapling.

  The function assigned to the device part is specified (identified) by using, for example, a markerless AR technology. For example, when a device portion is photographed by a camera (for example, the camera 42 of the terminal device 16), appearance image data representing the portion is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies (identifies) a function associated with the appearance image data in the device function management table. Thereby, the function assigned to the photographed part is specified (identified). For example, when the main body of the multifunction peripheral (B) is photographed by the camera 42, appearance image data representing the main body of the multifunction peripheral (B) is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies the print function associated with the appearance image data in the device function management table. Thereby, it is specified that the function assigned to the main body of the multifunction peripheral (B) is the print function.

  Of course, the function assigned to the part of the device may be specified (identified) by using the marker type AR technology. For example, each part of the device is provided with a marker such as a two-dimensional barcode in which part identification information (for example, part ID) for identifying the part is encoded. When a marker provided in a part is photographed by a camera and the marker type AR technology is applied, partial identification information (for example, a partial ID) of the part is acquired. The application of the marker type AR technology may be performed by the terminal device 16 or the server 14. When the partial identification information is acquired in this way, the specifying unit 38 of the server 14 specifies (identifies) the function associated with the partial identification information (for example, the partial ID) in the device function management table.

  FIG. 26 illustrates an example of a cooperation function management table according to the fifth modification. Data of the cooperation function management table according to the modification 6 is stored in the server 14 as cooperation function information 34. The cooperation function management table according to Modification 5 is information indicating a cooperation function that uses functions of a plurality of parts. In the cooperation function management table, as an example, information indicating a combination of device parts, and a part Information indicating a combination of IDs and information indicating a cooperation function that uses functions of a plurality of parts included in the combination are associated with each other. Of course, in the linkage function management table, information indicating the combination of the device portion and the entire device, and information indicating the linkage function using the function of the device portion and the function of the entire device are: It may be associated.

  A specific example will be described. A print function as a cooperation function is assigned to the combination of the display unit of the PC (A) and the main unit of the multifunction peripheral (B). Information indicating the combination of the partial ID and the partial ID of the main body of the multifunction peripheral (B) is associated with information indicating a print function as a cooperation function. The print function as the linkage function is a function that, for example, transmits data stored in the PC (A) to the multifunction device (B) and prints the data by the multifunction device (B).

  A print function as a cooperation function is assigned to the combination of the main body of the multifunction peripheral (B) and the main body of the projector (C). The information indicating the print function as the cooperation function is associated with the information indicating the combination with the partial ID of the main body portion of C). The print function as the cooperation function is, for example, a function of transmitting data projected by the projector (C) to the multifunction device (B) and printing the data by the multifunction device (B).

  Further, a scan projection function as a cooperation function is assigned to the combination of the reading unit of the multifunction device (B) and the main body of the projector (C), and the partial ID of the reading unit of the multifunction device (B) and the projector The information indicating the combination with the partial ID of the main body part in (C) is associated with information indicating the scan projection function as the cooperation function. The scan projection function as the cooperation function is a function of, for example, transmitting data generated by scanning by the multifunction peripheral (B) to the projector (C) and projecting the data by the projector (C).

  Note that the linkage function may be a function that uses a function of a plurality of parts included in the same device, or a function that uses a function of a part of a plurality of different devices. In addition, the cooperation function may be a function that uses functions of three or more parts.

  For example, when a plurality of parts of a device (for example, a plurality of parts of a plurality of different devices or a plurality of parts of the same device) are specified (identified) using the marker type AR technology or the markerless type AR technology. The specifying unit 38 of the server 14 specifies (identifies) a cooperative function associated with a combination of a plurality of identified parts in the cooperative function management table. Thereby, the cooperation function using the function which the some part identified (for example, imaging | photography) has is pinpointed (identification). For example, the main body of the multifunction peripheral (B) and the main body of the projector (C) are photographed by the camera 42 of the terminal device 16, and the main body of the multifunction peripheral (B) and the main body of the projector (C) are identified. In this case, the specifying unit 38 of the server 14 specifies a print function or the like as a link function associated with a combination of the main body unit of the multifunction peripheral (B) and the main body unit of the projector (C) in the link function management table. To do.

  Also in the modification 5, the information regarding a single use function or the information regarding a cooperation function may be displayed on the UI part 46 of the terminal device 16 similarly to embodiment mentioned above. For example, when the display instruction of the single use function is given by the user or only one part in the device is identified, the control unit 48 of the terminal device 16 displays information on the function of the identified one part. The information may be displayed on the UI unit 46 of the terminal device 16. On the other hand, when a plurality of parts of a plurality of devices are identified, a plurality of parts in the same device are identified, or a display instruction of a cooperation function is given after the plurality of parts are identified, the terminal device 16 The control unit 48 may cause the UI unit 46 of the terminal device 16 to display information related to the cooperation function that uses the functions of the plurality of identified parts.

  Hereinafter, Modification 5 will be described with a specific example. For example, the main body of the multifunction peripheral (B) as the image forming apparatus 10 is photographed by the camera 42 of the terminal device 16, and the main body of the multifunction peripheral (B) is identified by the marker type AR technology or the markerless type AR technology ( Identification). In this case, for example, as shown in FIG. 27, a device display screen 52 is displayed on the UI unit 46 of the terminal device 16, and a device partial image 121 associated with the main unit of the multifunction peripheral (B) is displayed on the device display screen 52. Is displayed. When the reading unit and the post-processing device of the multifunction peripheral (B) are not identified, the device partial image 122 associated with the reading unit and the device partial image 124 associated with the post-processing device are not displayed on the device display screen 52.

  For example, when the display instruction of the single use function is given by the user, or only one part is photographed without photographing a plurality of parts, the control unit 48 of the terminal device 16 controls the main body of the multifunction device (B). Information related to the function of the unit is displayed on the UI unit 46 of the terminal device 16 as information related to the single use function. Note that the function identification processing of the main unit of the multifunction peripheral (B) may be performed by the terminal device 16 or the server 14. In the example shown in FIG. 27, information (for example, the names of the functions) related to the functions of the main body of the multifunction peripheral (B) is displayed in the display area 126 provided in the device display screen 52. When information regarding the function displayed in the display area 126 is designated by the user and an instruction to execute the function is given, the designated function is executed. The display area 126 is arranged in the vicinity of the device partial image 121 associated with the main body of the multifunction peripheral (B). Of course, the control unit 48 of the terminal device 16 displays a function display screen different from the device display screen 52 on the UI unit 46 of the terminal device 16, and the main unit of the multifunction peripheral (B) has the function display screen. Information regarding the function may be displayed.

  When a device image representing the entire device is displayed on the UI unit 46 of the terminal device 16, information regarding a function corresponding to a portion designated by the user in the device image is information regarding a single use function. It may be displayed on the UI unit 46 of the terminal device 16. For example, different functions are assigned to each position in the device image, and information regarding each function corresponding to each position is stored in the server 14 as device function information 32. Information indicating the position specified by the user in the device image is transmitted from the terminal device 16 to the server 14, and the specifying unit 38 of the server 14 specifies (identifies) the function corresponding to the specified position. Information regarding the function is displayed on the UI unit 46 of the terminal device 16 as a single use function. As another example, a device image representing the entire device may be composed of a plurality of device partial images representing a plurality of portions constituting the device. As described above, since a function is assigned to the device partial image, a function corresponding to the device partial image designated by the user is specified, and information related to the function is displayed on the UI unit 46 of the terminal device 16. .

  When the entire device is photographed, the control unit 48 of the terminal device 16 displays information on one or more functions of the device on the UI unit 46 of the terminal device 16, and a part of the device is photographed. In this case, information related to the function assigned to a part of the information may be displayed on the UI unit 46 of the terminal device 16. For example, when the entire multifunction device (B) is photographed, as shown in FIG. 9, information on the functions of the multifunction device (B) is displayed, and when the main body of the multifunction device (B) is photographed, As shown in FIG. 27, information regarding the functions of the main body may be displayed.

  Further, the main body of the multifunction peripheral (B) as the image forming apparatus 10 and the main body of the projector (C) as the projector 12 are photographed by the camera 42 of the terminal device 16 and are subjected to the marker type AR technique or the markerless type AR technique. Assume that the main body of the multifunction peripheral (B) and the main body of the projector (C) are specified (identified). In this case, for example, as shown in FIG. 28, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and a device partial image 121 associated with the main unit of the multifunction peripheral (B) is displayed on the device display screen 68. A device partial image 128 associated with the main body of the projector (C) is displayed.

  For example, the display instruction of the cooperation function is given by the user, a plurality of parts are photographed within the photographing time, or the plurality of parts are photographed before the execution instruction of the function assigned to one part is given. In this case, the control unit 48 of the terminal device 16 displays information related to the cooperation function that uses the function of the main unit of the multifunction peripheral (B) and the function of the main unit of the projector (C) as the UI unit of the terminal device 16. 46 is displayed. In addition, the identification process of a cooperation function may be performed by the terminal device 16, and may be performed by the server 14. FIG. In the example illustrated in FIG. 28, information related to the cooperation function (for example, the name of the cooperation function) is displayed in the display area 130 provided in the device display screen 68. When information regarding the function displayed in the display area 130 is designated by the user and an instruction to execute the cooperation function is given, the designated cooperation function is executed. In the example shown in FIG. 28, information related to the function of the main body of the multifunction peripheral (B) is displayed in the display area 132, and information related to the function of the main body of the projector (C) is displayed in the display area 134. Has been. The display area 132 is disposed in the vicinity of the device partial image 121 associated with the main body of the multifunction peripheral (B), and the display area 134 is disposed in the vicinity of the device partial image 128 associated with the main body of the projector (C). ing. Of course, the control unit 48 of the terminal device 16 displays a function display screen different from the device display screen 68 on the UI unit 46 of the terminal device 16, and information related to the cooperation function or information related to the single use function is displayed in the function display screen. May be displayed. In addition, the display areas 132 and 134 may not be displayed, and display and non-display may be switched according to a user instruction.

  Note that the control unit 48 of the terminal device 16 may display on the device display screen 68 the name of a function (for example, printing or scanning) of the device associated with the device partial image. As a result, the user is provided with easy-to-understand information on what function is assigned to each part. Of course, the name of the function may not be displayed.

  As described above, according to the modified example 5, when a part of the device is identified, information regarding the function of the part is displayed, and when a plurality of parts are identified, the functions of the plurality of parts are displayed. Information about the linkage function to be used is displayed. For example, by identifying (for example, shooting) a part to be used in a device, information on the functions of the part is displayed, so that it is unnecessary for the user compared to the case of displaying information on all the functions of the device. This is convenient because it is difficult to display various information. Similarly, by identifying (for example, photographing) a plurality of parts, information related to a cooperative function that uses the functions of the plurality of parts is displayed, which makes it difficult to display unnecessary information for the user. is there.

(Modification 6)
Hereinafter, Modification 6 will be described. Modification 6 is a combination of Modifications 1 and 5. For example, it is assumed that the main body of the multifunction peripheral (B) and the main body of the projector (C) are photographed. In this case, for example, as shown in FIG. 29, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and a device partial image 121 associated with the main unit of the multifunction peripheral (B) is displayed on the device display screen 68. A device partial image 128 associated with the main body of the projector (C) is displayed. Similar to the first modification, a starting point display area 86 and a cooperation destination display area 88 are set on the device display screen 68. In this case, the part of the device associated with the device partial image displayed in the starting point display area 86 is designated as the starting point, and the part of the device associated with the device partial image displayed in the cooperation destination display area 88 is linked. Specified as the destination. In the example shown in FIG. 29, since the device partial image 121 is displayed in the starting point display area 86, the main body of the multifunction peripheral (B) is designated as the starting point, and the device partial image 128 is displayed in the cooperation destination display area 88. Therefore, the main body of the projector (C) is designated as the cooperation destination. In this case, the control unit 48 of the terminal device 16 uses the main unit of the multifunction peripheral (B) as a starting point and uses the main unit of the projector (C) as the cooperation destination to obtain information on the cooperation function. This is displayed on the unit 46. For example, the control unit 48 of the terminal device 16 uses the main body of the multifunction peripheral (B) as a starting point first, and then, the information on the cooperation function that uses the main body of the projector (C) as a cooperation destination. And displayed on the UI unit 46 of the terminal device 16. In other words, the control unit 48 of the terminal device 16 uses the UI unit 46 of the terminal device 16 to transmit information related to the cooperation function that moves data from the main body of the multifunction peripheral (B) as the starting point to the projector (C) as the cooperation destination. To display.

  Further, as in the first modification, the display position of the device partial image may be changed. For example, when the user moves the device partial image 121 to the cooperation destination display area 88, the main unit of the multifunction peripheral (B) associated with the device partial image 121 is handled as a portion of the cooperation destination device. Similarly, when the user moves the device partial image 128 to the starting point display area 86, the main body of the projector (C) associated with the device partial image 128 is treated as the starting device portion.

  Similarly to the first modification, the terminal device 16 transmits the device partial image data displayed in the starting point display area 86 to the server 14 as image data of the starting point device portion, and is displayed in the cooperation destination display area 88. The device partial image data being transmitted is transmitted to the server 14 as image data of the cooperation destination device portion. The specifying unit 38 of the server 14 sets the device portion associated with the image data of the starting device portion as the starting point, and sets the device portion associated with the image data of the cooperation destination device portion as the cooperation destination. The server 14 stores data of a cooperation function management table according to the modified example 6. In the cooperation function management table, the identification information of the part of the starting device, the identification information of the part of the cooperation destination device, and the cooperation Is associated with function identification information. The specifying unit 38 of the server 14 specifies the link function identification information associated with the identification information of the part of the originating device and the identification information of the part of the link destination device in the link function management table. When the cooperation function identification information is specified in this way, information related to the specified cooperation function is displayed on the UI unit 46 of the terminal device 16.

  Note that the display position of the device partial image may be changed by separating the device partial image representing the part from the device image representing the entire device. For example, as shown in FIG. 30, it is assumed that a device image 70 representing the entire MFP (B) is displayed in the start point display area 86. In this case, as indicated by an arrow 136, when the user moves the device partial image 122 included in the device image 70 from the starting point display region 86 to the cooperation destination display region 88, the function associated with the device partial image 122 has a function. (For example, the scanning function of the reading unit) is set as a function of the cooperation destination. Similarly, when the device partial image included in the device image displayed in the cooperation destination display area 88 is moved to the starting point display area 86 by the user, the function associated with the device partial image has the function of the starting point. Set as

  As described above, according to the modified example 6, since the cooperation function is changed according to the display position of the device partial image, the operability of changing the cooperation function is improved. Moreover, a cooperation function is changed by simple operation.

(Modification 7)
Hereinafter, Modification 7 will be described with reference to FIGS. 31 and 32. 31 and 32 show a device display screen 68 according to Modification 7. In the modified example 7, as in the modified example 1 described above, the starting point display area 86 and the cooperation destination display area 88 are set on the device display screen 68, and a plurality of device images are included in the same cooperation destination display area 88. (For example, device images 72, 116, and 118) are displayed. A device image 70 is displayed in the starting point display area 86. In this case, the order of use of each device associated with each device image displayed in the cooperation destination display area 88 (order of cooperation) may be specified by the user, or the relationship between the display positions of the device images. May be set automatically based on

  When setting the order of cooperation according to a user's instruction, for example, the user designates each device image displayed in the cooperation destination display area 88 in order. For example, when each device image is designated by the user in the order of the device image 72 associated with the projector (C), the device image 118 associated with the display (E), and the device image 116 associated with the camera (D), After the multifunction device (B) associated with the device image 70 displayed in the display area 86 is first used, the cooperation destination devices are linked (used) in the designated order.

  When automatically setting the cooperation order, the cooperation order may be assigned to each device in the order closer to the device image 70 displayed in the start point display area 86. In the example illustrated in FIG. 31, the device image 72, the device image 118, and the device image 116 are displayed in the cooperation destination display area 88 in order from the device image 70. In this case, after the multifunction device (B) is first used, the devices are linked (used) in the order of the projector (C), the display (E), and the camera (D).

  As another example, the device associated with the device image displayed in the position closest to the device image 70 displayed in the start point display area 86 is set as the first device of the cooperation destination, and the same cooperation destination display. Based on the positional relationship between the device images displayed in the area 88, the cooperation order of the second and subsequent cooperation destination devices may be sequentially determined. For example, as shown in FIG. 32, an image associated with the device image 72 displayed at the position closest to the device image 70 is set as the first device to be linked. Next, in the cooperation destination display area 88, the camera (D) associated with the device image 116 displayed at the position closest to the device image 72 is set as the second cooperation destination device. Next, in the cooperation destination display area 88, the device image other than the device image (for example, the device image 72) associated with the device already set as the cooperation destination is displayed at the position closest to the device image 116. The display (E) associated with the device image 118 is set as the third cooperation destination device. The same applies to the cooperation order after that.

  As described above, when a plurality of device images are displayed in the same cooperation destination display area 88, the cooperation order is automatically set based on the relationship between the display positions of the device images. Convenient and saves time. Even when a plurality of device partial images are displayed in the same cooperation destination display area 88, the cooperation order of the portions of each device is determined by the same processing as described above. Of course, the device partial image may be displayed in the starting point display area 86.

  In the embodiment and the modification described above, the linkage function may be a function that uses a combination of all the devices, a function that uses a combination of parts of the devices, or a device. It is also possible to use a function that uses a combination of the entire device and the device portion.

  In addition, the function of each modification may be switched in accordance with a user instruction. For example, when the user instructs the execution of the function of the first modification, the function of the first modification may be executed, and thereby the starting point display area and the cooperation destination display area may be set on the device display screen. The same applies to other modified examples. When the user instructs execution of the function of the modified example, the function according to the modified example is executed.

  Hereinafter, in the embodiment and the modification described above, a process when executing a function of the device will be described. As an example, a process when executing the cooperation function will be described. In this case, a connection request is transmitted from the terminal device 16 to the cooperation target device, and the terminal device 16 and the cooperation target device are connected. Hereinafter, this connection process will be described with reference to FIG. FIG. 33 is a sequence diagram showing the processing.

  First, the user uses the terminal device 16 to instruct activation of an application (program) for executing a device function. The control unit 48 of the terminal device 16 activates the application according to the instruction (S01). The application may be stored in advance in the storage unit 44 of the terminal device 16 or may be downloaded from the server 14 or the like.

  Next, by applying the marker type AR technology, the markerless type AR technology, or the position information AR technology, a device to be linked is identified (S02). Of course, a device to be linked may be identified by a technology other than the AR technology. When applying the marker type AR technology or the markerless type AR technology, the user photographs the device to be linked by the camera 42 of the terminal device 16. As an example, when the image forming apparatus 10 (multifunction machine (B)) and the projector 12 (projector (C)) are used as linked devices, the user photographs the image forming apparatus 10 and the projector 12 with the camera 42. Device identification information of the image forming apparatus 10 and the projector 12 is acquired, and the image forming apparatus 10 and the projector 12 are identified as devices to be linked. When the position information AR technology is applied, the position information of each of the image forming apparatus 10 and the projector 12 is acquired, and the device identification information of each of the image forming apparatus 10 and the projector 12 is specified based on the position information. The forming apparatus 10 and the projector 12 are identified.

  As described above, when a plurality of devices are identified, an instruction to display a cooperation function is given, or another device is identified before an instruction to execute a single use function of one device is given, A linkage function that uses the function of the device is identified. The UI unit 46 of the terminal device 16 displays information regarding the identified cooperation function. (S03). The identification process of the cooperation function may be performed by the server 14 or the terminal device 16.

  Next, in the terminal device 16, when the cooperation function to be executed is designated by the user, the terminal device 16 sends a connection request to the cooperation target devices (for example, the image forming apparatus 10 and the projector 12) that execute the cooperation function. The information shown is transmitted (S04). For example, when the address information indicating the address of the cooperation target device is stored in the server 14, the terminal device 16 acquires the address information of the cooperation target device from the server 14. When the address information is included in the device identification information, the terminal device 16 may acquire each address information from the device identification information of the cooperation target device. As another example, the terminal device 16 may store address information of devices to be linked. Of course, the terminal device 16 may acquire the address information of the cooperation target device by another method. The terminal device 16 transmits information indicating a connection request to the cooperation target devices (for example, the image forming apparatus 10 and the projector 12) using the address information of the cooperation target devices (for example, the image forming apparatus 10 and the projector 12).

  Upon receiving the information indicating the connection request, the image forming apparatus 10 and the projector 12 permit or do not permit connection with the terminal device 16 (S05). For example, when the image forming apparatus 10 and the projector 12 correspond to devices that are not permitted to be connected, or when the number of devices that request connection exceeds the upper limit, the connection is not permitted. When the connection from the terminal device 16 is permitted, the change operation may be prohibited so that the setting information unique to the image forming apparatus 10 and the projector 12 is not changed from the terminal device 16. For example, it may be prohibited to change the color parameters of the image forming apparatus 10 or the set time when shifting to power saving. As a result, the security for the cooperation target device is improved. As another example, when the devices are linked, the change of the setting information may be limited as compared to the case where the devices are used alone without being linked. For example, fewer setting item changes may be permitted as compared to the case where the device is used alone. In addition, browsing of personal information of other users such as operation history may be prohibited. Thereby, the security with respect to a user's personal information improves.

  Result information indicating permission or non-permission of the connection is transmitted from the image forming apparatus 10 and the projector 12 to the terminal device 16 (S06). When the connection to the image forming apparatus 10 and the projector 12 is permitted, communication is established between the terminal device 16 and the image forming apparatus 10 and the projector 12.

  Next, the user uses the terminal device 16 to instruct execution of the cooperation function (S07). In response to this instruction, execution instruction information indicating an instruction to execute the cooperation function is transmitted from the terminal device 16 to the image forming apparatus 10 and the projector 12 (S08). The execution instruction information transmitted to the image forming apparatus 10 includes information (for example, job information) indicating processing executed in the image forming apparatus 10, and the execution instruction information transmitted to the projector 12 includes the projector 12. Includes information (for example, job information) indicating the process executed in step.

  Receiving the execution instruction information, the image forming apparatus 10 and the projector 12 execute the function according to the execution instruction information (S09). For example, the image forming apparatus 10 can be linked to the image forming apparatus 10 such as a function of transferring scan data from the image forming apparatus 10 (multifunction machine (B)) to the projector 12 (PC (A)) and projecting data by the projector 12. When processing for transmitting and receiving data to and from the projector 12 is included, communication is established between the image forming apparatus 10 and the projector 12. In this case, for example, the execution instruction information transmitted to the image forming apparatus 10 includes the address information of the projector 12, and the execution instruction information transmitted to the projector 12 includes the address information of the image forming apparatus 10. Communication is established between the image forming apparatus 10 and the projector 12 using the address information.

  When the execution of the cooperation function is completed, information indicating completion of execution of the cooperation function is transmitted from the image forming apparatus 10 and the projector 12 to the terminal device 16 (S10). Information indicating that the execution of the cooperation function is completed is displayed on the UI unit 46 of the terminal device 16 (S11). When information indicating execution completion is not displayed even after a preset time has elapsed since the execution instruction is given, the control unit 48 of the terminal device 16 causes the UI unit 46 to display information indicating an error, The execution instruction information or information indicating the connection request may be transmitted to the image forming apparatus 10 and the projector 12 again.

  Next, the user confirms whether or not to cancel the cooperative state between the image forming apparatus 10 and the projector 12 (S12), and performs processing according to the presence or absence of the cancellation (S13). When canceling the linked state, the user uses the terminal device 16 to give a cancellation instruction. Thereby, the communication between the terminal device 16 and the image forming apparatus 10 and the projector 12 is released. Similarly, communication between the image forming apparatus 10 and the projector 12 is also released. If the cooperation state is not canceled, an execution instruction may be given continuously.

  Moreover, you may increase the apparatus of cooperation object. For example, device identification information of a third device may be acquired, and a cooperation function executed by linking three devices including the image forming apparatus 10 and the projector 12 may be specified. Information indicating that the image forming apparatus 10 and the projector 12 have already been identified as the cooperation function is stored in the server 14 or the terminal device 16.

  The device identification information indicating the device to be linked and the linkage function identification information indicating the executed linkage function may be stored in the terminal device 16, the server 14, or the like. For example, user account information (user identification information) of a user who uses the terminal device 16 is acquired, the user account information, device identification information indicating a cooperation target device, and cooperation function identification information indicating an executed cooperation function May be created and stored in the terminal device 16, the server 14, or the like. The history information may be created by the terminal device 16 or the server 14. By referring to the history information, it is specified which cooperation function is executed using which device group.

  Note that the devices to be linked (for example, the image forming apparatus 10 and the projector 12) may store the user account information of the user who has requested the connection and the terminal identification information indicating the connection request source terminal device 16 as history information. Good. By referring to the history information, the user who uses the device is specified. For example, the history information may be used to specify a user when specifying a user who has used the device when the device is broken, or when charging a consumable item. The history information may be stored in the server 14 or the terminal device 16 or may be stored in another device.

  For example, the user account information is stored in advance in the storage unit 44 of the terminal device 16, and the control unit 48 of the terminal device 16 functions as an example of a user identification unit, and reads the user account information of the user from the storage unit 44. Thus, the user who uses the terminal device 16 is identified. When user account information of a plurality of users is stored in the storage unit 44, the user uses the terminal device 16 to specify his / her user account information. Thereby, the user account information of the user is read and the user is identified. As another example, the control unit 48 of the terminal device 16 may identify the user by reading the user account information of the user who has logged into the terminal device 16. As yet another example, when only one user account information is stored in the same terminal device 16, the control unit 48 of the terminal device 16 may identify the user by reading the user account information. . If no user account is set and user account information is not created, initial setting is performed, thereby creating user account information.

  The use history of the cooperation function is managed for each user, and information indicating the cooperation function used in the past by the user indicated by the read user account information may be displayed on the UI unit 46 of the terminal device 16. Information indicating the usage history may be stored in the terminal device 16 or may be stored in the server 14. Moreover, the information which shows the cooperation function currently used with the frequency more than the preset usage frequency may be displayed. By providing such a shortcut function, it is possible to reduce the time and effort of the user's operation regarding the linkage function.

  When the single use function is executed, information indicating an instruction to execute the single use function is transmitted from the terminal device 16 to a device that executes the single use function. The device executes the single use function according to the execution instruction.

  Each of the image forming apparatus 10, the server 14, and the terminal device 16 is realized by cooperation of hardware and software, for example. Specifically, each of the image forming apparatus 10, the server 14, and the terminal device 16 includes one or a plurality of processors such as a CPU (not shown). The function of each unit of the image forming apparatus 10, the server 14, and the terminal device 16 is realized by the one or more processors reading and executing a program stored in a storage device (not shown). The program is stored in the storage device via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, each unit of the image forming apparatus 10, the server 14, and the terminal device 16 may be realized by hardware resources such as a processor and an electronic circuit. In the realization, a device such as a memory may be used. As another example, each part of the image forming apparatus 10, the server 14, and the terminal device 16 may be realized by a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or the like.

DESCRIPTION OF SYMBOLS 10 apparatus (image forming apparatus), 12 apparatus (projector), 14 server, 16 terminal device, 32 apparatus function information, 34 cooperation function information, 38 specification part.

Claims (9)

  By combining the first function possessed by the first device and the second function possessed by the second device in accordance with the display position of the first image associated with the first device and the second image associated with the second device. An information processing apparatus having a control unit that switches display of executable cooperation functions. The first image is an image obtained by photographing the first device;
The second image is an image obtained by photographing the second device.
The information processing apparatus according to claim 1. Identifying means for identifying the first device based on the first image and identifying the second device based on the second image;
The control unit switches the display of the cooperation function according to the display position when the first device and the second device are identified by the identification unit.
The information processing apparatus according to claim 2.   The display of the cooperation function executable by combining the first function and the second function is controlled according to the display position of the first image associated with the first function and the second image associated with the second function. An information processing apparatus having a control means. The control means has a function associated with a partial image in the first image and a partial image in the second image according to a display position of the partial image in the first image and the partial image in the second image. To switch the display of the linkage function that can be executed by combining the function associated with
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. Different functions are linked to each partial image,
The information processing apparatus according to claim 5. The cooperation function is a function that could not be used before cooperation.
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. Computer
By combining the first function possessed by the first device and the second function possessed by the second device in accordance with the display position of the first image associated with the first device and the second image associated with the second device. Switching means for switching the display of executable linkage functions;
Program to function as. Computer
Control the display of executable functions that can be executed by combining the first function and the second function in accordance with the display position of the first image associated with the first function and the second image associated with the second function. Control means,
Program to function as.