JP6129136B2 - Apparatus operating device and apparatus operating method - Google Patents

Description

  The present invention relates to an apparatus operating device and an apparatus operating method for operating an electronic apparatus having a plurality of functions.

  In order to improve the operability of an electronic device, a technique is known in which various functions of the electronic device are operated using a screen on an operation terminal capable of cooperating with the electronic device. Since the user who uses an electronic device can operate the electronic device by an operation using, for example, a touch panel of a tablet terminal, the operability of the electronic device is improved. Japanese Patent Application Laid-Open No. 2004-151867 discloses a technique for specifying an electronic device by reading an identification number of the electronic device included in video data captured by an operation terminal and generating and displaying a virtual user interface for operating the electronic device. Is published.

JP 2013-172432 A

  However, since the conventional operation terminal acquires and displays the function list of the electronic device after specifying the electronic device, the user has to select the function to be operated from the displayed function list. . Therefore, since the user has to perform an operation of selecting a desired function from the function list, there is a problem that the operability is poor.

  Further, the operation terminal needs to exchange with the electronic device in order to acquire the function list from the electronic device. Accordingly, since a time for communication between the operation terminal and the electronic device occurs, a waiting time has occurred before the function list can be displayed. As a result, there is also a problem that it takes a long time to operate a desired function.

  Therefore, the present invention has been made in view of the above-described points, and an object thereof is to provide a device operating device that can improve the operability of an electronic device.

In order to solve the above-described problem, an apparatus operating device according to the present invention includes an imaging unit that acquires a captured image including a part of an electronic apparatus having a plurality of functions, and the electronic apparatus included in the captured image. Based on an image including an operation unit, a specifying unit that specifies a function of the electronic device related to the captured image, a display unit that displays an operation screen corresponding to the function specified by the specifying unit, and the operation screen And acquiring means for acquiring the operation content input in step (b), and transmission means for transmitting setting instruction information for setting the operation of the function to the electronic device based on the operation content.

  ADVANTAGE OF THE INVENTION According to this invention, there exists an effect that the operativity of an electronic device can be improved.

It is a key map of device operation system S concerning a 1st embodiment. 2 is a diagram illustrating configurations of an image display device 1 and a tablet terminal 2. FIG. 2 is a functional configuration diagram illustrating a configuration of functions executed by a CPU 10 of the image display device 1 and a CPU 20 of the tablet terminal 2. FIG. 5 is a flowchart showing processing of a control unit 40 in the tablet terminal 2. 4 is a diagram illustrating an example of a captured image captured using a capturing unit 41. FIG. It is a figure which shows an example of the setting information which the apparatus state management part 30 manages. It is a figure which shows an example of the operation screen 80 produced | generated by UI production | generation part 46. FIG. 5 is a flowchart showing processing of a part specifying unit 42. It is a figure which shows the example of the corresponding information of a two-dimensional barcode and a site | part. It is a figure which shows an example of the OSD screen in this embodiment. 4 is a diagram illustrating an example of a captured image captured using a capturing unit 41. FIG. It is a figure which shows the example of the corresponding information of a two-dimensional barcode and site | part ID. It is a figure which shows an example of the setting information which the apparatus state management part 30 manages. It is a figure which shows an example of the operation screen which UI generation part 46 produces | generates. 3 is a diagram illustrating an example of an OSD screen displayed on the image display device 1. FIG. 6 is a diagram illustrating an example of an operation screen displayed on the tablet terminal 2. FIG. 3 is a functional configuration diagram illustrating processing executed in the CPU 10 of the image display device 1 and the CPU 20 of the tablet terminal 2. FIG. 3 is a flowchart of processing of a control unit 210. 10 is a flowchart of processing of an enlargement / reduction ratio calculation unit 215. It is a figure which shows an example of an initial image and a picked-up image. 1 is a diagram illustrating an example of an image display device 1. FIG. FIG. 3 is a diagram illustrating an example of a captured image obtained by capturing a two-dimensional barcode of the image display device 1 with a tablet terminal 2.

<First Embodiment>
[Basic configuration of device operation system S]
FIG. 1 is a conceptual diagram of a device operation system S according to the first embodiment. The device operation system S includes an image display device 1 and a tablet terminal 2. The image display device 1 is a liquid crystal display, for example, and is a type of electronic device that provides a plurality of functions such as a power on / off function and an image quality adjustment function. The tablet terminal 2 is a kind of device operating device used for operating the function of the electronic device. The image display device 1 and the tablet terminal 2 can communicate bidirectionally via a network such as a wireless LAN (Local Area Network). In the present embodiment, the image display device 1 and the tablet terminal 2 are described as being able to communicate with each other wirelessly. However, the present invention is not limited to this, and the image display device 1 and the tablet terminal 2 may communicate with each other by wire.

  The tablet terminal 2 is a terminal capable of a touch operation using a touch panel, such as a smartphone or a tablet. A user 3 shown in FIG. 1 can execute the function of the image display device 1 by operating the tablet terminal 2.

  The tablet terminal 2 has a camera that can capture a captured image. The user 3 can take a photographed image including a part of the image display device 1 having a plurality of functions using the camera. The tablet terminal 2 identifies the function of the image display device 1 related to the photographed image obtained by photographing with the camera, generates an operation screen corresponding to the identified function, and displays the operation screen on the display panel. The tablet terminal 2 acquires the operation content of the user 3 through the operation screen displayed on the display. The tablet terminal 2 transmits setting instruction information for setting the operation of the specified function to the image display device 1 based on the acquired operation content. The image display device 1 operates a function based on the received setting instruction information, and updates the held setting value.

  For example, when the image of the power button is included in the captured image, the tablet terminal 2 displays an operation screen for turning on or off the display panel of the image display device 1. When receiving an operation for turning on the power of the image display device 1 from the user 3, the tablet terminal 2 transmits setting instruction information for instructing to turn on the power of the image display device 1 to the image display device 1. When the image display apparatus 1 detects that the setting instruction information received from the tablet terminal 2 includes an instruction to turn on the power, the image display apparatus 1 turns on the power of the display panel.

  The tablet terminal 2 has a sensor for detecting its own attribute. The tablet terminal 2 has, for example, a gyro sensor, detects its own direction by the sensor, and specifies an attribute indicating whether it is vertical or horizontal. The tablet terminal 2 displays an operation screen corresponding to the function specified by the captured image in a mode based on the specified attribute of the tablet terminal 2. For example, the tablet terminal 2 displays a long operation screen in the vertical direction when the user 3 has the tablet terminal 2 in the vertical direction, and in the horizontal direction when the user 3 has the tablet terminal 2 in the horizontal direction. Display a long operation screen.

[Configuration of Image Display Device 1]
FIG. 2 is a diagram illustrating configurations of the image display device 1 and the tablet terminal 2. The image display device 1 includes a CPU 10, a storage unit 11, a display unit 12, an operation unit 13, and a communication unit 14. Each of these blocks can communicate with each other via a bus 15.

  The CPU 10 is a central processing unit, and causes the image display device 1 to function by developing and executing a processing program stored in the storage unit 11. The CPU 10 controls each block of the image display device 1.

  The storage unit 11 includes a ROM that stores a processing program of the image display device 1 executed by the CPU 10 and a RAM that stores data processed by the CPU 10. The CPU 10 performs various processes by developing a program stored in the ROM in advance in the RAM and executing the program. The storage unit 11 stores setting information related to the setting contents of each function in association with a plurality of functions of the image display device 1.

The display unit 12 is a liquid crystal display, for example, and displays video data or image data input to the image display device 1 or an operation screen.
The operation unit 13 is a user interface used by the user 3 to operate the image display device 1. The operation unit 13 is a jog switch, for example, and generates a signal corresponding to the operation content of the user 3.

  The communication unit 14 is a communication interface for transmitting / receiving data to / from the tablet terminal 2. The communication unit 14 is, for example, a wireless LAN interface, and transmits / receives data to / from the tablet terminal 2 via the network N.

[Configuration of tablet terminal 2]
The tablet terminal 2 includes a CPU 20, a storage unit 21, a display unit 22, an operation unit 23, a communication unit 24, a sensor 25, and a camera 26. Each of these blocks can communicate with each other via a bus 27.

  The CPU 20 is a central processing unit, and causes the tablet terminal 2 to function by expanding and executing the processing program stored in the storage unit 21. The CPU 20 controls each block of the tablet terminal 2.

  The storage unit 21 includes a ROM that stores a processing program for the tablet terminal 2 executed by the CPU 20 and a RAM that stores data processed by the CPU 20. The CPU 20 performs various processes by developing a program stored in the ROM in advance in the RAM and executing the program.

The display unit 22 is a liquid crystal display, for example, and displays an operation screen for the user 3 to operate the image display device 1.
The operation unit 23 is, for example, a touch panel provided so as to overlap the display unit 22. The operation unit 23 detects a position where the user 3 touches the operation screen, and notifies the CPU 20 of information indicating the detected position.

The communication unit 24 is a communication interface for transmitting and receiving data to and from the image display device 1. The communication unit 24 is, for example, a wireless LAN interface, and transmits / receives data to / from the image display device 1 via the network N.
The sensor 25 detects the movement or orientation of the tablet terminal 2. The sensor 25 is, for example, a gyro sensor or an acceleration sensor.
The camera 26 includes an image sensor and a lens. The camera 26 generates a captured image including an image of the subject based on the control of the CPU 20.

[Functional Configuration of Image Display Device 1 and Tablet Terminal 2]
FIG. 3 is a functional configuration diagram illustrating a configuration of functions executed by the CPU 10 of the image display device 1 and the CPU 20 of the tablet terminal 2.

First, the functional configuration of the image display apparatus 1 executed by the CPU 10 will be described.
The CPU 10 functions as a device status management unit 30, an operation exclusion processing unit 31, a device setting processing unit 32, and a communication processing unit 33.
The device state management unit 30 manages setting information regarding each function of the image display device 1. The setting information includes options such as values or characteristics that can be set for each function, current setting values, and supplementary information. Such setting information is referred to by the tablet terminal 2 via the communication processing unit 33. Details of the setting information will be described later.

  The operation exclusion processing unit 31 performs an exclusion process on the operation of the image display device 1 in response to an exclusion instruction from the tablet terminal 2. Specifically, the operation exclusion processing unit 31 does not display the displayed OSD when an exclusion instruction is acquired from the tablet terminal 2 while displaying a UI (User Interface) such as an OSD (On Screen Display). The operation of the device on the image display device 1 is locked until the exclusive instruction from the tablet terminal 2 is released.

  The operation exclusion processing unit 31 is based on information other than the setting instruction information transmitted by the tablet terminal 2 for a predetermined time after the device setting processing unit 32 transmits the setting information such as the current setting value to the tablet terminal 2. The operation of the device on the image display device 1 may be locked by controlling not to update the setting information. By doing in this way, while the user 3 is operating the image display apparatus 1 using the tablet terminal 2, the other user 3 uses the operation screen of the image display apparatus 1, and functions the image display apparatus 1. Setting can be prevented. Therefore, it is possible to prevent the operation content on the tablet terminal 2 and the operation content on the image display device 1 from competing.

  The device setting processing unit 32 manages setting information regarding various functions of the image display device 1, and performs setting information update processing according to the operation of the user 3 on the tablet terminal 2. Specifically, in response to a request from the tablet terminal 2, the device setting processing unit 32 transmits the latest setting information to the tablet terminal 2 and acquires an instruction to reflect the new setting information from the tablet terminal 2. The managed setting information is updated based on the acquired instruction. For example, the device setting processing unit 32 updates a setting value corresponding to the setting information received from the tablet terminal 2 among the setting values stored in the storage unit 11 to a value included in the received setting information. Then, the image display device 1 is set. The device setting processing unit 32 updates the setting information based on information other than the setting instruction information transmitted by the tablet terminal 2 for a predetermined time after transmitting the setting information such as the current setting value to the tablet terminal 2. do not do.

  The communication processing unit 33 generates data to be transmitted to the tablet terminal 2 via the communication unit 14. Further, the communication processing unit 33 decodes the data received from the tablet terminal 2 via the communication unit 14, and sends the content of the decoded data to the device state management unit 30, the operation exclusion processing unit 31, and the device setting processing unit 32. Notice. The communication processing unit 33 receives a request for the latest setting information from the tablet terminal 2 and transmits the setting information stored in the storage unit 11 to the tablet terminal 2 in response to the received request. In addition, the communication processing unit 33 receives a setting value related to the function of the image display device 1 from the tablet terminal 2.

Subsequently, the functional configuration of the tablet terminal 2 executed by the CPU 20 will be described.
The CPU 20 includes a control unit 40, an imaging unit 41, a part specification unit 42, a function specification unit 43, an operation method determination unit 44, an attribute information management unit 45, a UI generation unit 46, a display control unit 47, and an operation input I / F unit 48. , Function as a setting processing unit 49, an exclusion instruction unit 50, and a communication processing unit 51.

  The control unit 40 is a block that performs overall control of the tablet terminal 2. An application for controlling the image display device 1 by mutual communication with the image display device 1 is installed in the tablet terminal 2, and the control unit 40 starts processing in a state where the application is activated. . Details of processing performed by the control unit 40 will be described later.

  The imaging unit 41 receives an imaging instruction from the control unit 40 according to the operation of the user 3 and captures a captured image including a part of the image display device 1 using the camera 26 included in the tablet terminal 2. . The imaging unit 41 provides the captured image to the control unit 40.

  The part specifying unit 42 specifies a part of the image display device 1 included in the photographed image using a photographed image including a part of the image display device 1 (for example, an operation unit). Specifically, the part specifying unit 42 acquires an instruction for specifying the part of the image display device 1 included in the photographed image and the photographed image from the control unit 40 and analyzes the photographed image to include the photographed image. Identify the site. For example, the part specifying unit 42 matches the image of each part of the image display device 1 stored in the storage unit 21 with the captured image acquired from the control unit 40, and captures the image among the images stored in the storage unit 21. An image having the largest correlation with the image is identified.

  The part specifying unit 42 specifies a part ID corresponding to the specified image by referring to a table stored in the storage unit 21 that associates the image of each part with the identification information (hereinafter referred to as part ID) of each part. To do. The part specifying unit 42 notifies the control part 40 of the specified part ID. The part specifying unit 42 is information that can be used for specifying a part such as a QR code (registered trademark) or a bar code provided at a position associated with the operation unit of the image display device 1. In the case where the image including the image is included, the part may be specified based on the information included in the image. Here, the position associated with the operation unit is a position where the distance between the position and the operation unit is shorter than the distance between the position and another operation unit, for example, adjacent to the operation unit. It is a position to do.

  The function specifying unit 43 specifies a function related to the captured image. Specifically, the function specifying unit 43 captures a captured image based on a region ID specified using a captured image including a part of the image display device 1 (for example, an operation unit) included in the captured image. Identify functions related to. More specifically, the function specifying unit 43 acquires an instruction for specifying a function and a part ID from the control unit 40, and specifies a function ID indicating the type of function corresponding to the acquired part ID. For example, the function specifying unit 43 specifies a function ID corresponding to the part ID specified by the part specifying unit 42 based on a correspondence table between the part ID and the function ID stored in the storage unit 21. In the above table, for example, the function ID corresponding to the part ID 01 is associated with the power on / off function (function ID: FUNC001). The function specifying unit 43 notifies the control unit 40 of the specified function ID.

  When the operation method determination unit 44 obtains an instruction to determine the operation method from the control unit 40, the operation method determination unit 44 specifies the operation method by referring to the attribute information of the tablet terminal 2 managed by the attribute information management unit 45. Specifically, the operation method determination unit 44 selects an operation method suitable for the attribute of the tablet terminal 2 from among a plurality of operation methods stored in the storage unit 21 in association with the function ID. When the tablet terminal 2 includes the gyro sensor, the operation method determination unit 44 sets the direction in which the user 3 holds the tablet terminal 2 based on the attribute information indicating the direction of the tablet terminal 2 indicated by the gyro sensor. Select a suitable operating method. For example, when the tablet terminal 2 is in the portrait orientation, the operation method determination unit 44 determines the operation method using the portrait operation screen, and when the tablet terminal 2 is in the landscape orientation, the operation using the landscape operation screen is performed. Decide on a method. The operation method determination unit 44 notifies the control unit 40 of the determined operation method.

  When the UI generation unit 46 acquires an instruction to generate an operation screen and the operation method determined by the operation method determination unit 44 from the control unit 40, the UI generation unit 46 performs setting related to the target function based on the operation method acquired from the control unit 40. The content of the operation screen is determined and an operation screen is generated. For example, for an operation method using an operation screen for portrait orientation, the UI generation unit 46 generates an operation screen that is long in the vertical direction. When generating the operation screen, the UI generation unit 46 notifies the exclusion instruction unit 50 of an exclusion request for prohibiting setting of the function of the image display device 1 using the operation screen displayed on the image display device 1.

  The display control unit 47 causes the display unit 22 to display an operation screen corresponding to the function specified by the function specifying unit 43. Specifically, the display control unit 47 generates display data by combining the operation screen generated by the UI generation unit 46 and the image data displayed on the tablet terminal 2, and displays an image based on the display data on the display unit 22. Display.

  The operation input I / F unit 48 acquires the operation content input on the operation screen via the operation unit 23 and notifies the control unit 40 of the acquired operation content. For example, the operation input I / F unit 48 updates the setting information of the image display device 1 input by the user 3 on the operation screen as the operation information indicating the operation content of the user 3 acquired through the operation unit 23. Information is acquired and the acquired update information is notified to the control unit 40. Specifically, when the operation unit 23 is a touch panel, the operation input I / F unit 48 acquires the coordinates of the position touched by the user 3 from the operation unit 23, and the position corresponding to the acquired coordinates on the operation screen. Identify the content displayed in. The operation input I / F unit 48 notifies the control unit 40 of update information indicated by the specified content.

  When the setting processing unit 49 acquires an instruction to reflect the update information and the update information from the control unit 40, the setting processing unit 49 generates a communication command necessary for communication with the image display device 1 based on the acquired update information. Specifically, the setting processing unit 49 indicates a setting instruction indicating an instruction for updating the setting information regarding the operation of the function specified by the function specifying unit 43 based on the operation content acquired by the operation input I / F unit 48. A communication command including information is generated. The setting processing unit 49 provides a communication command to the communication processing unit 51.

  The exclusion instruction unit 50 issues an exclusion instruction to the image display device 1 via the communication processing unit 51. Specifically, after the control unit 40 receives the current setting information from the image display device 1, the exclusion instruction unit 50 responds to an instruction from the UI generation unit 46 that has generated an operation screen including contents based on the setting information. Based on this, an exclusive instruction for prohibiting the setting of the operation of the image display device 1 by another device is transmitted to the image display device 1.

  The communication processing unit 51 generates data to be transmitted to the image display device 1 via the communication unit 24. For example, the communication processing unit 51 transmits a communication command including the setting instruction information acquired from the setting processing unit 49 to the image display apparatus 1. In addition, the communication processing unit 51 receives data from the image display device 1 via the communication unit 24. For example, the communication processing unit 51 receives setting information related to the function specified by the function specifying unit 43 from the image display device 1. In addition, the communication processing unit 51 receives a setting completion notification indicating that the setting is completed from the image display device 1 and notifies the control unit 40 of the setting completion notification.

[Operation Flowchart of Tablet Terminal 2]
FIG. 4 is a flowchart showing processing of the control unit 40 in the tablet terminal 2.
First, the control unit 40 captures a captured image using the capturing unit 41 in accordance with the operation of the user 3 (S101).

  FIG. 5 is an example of a captured image. An image displayed on the panel 60 of the tablet terminal 2 is a captured image. The captured image includes an image of the Input button 61 provided in the image display device 1. Next to the Input button 61, a two-dimensional barcode 62 including information indicating that the photographed button is the Input button 61 is printed. The photographed image also includes an image of the two-dimensional barcode 62. .

  Subsequently, the control unit 40 instructs the site specifying unit 42 to specify the site, and provides the captured image acquired from the imaging unit 41 (S102). When receiving the notification of the part ID from the part specifying unit 42 (in the case of yes in S103), the control unit 40 specifies the part based on the part ID (S104). In the present embodiment, it is assumed that the part ID = 01 and the part corresponding to the part ID is an input switching button. Details of the process in which the part specifying unit 42 specifies the part will be described later with reference to FIG.

  Subsequently, the control unit 40 instructs the function specifying unit 43 to specify the function (S105). When the control unit 40 receives a notification including information indicating the function specified by the function specifying unit 43 from the function specifying unit 43 (Yes in S106), the control unit 40 specifies the function based on the received notification. In the present embodiment, since the part ID = 01 is specified in S104, the control unit 40 specifies FUNC001 corresponding to the input switching function as the function ID corresponding to the part ID = 01 (S107).

  Subsequently, the control unit 40 instructs the operation method determination unit 44 to determine the operation method (S108). For example, the operation method determination unit 44 determines that the orientation of the tablet terminal 2 is vertical, and notifies the control unit 40 of an operation method corresponding to the vertical orientation. When receiving information indicating the operation method determined by the operation method determination unit 44 from the operation method determination unit 44 (Yes in S109), the control unit 40 specifies the operation method based on the received information (S110). .

  Subsequently, the control unit 40 acquires setting information of a function to be operated from the device state management unit 30 of the image display device 1 via the communication processing unit 51 (S111). Specifically, the control unit 40 acquires the setting information of the target function corresponding to the transmitted function ID by transmitting the function ID to the device state management unit 30. Here, after receiving the setting information from the device state management unit 30, the control unit 40 instructs the exclusion instruction unit 50 to display an exclusive instruction for prohibiting setting of the operation of the image display device 1 by another device. Transmit to the device 1.

  FIG. 6 is a diagram illustrating an example of setting information managed by the device state management unit 30. The setting information includes a function ID 70, a target function name 71, an option 72, a current setting value 73, and supplementary information 74. In FIG. 6, setting information 75 relating to the input switching function whose function ID is FUNC001 is shown. The option 72 includes information indicating that the input switching function has an option of Input1 / Input2 / Input3. The current setting value 73 includes information indicating that the current setting value is Input1. The supplementary information 74 includes information indicating that Input1 = PC, Input2 = camera, and Input3 = camera2.

  Returning to FIG. 4, the operation after S111 will be described. The control unit 40 passes the setting information acquired in S111 and the operation method specified in S110 to the UI generation unit 46, and instructs generation of an operation screen (S112). The UI generation unit 46 generates an operation screen including contents based on the setting information acquired by the control unit 40 from the image display device 1.

  FIG. 7 is a diagram illustrating an example of an operation screen generated by the UI generation unit 46. Fig.7 (a) shows the example of the operation screen in case the tablet terminal 2 is vertical. An image 80 in FIG. 7A is an operation screen for function setting generated by the UI generation unit 46, and is an operation screen in which choices of Input1 / Input2 / Input3 are expressed by rectangular button images. The user 3 can switch to a desired input by selecting these button images by a touch operation. Note that the Input 1 button 81 shown in the image 80 is the current setting value indicated by the setting information acquired by the control unit 40 from the image display device 1, and is displayed in a pattern and color different from other options.

  FIG. 7B shows an example of the operation screen when the tablet terminal 2 is in the landscape orientation. When the operation method is in the landscape orientation, the area that can be displayed on the tablet terminal 2 is larger than in the portrait orientation. Therefore, the UI generation unit 46 displays an image 82 corresponding to the supplementary information 74 together with an image corresponding to the option 72 in FIG. In the example shown in FIG. 7B, an image 82 including texts PC1, camera 1, and camera 2 corresponding to the supplementary information 74 in association with the image corresponding to the option 72 is the image 80 corresponding to the option 72. It is displayed next to it.

  In the present embodiment, the operation screen is displayed starting from the two-dimensional bar code 62 indicating the Input button. The display position of the operation screen may be displayed starting from another arbitrary position of the tablet terminal 2 (for example, the upper left position of the tablet terminal 2).

  Returning to FIG. 4, the operation after S112 will be described. When the user 3 touches the Input 2 button in the image 80 on the operation screen shown in FIG. 7 (Yes in S113), the control unit 40 receives Input 2 as setting information from the operation input I / F unit 48. .

  Subsequently, the control unit 40 notifies the setting processing unit 49 of a setting information reflection instruction for updating the setting information acquired in S111 to the new setting information (that is, update information) acquired in S113 (S114). In the present embodiment, the control unit 40 notifies Input2 as new setting information. The setting processing unit 49 transmits an input switching request for causing the image display device 1 to perform input switching to Input 2 to the image display device 1 via the communication processing unit 51. When receiving the input switching request, the image display device 1 executes input switching from Input1 to Input2.

FIG. 8 is a flowchart showing the processing of the part specifying unit 42.
The part specifying unit 42 acquires a part specifying instruction from the control unit 40 (S201). At this time, the part specifying unit 42 also acquires a captured image from the control unit 40 at the same time.

  Subsequently, the part specifying unit 42 analyzes the captured image acquired from the control unit 40 and specifies a target part included in the captured image (S202). Here, the part specifying method in the part specifying unit 42 will be described with reference to FIGS. 5 and 9. The part specifying unit 42 acquires the captured image shown in FIG. 5 and cuts out the image portion of the two-dimensional barcode 62 included in the captured image. The two-dimensional barcode 62 is printed in the vicinity of the operation means in the image display device 1 and includes information for identifying the part of the operation means. FIG. 9 is a diagram illustrating an example of correspondence information between a two-dimensional barcode and a part stored in advance in the tablet terminal 2.

  The correspondence information is information in which the two-dimensional barcode 65, the part ID 66, and the part name 67 are associated with each other. The two-dimensional barcode corresponding to the two-dimensional barcode 65 in this embodiment is the two-dimensional barcode 62 shown in FIG. 5. In the correspondence information 68, “part ID 66 = 01” and “part name 67 = input switching button” ". As illustrated in FIG. 8, the part specifying unit 42 notifies the control unit 40 of the part ID specified in S202 (S203).

  In this embodiment, although the site | part specific part 42 demonstrated the example which pinpoints a site | part by the image analysis using the two-dimensional barcode 62, it does not restrict to this. The part specifying unit 42 may specify a part or a function based on a lighting pattern of a plurality of lamps provided for each button of the image display device 1.

  In addition, the part specifying unit 42 performs image analysis on the shape and characters of each button of the image display device 1 shown in the captured image, for example, and the shape of each button and character pattern image held in advance in the tablet terminal 2. The site may be specified by comparing with. Moreover, the site | part specific part 42 may specify a site | part based on the position in the image display apparatus 1 of each recognized site | part, when a some site | part is recognized in the picked-up image. For example, the part specifying unit 42 can specify that the part included in the captured image is the input button by recognizing that the part is located at the lower right of the image display device 1.

[Effect in the first embodiment]
As described above, according to the present embodiment, the tablet terminal 2 captures and captures an operation unit and a two-dimensional barcode provided in the vicinity of the operation unit as a part of the image display device 1. The site included in the image is identified. And the tablet terminal 2 can display the operation screen for controlling the image display apparatus 1 by specifying the function corresponding to the specified site | part. Since the user 3 can operate the image display device 1 by operating the displayed operation screen, the operability of the image display device 1 by the user 3 is improved. Furthermore, since the tablet terminal 2 switches the content of the operation screen to be displayed based on the attribute information of the tablet terminal 2, an optimal UI can be displayed according to the specification and state of the tablet terminal 2. As a result, the convenience of the user 3 can be improved.

<Second Embodiment>
A second embodiment of the present invention will be described. In the description of this embodiment, functions different from those of the first embodiment will be described in detail, and descriptions of functions similar to those of the first embodiment will be omitted. In the first embodiment, the example in which the two-dimensional barcode is printed near the button of the image display device 1 has been described. However, the image display device 1 according to the present embodiment identifies the function in the OSD screen. The second embodiment is different from the first embodiment in that a two-dimensional barcode as an identification image including information to be displayed is displayed. The tablet terminal 2 according to the present embodiment specifies the function based on the two-dimensional barcode included in the captured image.

  FIG. 10 is a diagram showing an example of the OSD screen in the present embodiment. In the present embodiment, it is assumed that the image display apparatus 1 automatically automatically displays an OSD screen 90 that prompts the user 3 to execute calibration. The OSD screen 90 includes two buttons, a calibration execution button 91 and a cancel button 92, in addition to the words “please execute calibration”. On the OSD screen 90, a two-dimensional barcode 93 indicating that the content displayed on the OSD screen 90 is the calibration function is displayed in the vicinity of the calibration execution button 91.

  FIG. 11 is a diagram illustrating an example of a photographed image obtained by photographing a part of the screen illustrated in FIG. 10 using the photographing unit 41. In the present embodiment, a two-dimensional barcode 93 is included in the captured image as shown in FIG.

  FIG. 12 is a diagram illustrating an example of correspondence information between a two-dimensional barcode and a part ID used when the part specifying unit 42 specifies a part by performing image analysis on a captured image. The two-dimensional barcode 93 included in the photographed image shown in FIG. 11 is included in the correspondence information 100 shown in FIG. 12, and the two-dimensional barcode 93 includes “part ID = 02” and “part name = It corresponds to “Calibration button”. The function specifying unit 43 specifies that the function ID corresponding to “part ID = 02” is FUNC002 based on the correspondence table between the part ID and the function ID stored in the storage unit 21.

  The control unit 40 acquires setting information corresponding to the function ID specified by the function specifying unit 43 from the image display device 1. Specifically, the control unit 40 requests setting information from the image display device 1, and the image display device 1 transmits the setting information managed by the device state management unit 30 to the control unit 40. The UI generation unit 46 generates an operation screen including the setting information acquired by the control unit 40.

FIG. 13 is a diagram illustrating an example of setting information managed by the device state management unit 30. The control unit 40 specifies the setting information 110 related to the calibration function corresponding to the function ID = FUNC002 among the plurality of setting information. For the calibration function, the choices are target value setting / target value copy, and the current setting values are luminance = 120 cd / m ² , color temperature = 5000K, and color gamut = AdobeRGB.

  FIG. 14 is a diagram illustrating an example of an operation screen generated by the UI generation unit 46 according to the present embodiment. In the present embodiment, a description will be given using a screen example when the operation method is in the horizontal direction, the panel size is large, and the display area is wide. FIG. 14A shows the operation screen 120 generated by the UI generation unit 46. The operation screen 120 is an operation screen in which two options “target value setting” and “target value copy” are expressed by buttons.

  The user 3 can select a desired item by selecting these buttons by a touch operation. FIG. 14A shows a state where the target value setting button is selected, and a setting value corresponding to the current setting information is displayed on the right side of the target value setting button.

  The button 121 is a help button for accepting an instruction to display an explanation about the function, and help information is associated with details of each set value corresponding to each help button. When the user 3 touches the button 121, a screen 130 as shown in FIG. 14B is displayed, and a detailed explanation about the brightness setting is displayed. When the user 3 touches the button for closing the screen of the button 131, the screen display is returned to the state shown in FIG.

  The UI generation unit 46 generates an operation screen including a help button when the size of the area where the display unit 22 displays the operation screen satisfies a predetermined condition, and causes the display unit 22 to display the operation screen. For example, the UI generation unit 46 includes a help button when the size of the display area of the display unit 22 of the tablet terminal 2 is sufficiently larger than the size of the operation screen corresponding to the function specified by the function specifying unit 43. Generate an operation screen. The UI generation unit 46 may generate an operation screen including a help button when the size of the display area of the display unit 22 is larger than the size necessary for displaying the detailed description regarding the function.

  The UI generation unit 46 may generate an operation screen different from the aspect shown in FIG. For example, the UI generation unit 46 may generate an operation screen having a mode different from the OSD screen displayed on the image display device 1.

  FIG. 15 is a diagram illustrating an example of the OSD screen displayed on the image display device 1. FIG. 16 is a diagram illustrating an example of an operation screen displayed on the tablet terminal 2 when the OSD screen illustrated in FIG. 15 is displayed. In the image display apparatus 1 shown in FIG. 15, a two-dimensional barcode 141 indicating that the function corresponding to the OSD screen 140 is image quality setting is displayed together with the OSD screen 140 of the image quality setting function. As indicated by the OSD screen 140, the user 3 can operate each setting value using a slider. The tablet terminal 2 identifies that the OSD screen 140 displayed on the image display device 1 is a screen corresponding to the image quality setting function by photographing the two-dimensional barcode 141, and the image display device 1 Get setting function setting information.

  As illustrated in FIG. 16, the UI generation unit 46 generates an operation screen suitable for a touch operation, unlike the OSD screen 140 of the image display device 1 illustrated in FIG. 15, based on setting information acquired by the tablet terminal 2. . Specifically, on the operation screen displayed on the tablet terminal 2, the user 3 can perform a setting operation by moving the knob 151 by a touch operation along the circumference of the image 150. Note that the UI generation unit 46 may further display help information 152 when there is room in the display area of the tablet terminal 2.

  In addition, although this embodiment demonstrated the example which the tablet terminal 2 image | photographs one 2D barcode, the tablet terminal 2 may image | photograph a some 2D barcode simultaneously. In this case, the function specifying unit 43 may specify the function by causing the user 3 to select a desired function from the two-dimensional barcode recognized by the tablet terminal 2 by the operation of the user 3.

[Effects of Second Embodiment]
As described above, according to the present embodiment, the image display device 1 displays a two-dimensional barcode for each layer of the OSD, and the tablet terminal 2 specifies a function based on the displayed two-dimensional barcode. To do. Therefore, the tablet terminal 2 can specify a more detailed function than the first embodiment, and can provide an operation screen for the specified function. Moreover, since the tablet terminal 2 can display more detailed setting information and help information when the panel size is large and the display area is large, the convenience of the user 3 is further improved.

<Third Embodiment>
A third embodiment of the present invention will be described. In this embodiment, functions different from those in the first embodiment will be described in detail, and descriptions of functions similar to those in the first embodiment will be omitted.
In the first embodiment, the example in which the display content of the operation screen is switched depending on the orientation of the tablet terminal 2 has been described. In the present embodiment, an example will be described in which the user 3 enlarges or reduces a display image displayed on the image display device 1 by performing an enlargement / reduction operation while photographing a two-dimensional barcode using the tablet terminal 2.

  In the tablet terminal 2 according to the third embodiment, the display unit 22 displays an operation screen including a two-dimensional barcode as a captured image. When the tablet terminal 2 acquires the operation content for changing the size of the two-dimensional barcode included in the operation screen, the image display device 1 displays the operation content on the screen based on the size of the two-dimensional barcode indicated by the operation content. Setting instruction information for setting the size of the image is transmitted to the image display device 1.

  FIG. 17 is a functional configuration diagram illustrating processing executed in the CPU 10 of the image display device 1 and the CPU 20 of the tablet terminal 2 according to the present embodiment. The CPU 10 includes an operation exclusion processing unit 200, a device setting processing unit 201, and a communication processing unit 202. The operation exclusion processing unit 200, the device setting processing unit 201, and the communication processing unit 202 are the same as the operation exclusion processing unit 31, the device setting processing unit 32, and the communication processing unit 33 illustrated in FIG.

  The CPU 20 includes a control unit 210, a photographing unit 211, a function specifying unit 212, an operation method determination unit 213, an attribute information management unit 214, an enlargement / reduction ratio calculation unit 215, a UI generation unit 216, a display control unit 217, and an operation input I / F unit. 218, a setting processing unit 219, an exclusion instruction unit 220, and a communication processing unit 221.

  In addition to the function of the control unit 40 described in the first embodiment, the control unit 210 refers to the enlargement rate or reduction rate (hereinafter referred to as the enlargement / reduction rate) calculated by the enlargement / reduction rate calculation unit 215, and the setting processing unit 219. In response to this, the image display device 1 is instructed to enlarge / reduce the video data being displayed. In the present embodiment, when the user 3 selects the enlargement / reduction function on the OSD screen while displaying an image on the image display apparatus 1, a two-dimensional barcode indicating the enlargement / reduction function is displayed on the image display apparatus 1. Shall be. The control unit 210 captures the two-dimensional barcode using the imaging unit 211, and specifies that the target function is an enlargement / reduction function using the function specifying unit 212. Details of the processing of the control unit 210 will be described later.

  The image capturing unit 211, the function specifying unit 212, the operation method determining unit 213, and the attribute information managing unit 214 are the same as the image capturing unit 41, the function specifying unit 43, the operation method determining unit 44, and the attribute information managing unit 45 shown in FIG. Therefore, explanation is omitted. However, the function specifying unit 212 in this embodiment specifies a function by analyzing a two-dimensional barcode in a captured image acquired together with a function specifying instruction from the control unit 210.

  The enlargement / reduction ratio calculation unit 215 calculates the enlargement / reduction ratio based on the amount of change in the size of the two-dimensional barcode included in the captured image acquired from the imaging unit 211. Specifically, the enlargement / reduction ratio calculation unit 215 calculates the first size of the two-dimensional barcode at the time when the captured image is acquired from the imaging unit 211. Subsequently, when an enlargement / reduction operation by the user 3 is detected via the operation input I / F unit 48, the enlargement / reduction ratio calculation unit 215 performs second operation of the second two-dimensional barcode photographed by the photographing unit 211 after the user 3 operates. Calculate the size. Subsequently, the enlargement / reduction ratio calculation unit 215 calculates the ratio of the second size to the first size as the enlargement / reduction ratio.

  The UI generation unit 216, the display control unit 217, the operation input I / F unit 218, the setting processing unit 219, the exclusion instruction unit 220, and the communication processing unit 221 include the UI generation unit 46, the display control unit 47, and the operation input illustrated in FIG. Since it is the same as the I / F unit 48, the setting processing unit 49, the exclusion instruction unit 50, and the communication processing unit 51, description thereof will be omitted.

  FIG. 18 is a flowchart of processing of the control unit 210 in the present embodiment. S201 is the same as S101 in FIG. 4, and S202 to S207 are the same as S105 to S110 in FIG. However, in this embodiment, it is assumed that the target function specified in S204 is an enlargement / reduction function, and the operation method specified in S207 is “shooting with a camera”.

  In S208, when the control unit 210 receives input information obtained by performing an enlargement / reduction operation by the user 3 via the operation input I / F unit 218 (Yes in S208), the control unit 210 displays an operation screen on the UI generation unit 216. Instruct to generate (S209). Subsequently, the control unit 210 acquires the enlargement / reduction ratio from the enlargement / reduction ratio calculation unit 215 (S210). Details of the processing of the scaling factor calculation unit 215 will be described later.

  Subsequently, the control unit 210 instructs the setting processing unit 219 to reflect the setting information so as to be enlarged / reduced at the enlargement / reduction rate acquired in S210 (S211). When the enlargement / reduction ratio acquired in S210 is, for example, 9 times, the control unit 210 requests the image display device 1 to display the video data at the enlargement ratio of 9 times. In response to a request from the control unit 210, the image display device 1 enlarges the displayed video data by nine times. Subsequently, the control unit 210 confirms whether or not to end the application (S212). If it is determined that the application has not ended yet (in the case of no in S212), S208 to S211 are repeated.

FIG. 19 is a flowchart of processing of the enlargement / reduction ratio calculation unit 215.
The enlargement / reduction ratio calculation unit 215 waits until a captured image is acquired from the imaging unit 211 (S301). The enlargement / reduction ratio calculation unit 215 acquires a captured image from the imaging unit 211 (in the case of yes in S301), and if the acquisition is the first time after activation of the application (in the case of yes in S302), the captured image is used as an initial image. Store (S303).

  FIG. 20 is a diagram illustrating an example of an initial image and a captured image. FIG. 20A is a diagram illustrating an example of an initial image. The image display device 1 displays a two-dimensional barcode 161 together with the video data 160. The video data 160 displays the scenery of the sun and mountains. It is assumed that the two-dimensional barcode 161 is a two-dimensional barcode indicating an enlargement / reduction function.

  The enlargement / reduction ratio calculation unit 215 calculates the area of the two-dimensional barcode reflected in the initial image stored in S303 (S304). Here, the two-dimensional barcode shown in the initial image is the two-dimensional barcode 161a shown in FIG. The enlargement / reduction ratio calculation unit 215 analyzes the number of pixels of the width w and the height h occupied by the two-dimensional barcode 161a in the photographed image, and obtains the area by width w × height h. In the present embodiment, for example, if the width w of the two-dimensional barcode 161a included in the initial image is 10 pixels and the height h is 10 pixels, the area of the two-dimensional barcode 161a is 100 pixels.

  The enlargement / reduction ratio calculation unit 215 does not execute the processes of S302 to S303 when the captured image acquired in S301 is not the first acquisition after the application is started (in the case of no in S302). That is, the enlargement / reduction ratio calculation unit 215 executes the processing of S302 to S303 only for the first time after the application is started.

  Subsequently, the enlargement / reduction ratio calculation unit 215 calculates the area of the captured image acquired from the imaging unit 211 after the user 3 performs the enlargement operation on the tablet terminal 2 (S305). FIG. 20B shows an image taken after the user 3 performs an enlargement operation on the tablet terminal 2, and is a two-dimensional image larger than the two-dimensional barcode 161a in the initial image shown in FIG. Bar code 161b is shown. Here, the enlargement operation means that the user 3 touches the screen in the direction indicated by the arrow 170 in FIG. In this case, the two-dimensional barcode 161 is photographed on the tablet terminal 2 in the size of the two-dimensional barcode 161b. Here, if the width w 'of the two-dimensional barcode 161b is 30 pixels and the height h is 30 pixels, the area of the two-dimensional barcode 161b is 900 pixels.

  Subsequently, the enlargement / reduction ratio calculation unit 215 calculates the enlargement / reduction ratio from the area of the two-dimensional barcode 161a of the initial image calculated in S304 and the area of the two-dimensional barcode 161b of the captured image calculated in S305 (S306). Here, the enlargement / reduction ratio is calculated by (area of the two-dimensional barcode 161b of the captured image) / (area of the two-dimensional barcode 161a of the initial image). In this embodiment, since the area of the two-dimensional barcode 161a of the initial image calculated in S304 is 100 pixels and the area of the two-dimensional barcode 161b of the captured image calculated in S305 is 900 pixels, the enlargement / reduction ratio is nine times.

  The control unit 210 requests the image display device 1 to enlarge or reduce based on the enlargement / reduction ratio acquired from the enlargement / reduction ratio calculation unit 215. As a result, the image display apparatus 1 displays a video that is larger than the video data 160 in FIG. 20A, like video data 162 shown in FIG. In the present embodiment, the image display device 1 is shown as an example of enlarging with the upper right of the video data as a starting point, but other positions may be used as starting points.

  In the present embodiment, the example in which the video data of the image display device 1 is enlarged by the user 3 performing an enlargement operation on the tablet terminal 2 has been described. It is also possible to reduce the video data of the image display device 1 by performing a reduction operation in the opposite direction to the arrow 170 of FIG.

  Further, in the present embodiment, the example in which the video data of the image display device 1 is enlarged or reduced by operating the tablet terminal 2 has been described. However, the image display device 1 is an image display device according to the inclination of the tablet terminal 2. 1 may be moved up, down, left and right. Further, the tablet terminal 2 may adjust the video data displayed on the image display device 1 by using the rotation operation of the tablet terminal 2.

[Effect in the third embodiment]
As described above, according to the present embodiment, the user 3 performs a touch operation on the tablet terminal 2 after photographing the two-dimensional barcode displayed on the image display device 1, so that the image of the image display device 1 is displayed. Data can be easily expanded or reduced. Therefore, the user 3 can operate the size of the video data of the image display device 1 without performing a complicated operation, so that the operability is further improved.

<Fourth Embodiment>
A fourth embodiment of the present invention will be described. In the present embodiment, functions different from those in the first embodiment will be described in detail, and description of similar functions will be omitted.
In the first embodiment, the example is described in which the function is specified after the tablet terminal 2 has photographed the two-dimensional barcode printed in the vicinity of the button of the image display device 1 and identified the part. The function specifying unit 43 according to the present embodiment specifies the type of function before specifying the function of the image display device 1 related to the captured image, and displays an image based on the specified type of function and the captured image. The function of the device 1 is specified. In this embodiment, the type of function is specified on the application of the tablet terminal 2, and the display on the tablet terminal 2 is switched according to the function indicated by the two-dimensional barcode of the image display device 1 captured by the tablet terminal 2. Will be described.

  FIG. 21 is a diagram illustrating an example of the image display device 1 according to the present embodiment. The image display device 1 includes an input button 61, a two-dimensional barcode 62 indicating that the input button 61 is an input button, a power button 180, and a two-dimensional barcode indicating that the power button 180 is a power button. 181 is provided in advance.

  In the present embodiment, the user 3 shoots a two-dimensional barcode (not shown) indicating the help function printed in advance on the image display device 1 using the tablet terminal 2, so that the type of the target function is help. A case where the function is specified will be described.

  FIG. 22 is a diagram illustrating an example of a photographed image obtained by photographing the two-dimensional barcode of the image display device 1 with the tablet terminal 2. FIG. 22A is an example of a photographed image obtained by photographing the two-dimensional barcode 62 indicating the input button 61 by the user 3. The tablet terminal 2 uses the function specifying unit 43 to specify that the function corresponding to the Input button 61 is the input switching function based on the two-dimensional barcode 62, and displays the help screen 190 related to the input switching function on the tablet terminal. 2 is displayed. The help screen 190 displays a function name indicating that the function corresponding to the input button 61 is an input switching function, and a description regarding the function.

  FIG. 22B is an example of a photographed image obtained by photographing the two-dimensional barcode 181 indicating the Power button 180 following the photographing of FIG. 22A by the user 3. The tablet terminal 2 uses the function specifying unit 43 to specify that the function corresponding to the Power button 180 is the power control function based on the two-dimensional barcode 181, and displays the help screen 191 related to the power control function on the tablet terminal. 2 is displayed. The help screen 191 displays a function name indicating that the function corresponding to the Power button 180 is a power control function, and a description regarding the function.

  In the above description, the image display device 1 is printed with a two-dimensional barcode indicating a help function for displaying a description of any one of a plurality of functions of the image display device 1. Then, when the user 3 captures the two-dimensional barcode using the tablet terminal 2, the tablet terminal 2 specifies that the function type corresponding to the two-dimensional barcode is a help function, and based on the captured image. The help function of the function related to the identified part was executed based on the identified part. However, the method in which the tablet terminal 2 specifies that the type of the function that the user 3 wants to execute is the help function is not limited to the above example. The tablet terminal 2 specifies that the type of the function that the user 3 wants to execute is the help function based on the operation of the user 3 on the application that controls the image display device 1 installed in the tablet terminal 2. A help function of a function related to the specified part may be executed based on the part specified based on the captured image.

[Effects of the fourth embodiment]
As described above, according to the present embodiment, after the tablet terminal 2 specifies that the type of the function that the user 3 wants to execute is the help function, the image display device included in the captured image captured by the tablet terminal 2 The contents of the screen displayed on the tablet terminal can be switched in accordance with one part. As a result, when the user 3 wants to activate a function subordinate to a specific function, the operation time of the user 3 can be shortened, so that the convenience of the user 3 can be further improved.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment, A various deformation | transformation and change are possible within the range of the summary. is there.

41 photographing unit 43 function specifying unit 47 display control unit 48 operation input I / F unit 49 setting processing unit

Claims (15)

Photographing means for obtaining a photographed image including a part of an electronic device having a plurality of functions;
A specifying unit that specifies a function of the electronic device related to the captured image based on an image including an operation unit included in the electronic device included in the captured image ;
Display means for displaying an operation screen corresponding to the function specified by the specifying means;
Obtaining means for obtaining the operation content input on the operation screen;
Transmission means for transmitting setting instruction information for setting the operation of the function to the electronic device based on the operation content;
A device operating device characterized by comprising: Further comprising receiving means for receiving, from the electronic device, setting information related to the function specified by the specifying means;
The display means displays the operation screen including contents based on the setting information,
The acquisition means acquires update information obtained by updating the setting information,
The transmission means transmits the setting instruction information including the update information to the electronic device.
The apparatus operating device according to claim 1. The transmission unit transmits an exclusive instruction for prohibiting setting of operation of the electronic device by another device to the electronic device after the receiving unit has received the setting information.
The apparatus operating device according to claim 2. The identification unit is located at a position where a distance from the first operation unit among the plurality of operation units included in the electronic device included in the captured image is smaller than a distance from another operation unit included in the electronic device. The function is specified based on an identification image including information for identifying the first operation means provided,
The apparatus operating device according to any one of claims 1 to 3. The specifying unit specifies the function based on an identification image that is displayed on an operation screen displayed on the electronic device and is included in the captured image and includes information for identifying the function. To
The apparatus operating device according to any one of claims 1 to 3. The display means displays an image for receiving an instruction to display an explanation about the function when a size of a display area of the operation screen corresponding to the function satisfies a predetermined condition.
The apparatus operating device according to any one of claims 1 to 5 . The display means displays the operation screen including the captured image,
The acquisition means acquires the operation content for changing the size of the captured image included in the operation screen,
The transmission means transmits the setting instruction information for setting the size of an image displayed on the screen by the electronic device to the electronic device based on the size of the captured image indicated by the operation content. To
The apparatus operating device according to any one of claims 1 to 6 . The function is specified by an image including an operation unit included in the electronic device and included in the photographed image based on a user operation performed before the photographing unit acquires the photographed image. After specifying the type and specifying the type, the function corresponding to the operation means is specified based on the function type and an image including the operation means included in the electronic device ,
The apparatus operating device according to any one of claims 1 to 7 . Obtaining a captured image including a part of an electronic device having a plurality of functions;
Identifying a function of the electronic device related to the captured image based on an image including an operation unit included in the electronic device included in the captured image ;
Performing control to display an operation screen corresponding to the identified function;
Obtaining the operation content input on the operation screen;
Transmitting setting instruction information for setting the operation of the function to the electronic device based on the operation content;
A device operating method characterized by comprising: Receiving the setting information related to the function specified in the specifying step from the electronic device;
In the displaying step, the operation screen including contents based on the setting information is displayed,
In the obtaining step, obtaining update information obtained by updating the setting information;
In the transmitting step, the setting instruction information including the update information is transmitted to the electronic device.
The apparatus operating method according to claim 9 . After the receiving step, there is a step of transmitting an exclusive instruction for prohibiting setting of operation of the electronic device by another device to the electronic device,
The device operating method according to claim 10 . Determining whether a size of a display area of the operation screen corresponding to the function satisfies a predetermined condition;
A step of displaying an image for receiving an instruction to display an explanation about the function when a size of a display area of the operation screen satisfies the predetermined condition;
Further comprising:
The device operating method according to any one of claims 9 to 11 . Displaying the operation screen including the captured image;
Obtaining the operation content for changing the size of the captured image included in the operation screen;
Have
In the transmitting step, the setting instruction information for setting a size of an image displayed on the screen by the electronic device is transmitted to the electronic device based on a size of the captured image indicated by the operation content. And
The apparatus operating method according to any one of claims 9 to 12 . Based on a user operation performed before executing the step of acquiring the captured image, the type of the function specified by the image including the operation means included in the electronic device included in the captured image is specified. And further comprising steps
In the step of specifying the function of the electronic device, the function corresponding to the operation unit is specified based on the type of the function and an image including the operation unit included in the electronic device .
The device operating method according to any one of claims 9 to 13 . On the computer,
Obtaining a captured image including a part of an electronic device having a plurality of functions;
Identifying a function of the electronic device related to the captured image based on an image including an operation unit included in the electronic device included in the captured image ;
Performing control to display an operation screen corresponding to the identified function;
Obtaining the operation content input on the operation screen;
Transmitting setting instruction information for setting the operation of the function to the electronic device based on the operation content;
A device operation program characterized by causing