JP2014157463A - User interface device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the operability of a PC in a remote control operation.SOLUTION: An imager 16 outputs raw image data indicating a scene captured by an imaging surface. The output raw image data is converted into image data of a YUV form by a pre-processing circuit 20 and post-processing circuit 34. A CPU 26 searches the converted image data for one or two or more feature images indicating one or two or more GUI functions for operating a PC connected to the outside, and assigns the one or two or more GUI functions to a touch panel 38 on the basis of the search result. The CPU 26 also detects a GUI function assigned to the touch position on the touch panel 38, and transfers, to the PC, a command corresponding to the detected GUI function.

Description

  The present invention relates to a user interface device, and more particularly to a user interface device that displays one or more GUI (Graphic User Interface) images on a touch panel.

  An example of this type of device is disclosed in Patent Document 1. According to this background art, the touch panel is divided into a plurality of areas, and a predetermined operation is assigned to each area. This assignment is presented to the user when shifting to a full screen display. Thereby, simple and accurate switching of the reproduced image is realized.

JP 2009-10774 A

  However, the background art does not assume that an external device is remotely operated, and has limited operability.

  Therefore, a main object of the present invention is to provide a user interface device capable of enhancing the operability related to remote operation of an external device.

  A user interface device according to the present invention (10: reference numeral corresponding to the embodiment; the same applies hereinafter) is an imaging unit (16) for outputting an electronic image representing a scene captured on an imaging plane, and 1 for operating an external device. Alternatively, search means (S43 to S49) for searching for one or more feature images respectively representing two or more GUI functions from the output of the image pickup means, and one or more GUI functions based on the search result of the search means (touch panel) Allocating means (S51 to S53) to be assigned to 38), detecting means (S61 to S67) for detecting the GUI function assigned to the touch position on the touch panel in relation to the processing of the assigning means, and GUI detected by the detecting means Transfer means (S69) is provided for transferring a command corresponding to the function to the external device.

  Preferably, the assigning means assigns the GUI function represented by the feature image detected by the searching means to the touch panel corresponding to the position on the scene of the feature image detected by the searching means.

  Preferably, recording means (S13, S17 to S19) for recording the electronic image output from the imaging means in response to the recording operation, and reproducing means for reproducing the electronic image recorded by the recording means in response to the reproducing operation (S23 to S25) are further provided, and the search means pays attention to the electronic image reproduced by the reproduction means and executes the processing.

  More preferably, the reproducing means includes display means (S25) for displaying an electronic image on the touch panel, and the assigning means assigns one or more GUI functions to the touch panel in parallel with the processing of the display means.

  A user interface control program according to the present invention is for operating an external device on a processor (26) of a user interface device (10) including an imaging means (16) that outputs an electronic image representing a scene captured on an imaging surface. A search step (S43 to S49) for searching for one or two or more feature images respectively representing one or more GUI functions from the output of the imaging means, and one or more GUI functions based on the search result of the search step (38) an allocation step (S51 to S53), a detection step (S61 to S67) for detecting the GUI function allocated to the touch position on the touch panel in association with the process of the allocation step, and a detection step. A user interface control program for executing a transfer step (S69) for transferring a command corresponding to the GUI function to an external device. A gram.

  A user interface control method according to the present invention is a user interface control method executed by a user interface device (10) including an imaging means (16) for outputting an electronic image representing a scene captured on an imaging surface, and comprising an external device A search step (S43 to S49) for searching for one or more feature images respectively representing one or more GUI functions for operating the camera from the output of the imaging means, one or more based on the search result of the search step Assigning step (S51 to S53) for assigning the GUI function to the touch panel (38), detecting step (S61 to S67) for detecting the GUI function assigned to the touch position on the touch panel in relation to the process of the assigning step, and detecting A transfer step (S69) of transferring an instruction corresponding to the GUI function detected in the step to an external device is provided.

  An external control program according to the present invention includes an imaging means (16) for outputting an electronic image representing a scene captured on an imaging surface, and a processor (26) for executing processing according to an internal control program stored in a memory (48). An external control program supplied to a user interface device (10) comprising one or two or more feature images respectively representing one or more GUI functions for operating the external device are searched from the output of the imaging means. Search step (S43 to S49), assignment step (S51 to S53) for assigning one or more GUI functions to the touch panel (38) based on the search result of the search step, GUI function assigned to the touch position on the touch panel Corresponding to the detection step (S61 to S67) detected in relation to the processing of the allocation step, and the GUI function detected by the detection step For execution by the processor transfer step of transferring the decree to the external device (S69) in cooperation with the internal control program, which is an external control program.

  A user interface device (10) according to the present invention includes an imaging means (16) for outputting an electronic image representing a scene captured on an imaging surface, a capturing means (50) for capturing an external control program, and an external captured by the capturing means. A user interface device comprising a processor (26) for executing processing according to a control program and an internal control program stored in a memory (48), wherein the external control program is one or more for operating the external device. A search step (S43 to S49) for searching for one or more feature images each representing a GUI function from the output of the imaging means, and one or more GUI functions are assigned to the touch panel (38) based on the search result of the search step. Assigning step (S51 to S53), assigning the GUI function assigned to the touch position on the touch panel. A program for executing a detection step (S61 to S67) detected in connection with the internal control program and a transfer step (S69) for transferring a command corresponding to the GUI function detected by the detection step to an external device in cooperation with the internal control program It corresponds to.

  One or more feature images represent one or more GUI functions for operating an external device, respectively. When one or more feature images are captured by the imaging unit, the corresponding one or more GUIs are displayed. Functions are assigned to the touch panel. When the touch panel is touched, a command corresponding to the GUI function assigned to the touch position is transferred to the external device. This improves the operability related to the remote operation of the external device.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of one Example of this invention. It is a block diagram which shows the structure of one Example of this invention. FIG. 3 is an illustrative view showing one example of a mapping state of an SDRAM applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of the allocation state of the evaluation area in an imaging surface. FIG. 3 is an illustrative view showing one example of a configuration of a register applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of the sheet | seat with which the GUI image was drawn. It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2;

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the user interface device of this embodiment is basically configured as follows. The imaging unit 1 outputs an electronic image representing a scene captured on the imaging surface. The search means 2 searches the output of the imaging means 1 for one or more feature images respectively representing one or more GUI functions for operating an external device. The assigning means 3 assigns one or more GUI functions to the touch panel 6 based on the search result of the search means 2. The detecting means 4 detects the GUI function assigned to the touch position on the touch panel 6 in connection with the processing of the assigning means 3. The transfer unit 5 transfers an instruction corresponding to the GUI function detected by the detection unit 4 to the external device.

One or more feature images represent one or more GUI functions for operating an external device, respectively. When one or more feature images are captured by the imaging means 1, one or more corresponding images are displayed. A GUI function is assigned to the touch panel 6. When the touch panel 6 is touched, a command corresponding to the GUI function assigned to the touch position is transferred to the external device. This improves the operability related to the remote operation of the external device.
[Example]

  Referring to FIG. 2, the digital camera 10 of this embodiment includes a focus lens 12 and an aperture unit 14 driven by drivers 18a and 18b, respectively. The optical image that has passed through these members is irradiated onto the imaging surface of the imager 16 and subjected to photoelectric conversion. Thereby, a charge corresponding to the optical image is generated.

  When the camera mode is selected by the mode switch 28 md provided in the key input device 28, the CPU 26 instructs the driver 18 c, the pre-processing circuit 20 and the post-processing circuit 34 to execute moving image capture, and displays the moving image display on the LCD. Command the driver 36. In response to a vertical synchronization signal Vsync periodically generated from an SG (Signal Generator) (not shown), the driver 18c exposes the imaging surface and reads out the charges generated on the imaging surface in a raster scanning manner. From the imager 16, raw image data based on the read charges is periodically output.

  The preprocessing circuit 20 performs processing such as digital clamping, pixel defect correction, and gain control on the raw image data output from the imager 16. The raw image data subjected to these processes is written into the preprocessed image area 32a (see FIG. 3) of the SDRAM 32 through the memory control circuit 30.

  The post-processing circuit 34 reads the raw image data stored in the pre-processed image area 32a through the memory control circuit 30, and performs color separation processing, white balance adjustment processing, and YUV conversion processing on the read raw image data. The YUV format image data thus generated is written into the post-processing image area 32b (see FIG. 3) of the SDRAM 32 by the memory control circuit 30.

  The LCD driver 36 repeatedly reads out the image data stored in the post-processing image area 32b through the memory control circuit 30, and drives the LCD monitor 40 forming the touch panel 38 based on the read image data. As a result, a real-time moving image (through image) representing the scene captured on the imaging surface is displayed on the monitor screen.

  Referring to FIG. 5, an evaluation area EVA is allocated at the center of the imaging surface. The evaluation area EVA is divided into 16 in each of the horizontal direction and the vertical direction, and 256 divided areas form the evaluation area EVA. In addition to the above-described processing, the preprocessing circuit 20 shown in FIG. 2 executes simple Y conversion processing that simply converts raw image data into Y data.

  The AE evaluation circuit 22 integrates Y data belonging to the evaluation area EVA among the Y data generated by the preprocessing circuit 20 every time the vertical synchronization signal Vsync is generated. As a result, 256 integral values, that is, 256 AE evaluation values, are output from the AE evaluation circuit 22 in response to the vertical synchronization signal Vsync. The AF evaluation circuit 24 integrates the high-frequency component of Y data belonging to the evaluation area EVA among the Y data generated by the preprocessing circuit 20 every time the vertical synchronization signal Vsync is generated. As a result, 256 integral values, that is, 256 AF evaluation values, are output from the AF evaluation circuit 24 in response to the vertical synchronization signal Vsync.

  When the shutter button 28sh provided in the key input device 28 is in a non-operating state, the CPU 26 executes a simple AE process based on the 256 AE evaluation values output from the AE evaluation circuit 22, and sets an appropriate EV value. calculate. The aperture amount and exposure time that define the calculated appropriate EV value are set in the drivers 18b and 18c, whereby the brightness of the through image is roughly adjusted.

  When the shutter button 28 sh is half-pressed, the CPU 26 executes a strict AE process based on the 256 AE evaluation values output from the AE evaluation circuit 22. Thus, the optimum EV value is calculated, and the aperture amount and the exposure time that define the calculated optimum EV value are also set in the drivers 18b and 18c. As a result, the brightness of the through image is adjusted to the optimum value.

  The CPU 26 also executes AF processing based on the 256 AF evaluation values output from the AF evaluation circuit 24. The focus lens 12 is moved in the optical axis direction by the driver 18a in order to search for a focal point, and is arranged at the focal point found by this. As a result, the sharpness of the through image is improved.

  When the shutter button 28sh is fully pressed, the CPU 26 executes the still image capturing process by itself and instructs the memory I / F 44 to execute the recording process. One frame of image data representing the scene at the time when the shutter button 28sh is fully pressed is saved from the post-processing image area 32b to the recording image area 32c (see FIG. 3) by the still image capturing process. The memory I / F 44 reads one frame of image data stored in the recording image area 32 c through the memory control circuit 30 and records an image file containing the read image data on the recording medium 46.

  When the playback mode is selected by the mode switch 28md, the CPU 26 designates the latest image file recorded on the recording medium 44 and instructs the memory I / F 42 to play the designated image file. The memory I / F 42 reads the image data stored in the designated image file, and writes the read image data into the reproduction image area 32d (see FIG. 3) through the memory control circuit 30.

  The CPU 26 also instructs the LCD driver 36 to start playback image display. The LCD driver 36 reads the image data stored in the reproduction image area 32d through the memory control circuit 30, and drives the LCD monitor 40 based on the read image data. As a result, the reproduced image is displayed on the LCD monitor 40.

  When an update button 28ud provided on the key input device 28 is operated, the CPU 26 considers that an image update operation has been performed, designates another image file recorded on the recording medium 46, and reproduces the designated image file. Commands the memory I / F 44. As a result, the reproduced image on the LCD monitor 40 is updated.

  When the GUI setting button 28gui provided on the key input device 28 is operated, the CPU 26 considers that the GUI setting operation has been performed, and executes the GUI image search process for the image data stored in the reproduction image area 32d. . The GUI image search process is executed as follows.

  First, each of the flag FLG and the variable K is set to “0”, and a GUI image (= a rectangular frame in which characters are described inside) is searched from the target image data. When the GUI image is detected, the position and size of the rectangular frame defining the detected GUI image are registered in the Kth column of the register RGST shown in FIG. Further, the GUI function is specified based on the detected character on the GUI image, and the specified GUI function is registered in the Kth column of the register RGST. When the registration is completed, the variable K is incremented, and the search for another GUI image is resumed. The flag FLG is updated to “1” when at least one GUI image is detected.

  Accordingly, the GUI setting is performed in a state where the sheet ST on which the five GUI images M0 to M4 are drawn in the manner shown in FIG. 6 is photographed under the camera mode and the reproduced image representing the sheet ST is displayed on the LCD monitor 40. When the button 28gui is operated, the position, size, and GUI function of the GUI image M0 are registered in the 0th column of the register RGST, and the position, size, and GUI function of the GUI image M1 are registered in the first column of the register RGST. Is done. Similarly, the position, size, and GUI function of the GUI image M2 are registered in the second column of the register RGST, and the position, size, and GUI function of the GUI image M3 are registered in the third column of the register RGST, and the GUI image M4 The position, size, and GUI function are registered in the fourth column of the register RGST.

  Returning to FIG. 2, the touch sensor 42 forming the touch panel 38 detects the presence / absence of touch on the monitor screen and the touch coordinates, and gives the detection result to the CPU 26. A remote operation target PC (Personal Computer) is connected to the communication I / F 50.

  When the flag FLG indicates “1” when the GUI search process is completed, the CPU 26 executes the following process to control the PC based on the output of the touch sensor 42. Note that a reproduced image representing the target image data is continuously displayed on the LCD monitor 40.

  First, when a touch on the monitor screen is detected by the touch sensor 42, the CPU 26 detects the touch coordinates, and determines the positional relationship between the detected touch coordinates and the GUI image with reference to the register RGST. If the detected touch coordinates are on the GUI image, the CPU 26 detects the GUI function assigned to the touched GUI image with reference to the register RGST, and sends a command corresponding to the detected GUI function to the communication I / O. Transfer to PC through F50.

  Therefore, when the GUI image M0 is touched, a command corresponding to the operation of the “F1 key” is transferred to the PC, and when the GUI image M1 is touched, a command corresponding to the operation of the “F2 key” is transferred to the PC. Is done. When the GUI image M2 is touched, an instruction to start application software corresponding to “App1” is transferred to the PC, and when the GUI image M3 is touched, application software corresponding to “App2” is An instruction to start is transferred to the PC. Further, when the GUI image M4 is touched, a command related to the placement of the cursor is transferred to the PC.

  The CPU 26 executes a plurality of tasks including a shooting task shown in FIG. 7, a reproduction task shown in FIGS. 8 to 9 and a PC operation task shown in FIG. 10 in parallel under the control of the multitask OS. Note that control programs corresponding to these tasks are stored in the flash memory 48.

  Referring to FIG. 7, in step S1, moving image capturing is started, and in step S3, moving image display is started. As a result of the processing in step S1, raw image data representing a scene captured on the imaging surface is output from the imager 16, and corresponding YUV format image data is output from the post-processing circuit. Further, as a result of the processing in step S3, a through image based on the image data output from the post-processing circuit 34 is displayed on the LCD monitor 40.

  In step S5, it is determined whether or not the shutter button 28sh is half-pressed. As long as the determination result is NO, the simple AE process is repeated in step S7. As a result, the exposure amount is adjusted so that the brightness of the image belonging to the evaluation area EVA shows an appropriate value.

  When the determination result in step S5 is updated from NO to YES, the strict AE process is executed in step S9, and the AF process is executed in step S11. As a result, the exposure amount and the focus are adjusted so that the brightness and sharpness of the image belonging to the evaluation area EVA show the optimum values.

  In step S13, it is determined whether or not the shutter button 28sh has been fully pressed. In step S15, it is determined whether or not the operation of the shutter button 28sh has been released. If the determination result of step S15 is YES, it will return to step S5, and if the determination result of step S13 is YES, it will progress to step S17.

  In step S17, the still image capturing process is executed, and in the subsequent step S19, the memory I / F 44 is instructed to execute the recording process. One frame of image data representing a scene at the time when the shutter button 28 sh is fully pressed is saved from the post-processing image area 32 b to the recording image area 32 c by the still image capturing process. The memory I / F 44 reads the image data saved in this way from the recording image area 32 c through the memory control circuit 30, and records an image file containing the read image data on the recording medium 46. When the process of step S19 is completed, the process returns to step S5.

  Referring to FIG. 8, in step S21, the latest image file recorded on the recording medium 46 is designated, and in step S23, the memory I / F 44 is instructed to reproduce the designated image file. The memory I / F 44 reads the image data stored in the designated image file, and writes the read image data into the reproduction image area 32d through the memory control circuit 30.

  In step S25, the LCD driver 36 is instructed to start playback image display. The LCD driver 36 reads the image data stored in the reproduction image area 32d through the memory control circuit 30, and drives the LCD monitor 40 based on the read image data. As a result, the reproduced image is displayed on the LCD monitor 40.

  In step S27, it is determined whether a GUI setting operation has been performed. In step S29, it is determined whether an image update operation has been performed. If the decision result in the step S29 is YES, another image file recorded on the recording medium 46 is designated in the step S31, and thereafter, the process returns to the step S23.

  On the other hand, if the decision result in the step S27 is YES, a GUI image search process is executed in a step S33. As a result, the GUI image is searched from the image data stored in the reproduction image area 32d, that is, the target image data. The flag FLG is set to “1” when at least one GUI image is detected, and is set to “0” when no GUI image is detected.

  When the GUI image search process is completed, it is determined in step S35 whether or not the flag FLG indicates “1”. If a determination result is NO, it will return to Step S27, and if a determination result is YES, it will progress to Step S37. In step S37, the PC operation task is activated, and then the reproduction task is terminated.

  The GUI image search process in step S33 is executed according to a subroutine shown in FIG. First, in step S41, each of the flag FLG and the variable K is set to “0”. In step S43, a rectangular frame is searched from the target image data. In step S45, it is determined whether or not the rectangular frame has been detected. If a determination result is YES, it will progress to step S47 and will search for a character from the inside of the detected rectangular frame.

  In step S49, it is determined whether or not a character has been detected. If a determination result is NO, it will return to Step S43. On the other hand, if the determination result is YES, it is considered that the GUI image has been detected, and the process proceeds to step S51. In step S51, the position and size of the rectangular frame defining the detected GUI image are registered in the Kth column of the register RGST. In step S53, the GUI function is specified based on the detected character on the GUI image, and the specified GUI function is registered in the Kth column of the register RGST. When the process of step S53 is completed, the variable K is incremented in step S55, and then the process returns to step S43.

  In the second and subsequent processing in step S43, the detected rectangular frame is excluded from the search target. Therefore, for example, when three GUI images are present in the target image data, the processes in steps S47 to S55 are repeated three times.

  If the determination result in the step S45 is NO, it is determined whether or not the variable K is “1” or more in a step S57. If the determination result is NO, it is considered that no GUI image has been detected from the target image data, and the process returns to the upper layer routine. On the other hand, if the determination result is YES, it is considered that at least one GUI image has been detected from the target image data, the flag FLG is updated to “1” in step S59, and then the process returns to the upper hierarchy routine.

  Referring to FIG. 10, in step S61, it is repeatedly determined based on the output of touch sensor 42 whether or not the monitor screen is touched. When the determination result is updated from NO to YES, the touch coordinates are detected in step S63, and it is determined with reference to the register RGST whether or not the detected touch coordinates are on the GUI image. If a determination result is NO, it will return to step S61, and if a determination result is YES, it will be considered that the GUI image was touched and it will progress to step S67.

  In step S67, the GUI function assigned to the touched GUI image is detected with reference to the register RGST. In step S69, a command corresponding to the detected GUI function is transferred to the PC through the communication I / F 50. When the process of step S69 is completed, the process returns to step S61.

  As can be seen from the above description, the imager 16 outputs raw image data representing a scene captured on the imaging surface. The output raw image data is converted into YUV format image data by the pre-processing circuit 20 and the post-processing circuit 34. The CPU 26 searches for one or more GUI images respectively representing one or more GUI functions for operating an externally connected PC from the converted image data (S43 to S49), and based on the search result. One or more GUI functions are assigned to the touch panel 38 (S51 to S53). The CPU 26 also detects the GUI function assigned to the touch position on the touch panel 38 (S61 to S67), and transfers a command corresponding to the detected GUI function to the PC (S69).

  One or more GUI images represent one or more GUI functions for operating the PC, respectively. When one or more GUI images are captured by the imager 16, the corresponding one or more GUI functions are displayed. Is assigned to the touch panel 38. When the touch panel 38 is touched, a command corresponding to the GUI function assigned to the touch position is transferred to the PC. This improves the operability related to the remote operation of the PC.

  In this embodiment, the GUI image search process is executed in response to the GUI setting operation under the reproduction mode, and the PC operation task is activated when at least one GUI image is detected. However, when still image data is acquired in response to the GUI setting operation under the camera mode, GUI image search processing is executed on the acquired still image data, and at least one GUI image is detected Alternatively, the PC operation task may be activated. Thereby, it is possible to save the trouble of switching the operation mode from the camera mode to the reproduction mode.

  In this embodiment, a rectangular frame in which characters are described inside is assumed as a GUI image. However, a feature image such as a two-dimensional barcode may be assumed as a GUI image instead. . However, in this case, it is necessary to display the PC operation screen to which the detected GUI function is assigned on the LCD monitor 40 under the PC operation task.

  Further, in this embodiment, a multitask OS and a control program corresponding to a plurality of tasks executed thereby are stored in the flash memory 48 in advance. However, a part of the control program may be prepared in the flash memory 48 from the beginning as an internal control program, while another part of the control program may be acquired from an external server as an external control program. In this case, the above-described operation is realized by cooperation of the internal control program and the external control program.

  In this embodiment, the process executed by the CPU 26 is divided into a plurality of tasks as described above. However, each task may be further divided into a plurality of small tasks, and a part of the divided plurality of small tasks may be integrated with other tasks. Further, when each task is divided into a plurality of small tasks, all or part of the tasks may be acquired from an external server.

10 ... Digital camera 16 ... Imager 26 ... CPU
28 ... Key input device 32 ... SDRAM
40 ... LCD monitor 42 ... Touch sensor 50 ... Communication I / F

Claims (8)

Imaging means for outputting an electronic image representing a scene captured on the imaging surface;
Search means for searching for one or more feature images respectively representing one or more GUI functions for operating an external device from the output of the imaging means;
An assigning means for assigning one or more GUI functions to the touch panel based on a search result of the search means;
Detection means for detecting a GUI function assigned to a touch position on the touch panel in association with processing of the assignment means, and transfer means for transferring a command corresponding to the GUI function detected by the detection means to the external device A user interface device comprising:   2. The user interface device according to claim 1, wherein the assigning unit assigns a GUI function represented by the feature image detected by the searching unit to the touch panel corresponding to a position on the scene of the feature image detected by the searching unit. . A recording unit that records the electronic image output from the imaging unit in response to a recording operation; and a reproducing unit that reproduces the electronic image recorded by the recording unit in response to a reproducing operation;
The user interface device according to claim 1, wherein the search unit performs processing while paying attention to the electronic image reproduced by the reproduction unit. The reproduction means includes display means for displaying the electronic image on the touch panel,
The user interface device according to claim 3, wherein the assigning unit assigns the one or more GUI functions to the touch panel in parallel with the processing of the display unit. In a processor of a user interface device including an imaging unit that outputs an electronic image representing a scene captured on an imaging surface,
A search step of searching for one or more feature images respectively representing one or more GUI functions for operating an external device from the output of the imaging means;
Assigning one or more GUI functions to the touch panel based on the search result of the search step;
A detection step of detecting a GUI function assigned to a touch position on the touch panel in relation to the processing of the assignment step; and a transfer step of transferring a command corresponding to the GUI function detected by the detection step to the external device A user interface control program for executing A user interface control method executed by a user interface device including an imaging unit that outputs an electronic image representing a scene captured on an imaging surface,
A search step of searching for one or more feature images respectively representing one or more GUI functions for operating an external device from the output of the imaging means;
Assigning one or more GUI functions to the touch panel based on the search result of the search step;
A detection step of detecting a GUI function assigned to a touch position on the touch panel in relation to the processing of the assignment step; and a transfer step of transferring a command corresponding to the GUI function detected by the detection step to the external device A user interface control method comprising: An external control program supplied to a user interface device comprising an imaging means for outputting an electronic image representing a scene captured on an imaging surface, and a processor for executing processing according to an internal control program stored in a memory,
A search step of searching for one or more feature images respectively representing one or more GUI functions for operating an external device from the output of the imaging means;
Assigning one or more GUI functions to the touch panel based on the search result of the search step;
A detection step of detecting a GUI function assigned to a touch position on the touch panel in relation to the processing of the assignment step; and a transfer step of transferring a command corresponding to the GUI function detected by the detection step to the external device An external control program for causing the processor to execute in cooperation with the internal control program. Imaging means for outputting an electronic image representing a scene captured on the imaging surface;
A user interface device comprising: a capturing unit that captures an external control program; and a processor that executes processing according to the external control program captured by the capturing unit and the internal control program stored in a memory,
The external control program is
A search step of searching for one or more feature images respectively representing one or more GUI functions for operating an external device from the output of the imaging means;
Assigning one or more GUI functions to the touch panel based on the search result of the search step;
A detection step of detecting a GUI function assigned to a touch position on the touch panel in relation to the processing of the assignment step; and a transfer step of transferring a command corresponding to the GUI function detected by the detection step to the external device A user interface device corresponding to a program that executes in cooperation with the internal control program.

Images