JP2015219548A - Device control system, device control program, and device control apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device control mechanism with high practicability and with future potential.SOLUTION: A device control system comprises: a camera for photographing a predetermined real space; and an information processor for processing an image photographed by the camera. The information processor includes: device area control means for controlling an area of a device on the image photographed by the camera; device removal means for extracting an area to be a candidate for a device from the image photographed by the camera and for removing association between an area, which is found among areas associated with the device, and user interface information of the device; irradiation point detection means for detecting an irradiation point indicated by a laser pointer from the image photographed by the camera; operation detection means for detecting a predetermined operation from behavior of the irradiation point in the area of the device; and device control means for appropriately controlling the device in accordance with the detected operation.

Description

  The present invention relates to a technique for controlling a device.

  There are known user interfaces (UI) by gestures using a motion sensor by a depth camera (see, for example, Patent Document 1), and the user interfaces by these gestures are applied to control devices such as actual devices. Various studies have been conducted.

  The above-described user interface using gestures is advantageous in that operations can be performed only with movements of hands and bodies. However, if you try to use it as a user interface for an actual device such as a TV or a lighting device, you may not be able to get a fine accuracy due to the movement of your hands or body, or there may be a lot of misrecognitions. There are many problems as an interface.

  In addition, the user interface by gesture is applied when the sensor is installed on the device side because it is necessary to capture the user's appearance toward the device, so the sensor is provided on the device side. It was difficult. Further, when controlling a plurality of devices, a sensor is required for each of the devices, which is difficult to apply in terms of cost.

  The present invention has been proposed in view of the above-described conventional problems, and an object of the present invention is to provide a highly practical device control mechanism with a sense of future.

  In order to solve the above-described problems, the present invention includes a camera that captures a predetermined real space and an information processing device that processes a captured image of the camera, and the information processing device captures the camera. A device region management unit that manages a device region on the image; and a region that is a candidate for the device is extracted from the captured image of the camera, and the region that is not found among the regions associated with the device Device removal means for deleting the association with the user interface information of the device, irradiation point detection means for detecting the irradiation point by the laser pointer from the photographed image of the camera, and predetermined behavior from the behavior of the irradiation point in the device area An operation detection means for detecting the operation of the device, and a device control means for controlling the device corresponding to the detected operation. Obtain.

  In the present invention, it is possible to provide a highly practical device control mechanism with a sense of future.

It is a figure which shows the structural example of the system concerning one Embodiment of this invention. It is a figure which shows the internal structural example of each apparatus. It is a figure which shows the example of the information used for a process. It is a sequence diagram which shows the process example of embodiment. It is FIG. (1) which shows the example of operation of the device by a laser pointer. FIG. 10 is a second diagram illustrating an example of device operation by a laser pointer; It is a flowchart which shows the more detailed process example of embodiment. It is a figure which shows the example of the pattern for conversion. It is a flowchart which shows the example of a device addition process. It is a flowchart which shows the example of a device removal process.

  Hereinafter, preferred embodiments of the present invention will be described.

<Configuration>
FIG. 1 is a diagram showing a configuration example of a system according to an embodiment of the present invention. In FIG. 1, an information processing device 2 such as a PC (Personal Computer), a projector 3 for projecting an image, and a camera 4 for photographing are provided in a real space 1 such as a room where a TV, furniture, and the like are arranged. The projector 3 and the camera 4 are connected to the information processing apparatus 2. The projector 3 is disposed so as to project an image within a predetermined range of the real space 1, and the camera 4 is disposed so as to photograph the projection range of the projector 3. In addition, the information processing apparatus 2 is connected to the server 7 via the network 6. Furthermore, the information processing apparatus 2 is connected to a device 11 such as a television directly or via the network 6. The connection form between the information processing apparatus 2 and the device 11 may be wired or wireless. The user can control the device 11 disposed in the real space 1 by irradiating the real space 1 with a light beam by the laser pointer 5 and performing a pointing operation.

  FIG. 2 is a diagram illustrating an internal configuration example of each apparatus. In FIG. 2, the information processing apparatus 2 includes an overall control unit 201, an imaging control unit 202, a conversion pattern output unit 203, a video signal transmission unit 204, a video signal input unit 205, a conversion coefficient calculation unit 206, and an irradiation point detection unit 207. A conversion processing unit 208, a device setting unit 209, a communication unit 210, an operation determination unit 211, a UI image generation unit 212, and a device control unit 213. Each unit is realized mainly by computer hardware and a computer program executed on the computer hardware.

  The projector 3 includes a video signal input unit 301, a video processing unit 302, a drive control unit 303, a lamp power source 304, a lamp 305, a motor 306, a color wheel 307, an image forming unit 308, and a synchronization signal output unit 309. Each unit excluding the lamp power supply 304, the lamp 305, the motor 306, and the color wheel 307 is mainly realized by computer hardware and a computer program executed thereon. Note that illustration of optical systems such as lenses and mirrors is omitted.

  The overall control unit 201 of the information processing apparatus 2 has a function of performing overall control of processing in the information processing apparatus 2.

  The shooting control unit 202 has a function of inputting a synchronization signal from the synchronization signal output unit 309 of the projector 3 and controlling the shooting period for the camera 4. Here, the synchronization signal prevents the emission color of the laser pointer 5 (FIG. 1) from covering the light color projected from the projector 3 and prevents the irradiation point of the laser pointer 5 from being detected. Is. More specifically, when the image forming unit 308 of the projector 3 is configured by one DMD (Digital Micromirror Device) element and the color wheel 307 is used for projecting a color image, the emission color of the laser pointer 5 and The synchronization signal indicates a period in which the color at the rotational position of the color wheel 307 is not covered. When the image forming unit 308 of the projector 3 is configured by an LCD (Liquid Crystal Display) element, the synchronization signal indicates a period during which no image is projected in units of frames.

  The conversion pattern output unit 203 has a function of outputting a conversion pattern to be projected from the projector 3 in the initial setting mode. The conversion pattern calculates a conversion coefficient (matrix value) for matching the pixel position of the image projected by the projector 3 and the pixel position of the image of the real space 1 captured by the camera 4 by projective transformation. Used for.

  The video signal transmission unit 204 has a function of outputting a video signal to the projector 3. More specifically, the conversion pattern is output from the conversion pattern output unit 203 in the initial setting mode, and the video signal of the entire black image (in the case of projecting a UI image, the UI image generation unit 212 in the normal mode). The video signal including the UI image to be output by is output.

  The video signal input unit 205 has a function of inputting a video signal captured by the camera 4.

  In the initial setting mode, the conversion coefficient calculation unit 206 calculates a conversion coefficient for projective conversion that matches the pixel position of the captured image of the conversion pattern with the pixel position of the original conversion pattern (projection image), and calculates the calculated conversion coefficient It has a function to hold. That is, the conversion coefficient calculation unit 206 calculates a conversion coefficient based on the conversion pattern output from the conversion pattern output unit 203 and the captured image of the video signal output from the video signal input unit 205, and holds the calculated conversion coefficient. To do.

  The irradiation point detection unit 207 has a function of detecting the irradiation point by the laser pointer 5 (detecting the coordinate value of the irradiation point in the captured image) from the captured image of the video signal output from the video signal input unit 205 in the normal mode. doing.

  The conversion processing unit 208 performs coordinate conversion (projection conversion) of the coordinate value in the captured image of the irradiation point detected by the conversion processing unit 208 into the coordinate value in the projection image based on the conversion coefficient held by the conversion coefficient calculation unit 206. It has a function.

  The device setting unit 209 has a function of setting the relationship between the coordinate range of the irradiation point by the laser pointer 5 and the UI information of the device 11 such as a device to be controlled in the device setting mode. That is, the device setting unit 209 extracts the outline of a possible object of the device 11 from the image of the video signal output from the video signal input unit 205 and displays it on the monitor of the information processing apparatus 2 or the real space 1. The device candidates are displayed, and the user is made to associate the device 11 with the UI information. At this time, the device setting unit 209 internally holds the coordinate information of the extracted contour (including the contour that has not been selected, but it is desirable to exclude the contour related to the deleted device). In addition, the device setting unit 209 internally holds the coordinate information of the contour associated with the UI information. Note that the user may be allowed to designate the contour coordinates of the device on the photographed image and then associate the device 11 with the UI information.

  In the normal mode, the device setting unit 209 extracts a new device candidate area from a captured image of the camera 4 at a predetermined timing (at the start of processing, a predetermined cycle, etc.), and extracts the extracted area and the device UI. It also has a function for associating information. That is, the device setting unit 209 extracts the contour of a possible object of the device 11 from the image of the video signal output from the video signal input unit 205, compares it with the coordinate information of the contour held internally, If it is a new outline, the possibility of adding a device to the monitor of the information processing apparatus 2 or the real space 1 is displayed, and further candidates for the device are displayed so that the user can respond to the device 11 and the UI information. Let them be attached.

  In the normal mode, the device setting unit 209 extracts a device candidate area from the captured image of the camera 4 at a predetermined timing (at the start of processing, a predetermined cycle, etc.), and is associated with the UI information at this time. When the existing area has disappeared, it also has a function of deleting the association between the area and the UI information of the device. That is, the device setting unit 209 extracts the contour of the object that may be the device 11 from the image of the video signal output from the video signal input unit 205, and is associated with the UI information that is internally held. Compared with the coordinate information of the contour, if there is no coordinate information of the contour associated with the UI information, the correspondence between the region and the UI information of the device is deleted assuming that the corresponding device is removed. To do.

  The communication unit 210 has a function of performing data communication with the outside. More specifically, when the device setting unit 209 does not hold the UI information of the corresponding device 11, the UI information is acquired from the server 7 via the network 6. Further, when the device control unit 213 controls the device 11 via the network, it mediates communication with the device 11.

  The operation determination unit 211 determines whether or not the irradiation point by the laser pointer 5 has entered the coordinate range (area) of any device 11 and whether or not a predetermined operation has been performed on the UI displayed by entering the coordinate range. It has a function to determine whether. In order to be determined as the predetermined operation, a predetermined behavior is given as a condition such that the irradiation point stays in the region for a predetermined time so as not to make an erroneous determination when the irradiation point simply passes.

  The UI image generation unit 212 has a function of generating a corresponding UI image in accordance with the operation content determined by the operation determination unit 211. In addition, when displaying also the response of the result of having controlled to the device 11, a response image is produced | generated as a part of UI image.

  The device control unit 213 has a function of controlling the device 11 directly or via the communication unit 210 according to the operation content determined by the operation determination unit 211.

  On the other hand, the video signal input unit 301 of the projector 3 has a function of inputting a video signal for projection from the video signal transmission unit 204 of the information processing apparatus 2.

  The video processing unit 302 has a function of performing various types of image processing on the video signal input by the video signal input unit 301. Specifically, image processing such as contrast, brightness, saturation, hue, RGB gain, sharpness, and enlargement / reduction is performed.

  The drive control unit 303 has a function of controlling the lamp power source 304, the motor 306, and the image forming unit 308 in accordance with the video signal to be projected.

  The lamp power supply 304 has a function of supplying power to the lamp 305 serving as a light source.

  The motor 306 has a function of synchronously rotating the color wheel 307 for projecting a color image when the image forming unit 308 is configured with one DMD element.

  The image forming unit 308 has a function of reproducing a video signal that is an output of the video processing unit 302, and includes a DMD element or an LCD element.

  The synchronization signal output unit 309 has a function of outputting a synchronization signal for photographing to the information processing device 2 based on the control timing of the drive control unit 303.

  FIG. 3 is a diagram illustrating an example of information used for processing in the information processing apparatus 2, which is information that is mainly set by the device setting unit 209 and is referenced by the operation determination unit 211. In FIG. 3, the UI existence table holds the effective coordinate range of the image after coordinate conversion and the UI information ID in association with each other. The UI information ID is information for identifying UI information. For example, if the first device is “TV”, the upper left coordinate of the casing outline is (100,100), and the lower right coordinate is (200,200), the effective coordinate range is “(100,100) ⇒ (200,200)” The UI information ID is, for example, “UI0001”.

  Each UI information ID is associated with UI information, and the UI information includes a UI element ID, an element pattern, a graphic ID, and an action ID. The UI element ID is information for identifying a UI element. The element pattern is information indicating the type of UI element. The graphic ID is information for identifying a graphic file when an image corresponds to a UI element. The action ID is information for identifying an action definition file indicating an operation condition in which an operation on the UI element is valid and a control content for the device. The operation condition includes, for example, detection of an irradiation point for a predetermined time on a predetermined area of the UI element. Examples of control contents include power ON / OFF, volume increase / decrease, and next / previous channel.

<Operation>
FIG. 4 is a sequence diagram showing a processing example of the above embodiment. An outline of processing in the normal mode will be described. Details of the process including the initial setting mode and the device setting mode will be described later.

  In FIG. 4, when the user performs pointing with respect to the real space 1 using the laser pointer 5 (step S101), the information processing apparatus 2 performs irradiation point detection and coordinate conversion (step S102).

  Next, the information processing apparatus 2 determines whether or not the coordinate value after conversion of the irradiation point is within the effective coordinate range of any device 11 in the UI presence table (FIG. 3) (step S103).

  Next, if it is within the effective coordinate range, a UI image is projected by the projector 3 based on the corresponding UI information, and the UI operation standby mode is entered (step S104). When generating the UI image, if the state (channel, volume, etc.) of the device 11 to be controlled is required, information is acquired by communicating with the device 11. Note that there is a case where the on / off of the device 11 is controlled by the irradiation point staying in the effective coordinate range for a predetermined time without displaying the UI image.

  FIG. 5 shows that the irradiation point of the laser pointer 5 is detected for a predetermined time within the effective coordinate range of “TV” which is one of the devices 11, and “−Channel 4ch +” and “− “Volume 21+” is projected.

  Returning to FIG. 4, when the user performs pointing on the real space 1 using the laser pointer 5 in the UI operation standby mode (step S <b> 105), the information processing apparatus 2 performs irradiation point detection and coordinate conversion. This is performed (step S106). In this case, the pointing (step S105) may be continuously performed while irradiating light from the first pointing (step S101).

  Next, the information processing apparatus 2 determines whether the UI operation is defined for the displayed (projected) UI element image (step S107).

  If it is determined that the UI operation is defined, the control defined in the UI element is performed on the corresponding device 11 (step S108). FIG. 6 shows a state in which pointing is performed by the laser pointer 5 on the “+” portion of the UI image “−Channel 4ch +”. As a result, the television channel is up.

  Returning to FIG. 4, the device 11 returns a response as a result of the control (for example, if the control is to change the channel, the channel number after the change, if the control is to change the volume, etc., the volume level after the change). In step S109, the information processing apparatus 2 can project a response image (changed channel number, volume level, etc.) (step S110). Thereafter, the information processing apparatus 2 cancels the UI operation standby mode (step S111).

  FIG. 7 is a flowchart showing a more detailed processing example of the above embodiment. In FIG. 7, when the information processing apparatus 2 starts processing, the information processing device 2 branches the processing depending on the current mode (step S121).

  In the initial setting mode (initial setting in step S121), an image of the conversion pattern is output from the conversion pattern output unit 203, and is output from the video signal transmission unit 204 to the projector 3 (step S122). 8A and 8B are diagrams showing examples of conversion patterns. FIG. 8A shows an example of a check pattern (checkered pattern), FIG. 8B shows an example of a lattice pattern, and FIG. 8C shows an example of a dot pattern.

  Returning to FIG. 7, the projector 3 receives a projection video signal from the video signal input unit 301, performs predetermined image processing in the video processing unit 302, and then forms an image in the image forming unit 308 by the drive control unit 303. Do. This is projected onto the real space 1 by the light source of the lamp 305.

  The camera 4 captures the real space 1 in synchronization with the synchronization signal under the control of the capture control unit 202, and the information processing apparatus 2 inputs the video signal captured by the camera 4 via the video signal input unit 205 (step S123). .

  The conversion coefficient calculation unit 206 calculates and holds a conversion coefficient for matching the conversion pattern output from the conversion pattern output unit 203 and the image of the video signal input by the video signal input unit 205 (step S124). ). Then, the process returns to the mode determination (step S121).

  On the other hand, in the device setting mode (device setting in step S121), the device setting unit 209 inputs a video signal captured by the camera 4 from the video signal input unit 205 (step S125), and extracts a contour from the image. Then, highlighting is performed (step S126).

  Then, selection of a contour and a device is received from the user (step S127), coordinate conversion of the contour is performed (step S128), and a UI presence table is set (step S129). That is, the coordinate range of the contour selected by the user is set as the effective coordinate range of the UI presence table, and the UI information ID corresponding to the selected device is associated. In addition, the device setting unit 209 internally holds coordinate information of the extracted contour (including a contour that has not been selected, but it is desirable to exclude the contour related to the deleted device). Separately from this, the coordinate information of the contour associated with the UI information is held internally. In the device setting mode, information on devices already set (valid coordinate range, UI information ID, etc.) can be deleted by user selection.

  On the other hand, in the normal mode (normal in step S121), is the predetermined timing for performing the device addition process (timing at the start of the process, every predetermined time, etc., excluding waiting for UI operation described later)? If it is determined (step S130) and the predetermined timing is reached (Yes in step S130), a device addition process is executed (step S131).

  FIG. 9 is a flowchart illustrating an example of device addition processing (step S131 in FIG. 7). In FIG. 9, the device setting unit 209 inputs a video signal captured by the camera 4 from the video signal input unit 205 (step S151), and extracts an outline of an object that may be a device from the image (step S151). S152).

  The device setting unit 209 internally retains the coordinate information of the extracted contour, and the previously extracted contour (including the contour that was not selected, but it is desirable to exclude the contour related to the deleted device) Is compared with the coordinate information (step S153), and it is determined whether or not it is a new contour (step S154).

  When the outline is a new outline (Yes in step S154), the possibility of adding a device to the monitor of the information processing apparatus 2 or the real space 1 is displayed (step S155), and the outline is highlighted (step S156). ).

  Then, the device receives a device selection (when a new contour is present, the selection of the contour is accompanied) (step S157), performs coordinate conversion to the coordinate value on the projected image of the contour (step S158), and the UI. The presence table is set (step S159). That is, the coordinate range of the new contour is set as the effective coordinate range of the UI presence table, and the UI information ID corresponding to the selected device is associated. In addition, the device setting unit 209 internally holds coordinate information of the extracted contour (including a contour that has not been selected, but it is desirable to exclude the contour related to the deleted device).

  Next, returning to FIG. 7, it is determined whether or not it is a predetermined timing for performing device removal processing (timing at the start of processing or every predetermined time, excluding waiting for UI operation described later) (step S132). ), If it is a predetermined timing (Yes in step S132), a device removal process is executed (step S133).

  FIG. 10 is a flowchart showing an example of the device removal process (step S133 in FIG. 7). In FIG. 10, the device setting unit 209 inputs a video signal captured by the camera 4 from the video signal input unit 205 (step S161), and extracts an outline of an object that may be a device from the image (step S161). S162).

  The device setting unit 209 compares the extracted coordinate information of the contour with the coordinate information of the contour associated with the UI information held internally (step S163), and the contour information associated with the UI information is compared. If the coordinate information disappears without being extracted (Yes in step S164), it is determined that the corresponding device has been removed, and the association between the area and the device UI information is deleted from the UI presence table (step S165). . By deleting the association, the contour associated with the held UI information is also updated.

  In order not to delete the association between the valid device area and the device UI information due to misrecognition, an alert is displayed to the user prior to the deletion, and the deletion is performed after waiting for the confirmation operation. You may make it perform. It is also possible to reduce the setting effort by separately holding a combination of the area deleted from the UI presence table and the UI information of the device so that the same device can be selected again.

  The device may be removed for the convenience of the user. If the UI information remains associated, a UI image is projected even when the laser pointer 5 is used to point to an area where no device exists. There are inconveniences. However, the automatic device removal process as described above eliminates the inconvenience because the association between the device that does not exist at that time and the UI information is deleted. Note that when the device is moved within the field of view of the projector 3 and the camera 4, the new area and UI information are associated with each other by the above-described manual device setting or semi-automatic device addition processing.

  In addition, although the case where the timing of the device addition processing and the timing of the device removal processing are separated has been described, when both are executed at the same timing, similar processing (video signal input, contour extraction, etc.) is performed. ) Can be performed in common. When the device addition process and the device removal process are performed at the same timing, when the device is moved within the field of view of the projector 3 and the camera 4, the correspondence change of UI information from the previous area to the new area may be performed simultaneously. it can.

  Next, returning to FIG. 7, the video signal for projecting the entire black image is output from the video signal transmission unit 204 to the projector 3 except when the UI operation is on standby (step S134). The projector 3 inputs a video signal from the video signal input unit 301, performs predetermined image processing by the video processing unit 302, and then forms an image on the image forming unit 308 by the drive control unit 303. This is projected onto the real space 1 by the light source of the lamp 305.

  The camera 4 captures the real space 1 in synchronization with the synchronization signal under the control of the capture control unit 202, and the information processing apparatus 2 inputs the video signal captured by the camera 4 via the video signal input unit 205 (step S135). .

  The irradiation point detection unit 207 detects the irradiation point of the laser pointer 5 from the image of the video signal input by the video signal input unit 205 (step S136).

  The conversion processing unit 208 performs coordinate conversion of the irradiation point based on the conversion coefficient held by the conversion coefficient calculation unit 206 (step S137).

  Next, when the UI operation standby mode is not set (No in step S138), the operation determination unit 211 determines that the coordinate value after the coordinate conversion of the irradiation point is not included in any effective coordinate range of the UI presence table. If so (No in step S139), the process returns to the mode determination (step S121).

  When it is determined that the coordinate value after the coordinate conversion of the irradiation point is in any effective coordinate range of the UI presence table (Yes in step S139), the UI image generation unit 212 corresponds to the corresponding UI image based on the UI information. And the video signal transmission unit 204 adds it to the video signal (step S140). Then, the UI operation standby mode is entered (step S141), and the process returns to the mode determination (step S121).

  On the other hand, in the UI operation standby mode (Yes in step S138), the operation determination unit 211 determines that the coordinate value after the coordinate conversion of the irradiation point is a predetermined UI operation for the activated UI element. If it is (Yes in step S142), the device control unit 213 transmits a control signal to the corresponding device 11, and receives a response when a response is returned (step S143).

  When the response is received, the UI image generation unit 212 generates a corresponding response image, and the video signal transmission unit 204 adds the response image to the video signal (step S144). Then, the UI operation standby mode is canceled (step S145), and the process returns to the mode determination (step S121). If the UI operation standby mode continues for a predetermined time, it is forcibly canceled.

<Summary>
As described above, according to this embodiment, it is possible to provide a highly practical device control mechanism with a sense of future.

  The present invention has been described above by the preferred embodiments of the present invention. While the invention has been described with reference to specific embodiments, various modifications and changes may be made to the embodiments without departing from the broad spirit and scope of the invention as defined in the claims. Obviously you can. In other words, the present invention should not be construed as being limited by the details of the specific examples and the accompanying drawings.

DESCRIPTION OF SYMBOLS 1 Real space 11 Device 2 Information processing apparatus 201 General control part 202 Image | photographing control part 203 Conversion pattern output part 204 Video signal transmission part 205 Video signal input part 206 Conversion coefficient calculating part 207 Irradiation point detection part 208 Conversion processing part 209 Device setting Unit 210 communication unit 211 operation determination unit 212 UI image generation unit 213 device control unit 3 projector 301 video signal input unit 302 video processing unit 303 drive control unit 304 lamp power source 305 lamp 306 motor 307 color wheel 308 image forming unit 309 synchronization signal output Part 4 Camera 5 Laser pointer 6 Network 7 Server

Special table 2013-517051 gazette

Claims (7)

A camera for photographing a predetermined real space;
An information processing device for processing a photographed image of the camera,
The information processing apparatus includes:
Device area management means for managing the area of the device on the captured image of the camera;
A device removing unit that extracts a device candidate region from the photographed image of the camera and deletes the association between the region and the user interface information of the device for a region that is not found among the regions associated with the device. When,
An irradiation point detecting means for detecting an irradiation point by a laser pointer from a photographed image of the camera;
Operation detecting means for detecting a predetermined operation from the behavior of the irradiation point in the region of the device;
A device control system comprising device control means for controlling the corresponding device in accordance with the detected operation. The device control system according to claim 1,
The device removal unit requests the user to confirm the deletion before deleting the association between the area and the user interface information of the device. In the apparatus control system as described in any one of Claim 1 or 2,
An apparatus control system comprising: a device adding unit that extracts a region as a new device candidate from a captured image of the camera, and associates the extracted region with user interface information of the device. In the equipment control system according to any one of claims 1 to 3,
A projector for projecting a predetermined image in the real space;
The information processing apparatus includes:
Coordinate value management means for managing the correspondence between the coordinate value on the image of the projector and the coordinate value on the captured image of the camera;
A user interface image output means for outputting a predetermined user interface image as a projection image to the projector when a predetermined operation is detected from the behavior of the irradiation point in the device region;
The device control system, wherein the operation detection means detects a predetermined operation from a behavior of the irradiation point in a display area of the user interface image. In the apparatus control system according to claim 4,
The apparatus control system, wherein the operation detection unit enters a UI operation standby mode by projecting the user interface image, and cancels the UI operation standby mode by detecting a predetermined operation. Computer
Captured image acquisition means for acquiring a captured image from a camera that captures a predetermined real space,
Device area management means for managing the area of the device on the captured image of the camera;
A device removing unit that extracts a device candidate region from the photographed image of the camera and deletes the association between the region and the user interface information of the device for a region that is not found among the regions associated with the device. ,
Irradiation point detecting means for detecting an irradiation point by a laser pointer from a photographed image of the camera,
Operation detection means for detecting a predetermined operation from the behavior of the irradiation point in the device region;
An apparatus control program that functions as device control means for controlling the corresponding device in accordance with a detected operation. Captured image acquisition means for acquiring a captured image from a camera that captures a predetermined real space;
Device area management means for managing the area of the device on the captured image of the camera;
A device removing unit that extracts a device candidate region from the photographed image of the camera and deletes the association between the region and the user interface information of the device for a region that is not found among the regions associated with the device. When,
An irradiation point detecting means for detecting an irradiation point by a laser pointer from a photographed image of the camera;
Operation detecting means for detecting a predetermined operation from the behavior of the irradiation point in the region of the device;
A device control apparatus comprising device control means for controlling the corresponding device in accordance with the detected operation.