JP2008529147A - Device control method - Google Patents

Abstract

The present invention, which describes a method for controlling the device D _1, D _2, D _3, the method directs the pointing device 1 with a camera 2 in the apparatus D _1, D _2, D ₃ should be controlled A step of generating an image 3 of the target area A directed by the pointing device 1, and a unique visual associated with the devices D ₁ , D ₂ , D ₃ to be interpreted and to be controlled Determining the presence of unique identifiers V ₁ , V ₂ , V ₃ . Based on the analysis of the target area image, the devices D ₁ , D ₂ , D ₃ to which the pointing device 1 is directed are identified and a control signal 17 is sent to the identified devices D ₁ , D ₂ , D ₃ . The invention relates to a system for controlling the devices D ₁ , D ₂ , D ₃ and to such devices D ₁ , D ₂ , D ₃ . Using the present invention, any of a number of controllable devices D ₁ , D ₂ , D ₃ can be easily identified by visual identifiers V ₁ , V ₂ , V ₃ . In the preferred embodiment, the visual identifiers V ₁ , V ₂ , V ₃ are triggered by trigger signals T ₁ , T ₂ , T.

Description

  The present invention relates to an apparatus control method and system, and to such an apparatus.

  Remote control is generally used for controlling devices such as consumer electronics devices such as televisions, DVD players, tuners, and the like. However, in the average home, a large number of remote controls are required, one for each consumer electronics device. Even those familiar with the consumer electronics devices they own, it is difficult to remember what purpose each button on each remote control is for. Furthermore, the on-screen menu driven navigation available for certain consumer electronics is not intuitive for users who do not have detailed knowledge of the options available for the device. As a result, the user needs to continuously examine the menu displayed on the screen to find the option he is looking for, and then look down on the remote control to find the appropriate button. The buttons may be given a non-intuitive naming or abbreviation. In addition, remote control buttons perform additional functions that are accessed by first pressing the mode button.

  One remote control device may be used to control multiple devices. Since the remote control allows one device to distinguish another device, the remote control must be provided with a dedicated button for each device, so that the user can interact with the desired device before interacting with it. You need to press the appropriate button explicitly.

  A possible alternative to normal remote control is a laser pointer or a pointer such as a “wand”. Pointers that have been available so far have been popular in recent years and are typically passive in nature because they are only used to indicate objects, typically to show screen or projection items to audience members. It is a device. However, developments in the field of camera technology have led to the wide use of handheld pointing device types, including cameras, for remote control of electronic devices, because the camera follows the point where the user points the pointing device. Used for. However, even this type of remote control remains a problem for the user to determine the device the pointing device is pointing at, for example, a television or computer monitor bank of the same shape and appearance that looks similar It makes it more difficult when the devices are similar in shape and appearance, such as consumer electronics such as some video cassette recorders, DVD players and the like. When the device is placed remotely under normal circumstances, it is no longer possible to do this in poor viewing conditions, for example in a darkened room.

  Accordingly, it is an object of the present invention to provide an easy and intuitive way to automatically identify a device to be controlled among a number of devices.

  In order to achieve the above object, the present invention provides a method for controlling an apparatus, the method directing a pointing device having a camera to an apparatus to be controlled, and a target pointed by the pointing device. Generating an image of the area. The target area image is interpreted to determine the presence of a visual identifier associated with the device to be controlled, the device to which the pointing device is directed is identified based on the target area image analysis, and the control signal is To the identified device.

  The term “visual identifier” is understood to mean an identifier other than “visible” by the human eye or by the camera. Such visual identifiers are generated in any region of the electromagnetic spectrum, for example an optical signal such as a flash or other change in brightness. For example, if the pointing device is pointed at the monitor, the area of the monitor screen will flash or change color or hue. The visual identifier is invisible to the human eye but can be identified by the camera of the pointing device. Another possible type of visual identifier is a simple gesture that can be seen for cases where a part of the device is moved, and the movement is recorded by the camera and interpreted by image analysis.

  According to the present invention, a system for controlling an apparatus includes a pointing device having a camera that generates an image of a target area in a direction in which the pointing device is directed, and an image that analyzes the image to determine the presence of a visual identifier Has an analysis unit. In addition, the system manages device list management that manages a list of possible devices, their associated visual identifiers, and / or trigger signals associated to activate the corresponding visual identifiers. Have a unit. A control signal is generated by the control signal generation unit for the device to be controlled according to the result of the image analysis and transmitted to the device to be controlled by an appropriate interface. The units of the system, such as the image analysis unit, the control signal generation unit, used for interpreting, analyzing and generating signals between the device to be controlled and the pointing device are referred to as “device control interface” for convenience. So simply called.

  An apparatus for use in the method according to the present invention has a visual identifier, which is activated in response to an action performed by the pointing device and identifies the apparatus from among a number of actions. To play a role. Such an apparatus is a personal computer, consumer electronics device, household appliance, or any type of electronically or electrically controllable apparatus in any environment. The visual identifier is physically superimposed on the device, for example a tag, sticker or other object type. Similarly, the visual identifier of the device is implemented as an integrated part of the device, for example as a display area of the device, such as a computer monitor. The device is identified by its visual identifier or by activation of the visual identifier in response to a specific trigger signal.

The method according to the present invention offers the possibility of increased use of the pointing device. Such a pointing device is a specific universal control tool, because it is necessary to point the device or object to only one pointing device for a control signal to be generated based on the generated image. In particular, the method described herein allows a user to easily identify and interact with any device, even if all devices are similar or identical in appearance, or the surroundings are dark. can do. The functionality of this pointing device, along with its convenient pointing modalities, as described above, is combined to make the present invention a powerful and practical tool for countless situations in everyday life.
The dependent claims and the subsequent description disclose particularly advantageous embodiments and features of the invention.

  The purpose of the pointing device of the present invention is not only to identify one device among a number of devices, but also to allow the user to interact with the identified device, so that the image generated by the camera is It is preferably further analyzed to determine the options displayed on the device selected by the user. Based on this image analysis, appropriate control signals are generated and sent to the device to cause the device to perform the desired user options. For example, the user shows the screen of the device to navigate through the menu hierarchy displayed on the screen. Once the device is identified, the image is analyzed to infer any options that the user is showing. However, the present invention is used as a typical multi-device remote control, where commands for each device are entered by the user by pressing one or more buttons on the remote control, but the user points to the pointing device. The device, the device for which the command is intended, is automatically identified according to the present invention.

  In a preferred embodiment of the present invention, the pointing device transmits a trigger signal, which is received by any device in the vicinity provided in a suitable receiver. Alternatively, the trigger signal is transmitted from a source other than the pointing device, for example in contact with the pointing device. In response to the trigger signal, the activator of the device activates its visual identifier. An activator is, for example, a motor that moves a part of the device, which is a visual identifier of the device, or that emits electromagnetic radiation in the visible or invisible region as required. Appropriate electronic circuit to be released in part. Thus, the visual identifier of the device has a visual signal emitted by an LED or light valve. Similarly, if the device is a television or computer monitor, the visual signal is generated by changing the monitor in part in brightness or color in a predefined manner. Another possible visual identifier implementation is a physical element that is built into or attached to the device and can generate the necessary electromagnetic radiation.

  Depending on the feasibility of the system, it may be desirable to identify one device from a group of devices by means of a unique trigger signal or unique visual identifier, in which case the term “unique” Is understood to apply within this group of devices.

  Thus, in a preferred further preferred embodiment of the present invention, for each device in the group of devices, a single trigger signal is sufficient to activate any of these visual identifiers. As is the case, a unique visual identifier is provided. All devices respond to the trigger signal when they are within the range of their transmission. The device currently pointed by the pointing device appears in the image of the target area, along with its visual identifier, and the identity of the device is determined based on image analysis of the unique visual identifier that appears in the image of the target area Is done. Visual identifiers differ in color, intensity, duration of activity, frequency, etc., and the term “frequency” here applies to the rate at which visual identifiers change in appearance, for example, Flush 10 times per second so that the visual identifier “works” at 10 Hz.

  Alternatively, a unique trigger signal is assigned to each device so that when the device receives its own dedicated trigger signal, only its visual identifier is activated. The device receives a trigger signal other than its dedicated trigger signal, and its visual identifier remains inactive. This allows the device to be implemented so that the device responds differently to different trigger signals.

  A further possibility is given by a device whose visual identifier is automatically activated at intervals without a trigger signal. This type of implementation is advantageous in cases where the device cannot be modified to include a receiver and an activator. A visual identifier that is continuously active is detected by the system whenever the pointing device is pointed at the device. In this case, the device is assigned a unique visual identifier to distinguish it from other devices.

  Since the user of the pointing device may be frustrated that the visual identifier can be seen for all devices that respond to the trigger signal, the visual identifier is, in the preferred embodiment of the present invention, a human identifier. It is generated in the region of the electromagnetic spectrum that is invisible but can be distinguished by the pointing device camera, for example in the infrared or ultraviolet region.

  The device list management unit of the system stores and manages descriptive information about each device, its associated visual identifier and / or its associated trigger signal. The device list management unit can be regarded as part of the device control interface. A device or group of devices can be given a visual identifier at the time of manufacture or can update or replace such a visual identifier. Similarly, a visual identifier can be assigned to a device that it has not previously had. Thus, in a preferred embodiment of the present invention, the system, for example, sends an appropriate signal to the device and causes the device to activate and / or identify its visual identifier in the designated manner in the future. By responding to the trigger signal, it is sufficient to use an allocation unit that assigns a visual identifier and / or trigger signal to a particular device.

  The system supports a certain type of training mode, where the device list management unit allows which devices are present in the group, which visual identifiers are associated with each device, and which types of triggers Learn if each device responds to the signal. The training process also includes collecting or learning a set of control signals or command protocols that are understood by the respective devices.

  In some cases, if the device, for example, the pointing device is pointed at a given device but has not detected its visual identifier in the image of the target area, the device has already activated its visual identifier. It may be desirable to notify the pointing device of whether or not Thus, in a further preferred embodiment of the invention, the apparatus is provided with means for supplying an end-of-feedback “end-of-feedback” signal to the pointing device indicating that the apparatus has already provided its visual feedback. It is done. For this, the device has a suitable transmitter for transmitting an end-of-feedback signal. In response to receiving this signal, the device control interface decides to cause the pointing device to again send an appropriate trigger signal or to try a different trigger signal.

  For cases where the device is intended to be operated by a given authorized person, for example if the user enables the pointing device with an appropriate access code or personal identification number, the pointing device includes: An authentication code may be provided. The pointing device sends this authentication code along with its trigger signal to the device to which the user points the pointing device. The device activates only its visual identifier if the authentication code is valid. This type of authentication is useful in situations where only certain persons operate the device or interact with the device. One example is a television, such as a prohibited area for children after a certain amount of time, or a security system in a laboratory that is accessible only to certain persons. The authentication information may be hardwired with a pointing device or may be entered by the user by a suitable input format such as a keypad, for example. Another way of defining the pointing device authentication information is by programming with an appropriate interface, similar to known methods of programming remote controls.

  The camera that generates the image of the object is preferably incorporated in the pointing device, but may be mounted on the pointing device and is arranged to generate an image of the area in front of the pointing device targeted by the user. It is preferable. The camera may be built in a basic way or may have powerful functions such as zoom function or certain types of filters.

  Therefore, the “target area” is an area in front of the pointing device that can be captured as an image by the camera. An image of a target area, i.e., a target area image, may cover only a small subset of the devices to which it is directed, or it may encompass the entire device or include an area surrounding the device. The size of the target area image for the entire device depends on the size of the device, the distance between the pointing device and the device to be controlled, and the capabilities of the camera itself. For example, when the user is standing at the other end of the room, the user is positioned such that the pointing device is some distance from the device to be controlled. Similarly, the user holds the pointing device very close to the device to obtain a more detailed image.

  The pointing device according to the present invention is used to interact with the device to cause the device to perform one or more functions desired by the user. For example, the user desires to select one of a number of options displayed on the device display or navigates through a menu hierarchy displayed on the device display. In the case of a device that does not have a display, the user wants to activate the function represented by the button on the device casing, such as the button on the front of the DVD player. Thus, the target area image is analyzed by the image analysis unit to determine the point where the user points the pointing device. This image analysis unit is the same as that used to detect the presence of a visual identifier in the target area image.

  The pointing device may have a control input to allow the user to define one or more predetermined actions. Such control input is a button that the user presses to indicate that an action is to be performed. The operation of the control input is encoded into an appropriate signal and transferred to the device control interface along with the image from the camera, where it is interpreted by the image when generating the device control signal. For example, the user points a pointing device at an area of a particular screen, such as an icon or item in a list of menu items, and simultaneously presses a control input to indicate that the icon or item has been selected. .

  To assist the user in accurately pointing the pointing device, a focused light beam source is provided inside or outside the pointing device so that the following light point appears more or less in the center of the target area captured by the camera. It is done. The source of the focused light beam may be a laser light source, such as a light source used in many types of laser pointers currently available. In the following, the light source of the focused light beam is a laser light source without limiting the scope of the invention.

  In order to easily determine the object the user is pointing at the pointing device, the image analysis unit of the device control interface applies a number of predefined images of the target area by applying normal image processing techniques or computer vision algorithms. Compare to a template. One predefined template is sufficient for comparison or it is necessary to compare the image data to more than one template.

  The predefined template is stored in the internal memory of the device control interface or accessed from an external source. Preferably, the device control interface comprises an access unit with a suitable interface for obtaining a predefined template of an object, for example from an internal or external memory, a memory stick, an intranet or the Internet. In this way, the manufacturer of the equipment controlled by the pointing device according to the present invention makes a template for these hearings available to the user of the device. A template is a graphic representation of any kind of object. If the object is an option for a menu that is displayed on a television, for example, the template shows the location of a number of menu options on the television and analyzes the image data in the target area when the user points the pointing device at the television. In doing so, the image analysis unit can determine which options are directed by the user.

  As described above, the device control interface is implemented so that the interaction with the apparatus is controlled. Such a device control interface includes an image analysis unit for analyzing an image, a control signal generation unit for generating a control signal for the device to be controlled, a device list for managing a relationship between the device, a visual identifier and a trigger signal It includes at least an interface for transmitting control signals to the management unit and the device to be controlled. Such a device control interface is implemented as an external unit that is incorporated into the pointing device or coupled with the pointing device through an appropriate communication interface.

  In the case where the device control interface is incorporated in the pointing device, the image can be acquired directly from the camera. Image analysis, device list management and control signal generation are performed at the pointing device, and the control signals are transmitted from the pointing device directly to the device to be controlled in a suitable format.

  On the other hand, the function of these units is limited by the physical dimensions of the pointing device that are preferably implemented to be held comfortably in the palm, so that such an image analysis unit can be used, for example, by the user The identification of the device to be directed is in principle sufficient for image analysis. The further advanced image processing required to determine the user's intention and generate the resulting control signal that requires a larger unit is done at the external device control interface.

  Thus, the pointing device as a whole may not require the functions of image analysis, device list management, and control signal generation, allowing these tasks to be performed by an external device control interface, thereby It enables the pointing device to be realized in a smaller and more compact form.

  The external device control interface described above is a stand-alone device, where trigger signals are transmitted from the device control interface, from a pointing device, or from another location, while device identification, image processing, device list In some cases, management and control signal generation are performed centrally. A receiver that receives the end-of-feedback signal can be implemented, for example, as part of a pointing device or incorporated into a device control interface.

  The camera of the pointing device is implemented such that the image of the target area has associated image data. For example, when identifying a device, it is necessary to detect the relative brightness of the image, or the shape of the visual identifier in the image. If the visual identifier operates in the invisible part of the electromagnetic spectrum for a human, the camera is provided with a suitable filter, for example to detect infrared or ultraviolet light. The image analysis unit detects a pattern, color or shape corresponding to the description of the visual identifier of the device list management unit. If the visual identifier identifies its associated device, for example by a pattern such as Morse code, amplitude modulation, frequency coding, etc., the image analysis unit identifies this pattern and identifies the device to which the visual identifier is associated. Identify. Another possible way of identifying the device to be controlled is to directly compare the received target area image containing the visual identifier with a predefined template of the device containing the visual identifier, Based on information from the device list management unit, it uses methods such as pattern matching or pattern recognition to infer the identity of the intended device.

  In order to determine what the user actually intends, for example to determine the options he is showing, more detailed image generation is required as described above. Thus, the camera captures important points of the entire image, such as enhanced contours, corners, edges, or more detailed images with image quality. In order to process the image data to determine the user's intention, it is convenient to apply computer vision technology to find the point of the device that the user is pointing at, ie the target point.

  Since the image of the target area contains options other than the actual option the user is pointing at the pointing device, the selected option can be identified by identifying an option in the image that includes or encompasses a specific target point in the target area. Preferably, it is determined. In one embodiment of the invention, the fixed point in the target area image is preferably the center of the target area image, optionally with an imaginary line in the direction of the long axis of the pointing device. It is obtained by extending and used as a target point.

  A method of processing an image of a target area of a device using a computer vision algorithm detects an individual point in the target image, determines a corresponding point in the template of the device, and a corresponding point in the template in the target image Includes expanding transformations that map points. The individual points in the target area are either unique points of the device or points in the area around the device. This transformation is used to determine the position and appearance of the pointing device with respect to the device, and the intersection of the axes of the pointing device and the device to be controlled is located in the template. The location of the intersection in this template corresponds to the target point of the device to be controlled and is used to easily determine which option is targeted by the user. Thus, comparing the image of the target area with a predefined template is limited to identifying and comparing only significant points, such as unique corner points. The term “comparison”, as applied in the present invention, should be understood in a broad sense, that is, by comparing enough features to quickly identify the option that the user is targeting.

  The invention all provides an easy and flexible way to interact with any type of device that can be controlled electrically or electronically in any environment. For ease of use, the pointing device is in the shape of a wand or pen in an elongated shape that can be comfortably gripped and easily carried by the user. The user can point the pointing device at the device to be controlled while being located at a distance from the device. Similarly, the pointing device is molded in the form of a pistol. Furthermore, a further light source is mounted inside or outside the pointing device and serves to illuminate the area to which the pointing device is directed, allowing the user to easily position the device to be controlled even in the dark be able to.

  The pointing device and device control interface described above are combined to provide a powerful control system for use in practically any kind of environment. For example, the system could find use in an environment equipped with a device that is interacted with a pointing device, such as an office, museum, hospital or hotel environment, in which case the user can determine the capabilities of the device and he Pointing devices can be used to identify and control devices that are not familiar in a convenient and intuitive manner without having to become familiar with them. The method according to the invention can be applied to devices that can be controlled electrically or electronically. Furthermore, the device to be controlled has any number of modules, components or units and is distributed in any way.

  Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It should be understood that the drawings are for purposes of illustration and are not designed as a definition of the limitations of the invention. In the drawings, like reference numerals refer to like objects throughout. The pointing device described is held and operated by a user, not shown.

FIG. 1 shows the concept of a system having a number of devices D ₁ , D ₂ , D ₃ and a pointing device 1 that identifies and interacts with these devices D ₁ , D ₂ , D _{3 according} to an embodiment of the invention. FIG.

In the example shown, the devices D ₁ , D ₂ , D _{3 with} which the pointing device 1 interacts have similar shapes and appearances and are difficult to explain visually apart. In the dialog with these apparatuses D ₁ , D ₂ , and D _3, it is assumed that the user points the pointing device 1 at a desired apparatus, and the user is pointing toward the apparatus D ₁ for explanation. The pointing device is provided with a transmitter 6 that transmits trigger signals T ₁ , T ₂ , T to any _one of the devices D ₁ , D ₂ , D ₃ in the range of the pointing device 1. The pointing device transmits each of these trigger signals T ₁ , T ₂ , T.

The control signal 19 defines which trigger signal is to be transmitted and is generated from the device control interface 18, which is shown in the figure as a separate block, but is integrated as a whole or pointing device 1 may be partially indicated by itself. The device list management unit 15 of the device control interface 18 is a list of devices D ₁ , D ₂ , D _{3 with which} the pointing device 1 interacts, and a visual identifier V associated with each device D ₁ , D ₂ , D _{3. 1} , V ₂ , V ₃ and / or information on trigger signals T ₁ , T ₂ , T. The components and functions of the device control interface 18 are described in further detail below.

In this example, trigger signal T ₁ is assigned to device D ₁ and trigger signal T ₂ is assigned to device D ₂ . The further trigger signal T is used to trigger one or more other devices.

The device D ₁ is provided with a receiver 40, which is tuned to receive its trigger signal T ₁ . Similarly, the device D ₂ is provided with a receiver 41, which is tuned to receive its trigger signal T ₂ . Receiver device D ₃ is configured to receive the trigger signal T in the range of that bandwidth.

When device D ₁ receives trigger signal T ₁ , activator 30 responds to trigger signal T ₁ by activating device D ₁ visual identifier V ₁ . In this example, the visual identifier V ₁ is a light source such as a blinking or flashing LED.

Similarly, visual identifiers V ₂ of the device D _2, the receiver 41 of the device D ₂ is whenever receiving the trigger signal T _2, it is activated by the activator 31. This visual identifier V ₂ is in this example a physical element attached to the device D ₂ , for example an infrared identifier. Furthermore, the apparatus D ₂ is provided with a transmitter 35 that transmits an end-of-feedback signal S _EOF to the pointing device 1.

The third device D ₃ is configured to respond to the trigger signal T. This device is provided with a suitable receiver 42 that activates the visual identifier V ₃ of this device D ₃ to the activator 32. In this example, device D ₃ is a computer monitor that displays user interface elements such as desktops and windows, and visual identifier V ₃ is responsive to trigger signal T for its brightness, intensity, color, Or a normal type of Windows frame or menu item that changes hue.

While the pointing device 1 is pointed at the devices D ₁ , D ₂ , D ₃ , an image of the area in front of the pointing device 1 is continuously generated and transmitted by the sending unit 4 to the device control interface 18, Received by the receiver 10. Since you are apparatuses D ₁ which is directed, D _2, to identify the D _3, to determine the intent of the user, which shows a specific part of the apparatuses D _1, D _2, D _3, generated by the camera 2 of the pointing device 1 The processed image is analyzed by the image analysis unit 12.

Control signal 17, 27 is generated, before being sent to the apparatuses D _1, D _2, D _3, system 5, apparatuses D ₁ being directed, D _2, identifies the beginning of the D _3. Therefore, the image analysis unit 12 extracts information about the visual identifiers V ₁ , V ₂ , V ₃ from the image and compares this information with that stored in the device list management list 15. For example, if the visual identifier located in the image repeatedly increases and decreases in intensity within a predetermined time, the visual identifier's appearance will be similar to its behavioral pattern, the visual identifier of device D ₁ Consistent with what is stored in the device list management unit 15 for V ₁ , the image analysis unit 12 infers that the device being directed must be device D ₁ .

  The steps of the decision making process required to identify a device from a number of devices are shown in FIG. Here, the process begins at step 100, which occurs, for example, after switching on the pointing device or after configuring the system again. In step 101, a list of all devices is compiled. This list includes an identifier for each device, such as an image template, a description of the visual identifier, and / or a trigger signal associated with this device. In a subsequent step 102, this list is sorted according to the probability of decrease directed by the pointing device. This step 102 ensures that no time is spent in the process of identifying the device by starting with the device most frequently directed by the user and ending with the least selected device. The system updates this list whenever an appropriate learning algorithm is sufficient and is needed.

  The first entry in the list is examined at step 103 when the user points the pointing device at the device to be controlled. The pointing device sends an appropriate trigger signal at step 104 and the system waits for a response. In step 105, the image analysis unit of the system analyzes the target area image to determine whether the target area image indicates a visual identifier. If a match with the visual identifier of the first device list entry is detected at step 106, the device has been successfully identified and the process ends at step 107.

  However, if a matching visual identifier is not detected at step 106, then at step 111, the process checks to see if a predefined time or timeout has elapsed. If not, it is checked at step 110 whether an end-of-feedback signal has been detected. If not, the process restarts at step 105 as described above. If an end-of-feedback signal is detected at step 110, or if a timeout is reached at step 111, the process proceeds to step 109 where it is checked whether the end of the list has been reached. If not, step 108 selects the next entry in the list and step 104 restarts the process. If the end of the list is reached, the process restarts at step 103 and returns to the start of the list.

  In step 101, updating the list of devices and their associated visual identifiers and / or trigger signals is required at several points. For example, a different device replaces the original device at the top of the list, a new device is introduced, a device is removed from the group, or the rate of probable use of individual devices changes over time May have. When such a condition occurs, the process routine returns from step 109 to step 101 and restarts from there. Appropriate monitoring arrangements detect such situations and react to the process routine accordingly.

Referring to FIG. 1, the step of cycling through trigger signals T ₁ , T ₂ , T is managed by device list management unit 15 based on the result of image analysis supplied by image analysis unit 12. An appropriate signal 19 is sent to the pointing device 1 to send the next trigger signals T ₁ , T ₂ , T as required.

Simultaneously with the identification process described above, in progress or subsequent to the identification process, the image 3 generated by the camera 2 of the pointing device 1 is interpreted to carry out the user's intention. For this reason, the image analysis unit 12 uses a known image processing technique such as pattern recognition in order to identify the template that best matches the image 3 from a large number of templates, and infers the option that the user is pointing to. For example, users can use the pointing device 1, or crossfeed a menu displayed on the screen of the device D _1, D _2, D _3, perform specific tasks device D _1, D _2, D ₃ The task is started using the pointing device 1.

For this reason, the control signal generation unit 13 uses the result of the image analysis in order to generate the control signal 17 of the device. The application interface 14 performs the necessary conversion to the control signal 17 before sending it in the form 27 appropriate for the device D ₁ , D ₂ , D ₃ to which it is directed.

In addition, the allocation unit 16 is shown here as part of the device control interface 18 and directs specific visual identifier behavior and / or trigger signal descriptions to specific devices D ₁ , D ₂ , D ₃ . Used to assign. For example, the allocation unit 16 specifies that the visual identifier V ₁ of the device D ₁ alternates colors between red and blue for a period of 5 seconds and trigger signal T ₁ transmitted at a predetermined frequency. This should be done in response to This information is encoded as required by interface 14 before being sent as signal 27 to the device in question.

If the device control interface 18 is not incorporated in the pointing device as shown in FIG. 1, the information transmitted from the pointing device 1 to the device control interface 18 is a wireless standard such as Bluetooth, 802.11b or a mobile telephone standard. Sent in the same manner. If the user carries his pointing device 1, the pointing device 1 may be connected to the device control interface 18 by a cable. The signal 27 transmitted from the device control interface 18 to the devices D ₁ , D ₂ , and D ₃ may be transmitted through a cable interface or may be transmitted in a wireless manner as necessary.

  The camera 2 continuously transmits the image 3 to the device control interface 18 or automatically stops transmission when the pointing device 1 is not moved for a predetermined time. For this reason, the pointing device 1 has a motion sensor (not shown). Although the pointing device 1 is not shown but is most likely to be powered by a battery, device control when needed, for example, when a user actually moves the pointing device 1 to extend the life of the battery It is convenient to send image 3 to the interface. The transmission of the image data 3 is started as soon as the user moves the pointing device, and then automatically stopped.

The user does not always point the pointing device directly at the device to be controlled directly from the front, but the pointing device 1 is likely to be directed at a more or less oblique angle of the device. This is because it is more convenient to point to the pointing device 1 than to change its own position. This is illustrated in Figure 3, this figure, the target area from a certain distance, at a certain inclination angle, in this case generated by the pointing device 1 is directed to apparatuses D ₁ is a television screen or computer monitor A conceptual representation of image 3 is shown, the scale and appearance of device D ₁ in target area A appear distorted in target area image 3. A number of options (M ₁ , M ₂ , M ₃ ) can be seen on the display of device D ₁ . A user (not shown) wishes to select one of these options (M ₁ , M ₂ , M ₃ ) with the pointing device 1. Further, the visible in the display of the apparatuses D ₁ is in this case a section of the display having a characteristic shape to increase or decrease in brightness, this apparatuses D ₁ is a visual identifier V _1. The pointing device 1 in this figure is provided with a laser light source 51, which collides the beam of the laser light _L against the display at a point P _L and assists the user in pointing the pointing device 1.

Once the pointing device 1 transmits its trigger signal, the image generated by the camera is analyzed to detect a visual identifier. In this example, a characteristic flash or blinking of the section of the display giving a visual identifier V ₁ is detected and the device is identified as being device D ₁ . Simple analysis of the brightness level of the image, and subsequently compared with the information supplied by the pattern and the device list management unit that occurs is sufficient to identify the device D _1.

Once the device is identified at D ₁ , the system continues to interpret user actions to control device D ₁ . For this reason, the target area image 3 is examined in more detail. Regardless of the angle of the pointing device 1 to a device D _1, the target area image 3 is always centered around a target point P _T. Further, the point P _L of the laser light, appear in the target area image 3, or a distance that is moved from the target point P _T, there is a case that matches the target point P _T. The image processing unit of the device control interface compares the target area image 3 with a predefined template 50 and determines the options (M ₁ , M ₂ , M ₃ ) directed by the user.

For this reason, the intersection point P _T of the long axis of the pointing device ₁ with the apparatus D ₁ is located in the target area image 3. The point in the template 50 that corresponds to the intersection point P _T is then placed.

The computer vision algorithm using the edge and corner detection method is the target area corresponding to the points [(x _a , y _a ), (x _b , y _b ), (x _c , y _c )] in the template 50 of the device D1. This is applied to place the points [(x _a ', y _a '), (x _b ', y _b '), (x _c ', y _c ')] on the image 3.

として表現される。次のステップとして、変換関数Ｔ_λが展開され、ターゲットエリア画像３をテンプレート５０にマッピングする。 Each point can be represented as a vector, for example, the point (x _a , y _a ) is
(Outside 1)

Is expressed as As the next step, the conversion function T _λ is developed and the target area image 3 is mapped to the template 50.

ここでベクトル
（外２）

はテンプレート５０における座標ペア（ｘ_i,ｙ_i）を表し、ベクトル
（外３）

はターゲットエリア画像３における対応する座標ペア（ｘ_i’,ｙ_i’）を表す。パラメータセットλは、画像の回転及び変換のパラメータを含み、最も費用対効果の高いソリューションを関数に与えるものであり、装置Ｄ₁に関してポインティングデバイス１の位置及び方向を決定するために適用される。コンピュータビジョンアルゴリズムは、ポインティングデバイス１内のカメラ２が固定され、ポインティングジェスチャの方向で「見ている」事実を利用する。次のステップは、装置Ｄ₁との、ポインティングの方向Ｐにおける、ポインティングデバイス１の長軸の交点Ｐ_Tを計算することである。このポイントは、ターゲットエリア画像３の中央となるように取られる場合がある。ひとたび交点の座標が計算されると、このポイントをテンプレート５０に配置することはシンプルな事項である。このように、システムは、ユーザが向けているオプション（Ｍ₁，Ｍ₂，Ｍ₃）を判定することができ、装置Ｄ₁について適切な制御信号を発生することができる。

Here vector (outside 2)

Represents a coordinate pair (x _i , y _i ) in the template 50 and is a vector (outside 3)

Represents a corresponding coordinate pair (x _i ′, y _i ′) in the target area image 3. The parameter set λ contains the image rotation and transformation parameters and gives the function the most cost-effective solution and is applied to determine the position and orientation of the pointing device 1 with respect to the device D ₁ . The computer vision algorithm takes advantage of the fact that the camera 2 in the pointing device 1 is fixed and “looking” in the direction of the pointing gesture. The next step is to calculate the intersection point P _T of the long axis of the pointing device 1 in the pointing direction P with the device D ₁ . This point may be taken to be the center of the target area image 3. Once the coordinates of the intersection are calculated, placing this point on the template 50 is a simple matter. In this way, the system can determine the options (M ₁ , M ₂ , M ₃ ) that the user is pointing at and can generate appropriate control signals for the device D ₁ .

  While the invention has been disclosed in the form of preferred embodiments and variations thereof, it is to be understood that various further changes and modifications can be made without departing from the scope of the invention. The pointing device functions as a universal user interface device in a home or other environment with an electrically or electronically controllable device. In short, there is a profit everywhere when the user expresses his intention by pointing. Its small form factor and its convenient and intuitive pointing style pulls such a simple pointing device into a powerful universal remote control. As an alternative to a pen shape, the pointing device is, for example, a personal digital assistant (PDA) with a built-in camera, a mobile phone with a built-in camera.

  For clarity, it should be understood that “comprising” does not exclude other steps or elements. A “unit” has a number of blocks or devices when not explicitly described as a single entity.

1 is a conceptual diagram of a system for controlling an apparatus, showing a pointing device and a large number of apparatuses according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating steps involved in identifying one device from a group of devices according to an embodiment of the invention. 1 is a conceptual diagram of a pointing device according to an embodiment of the present invention.

Claims (12)

An apparatus control method comprising:
The method is
Directing a pointing device having a camera to an apparatus to be controlled;
Generating an image of a target area directed by the pointing device;
Interpreting the image of the target area to determine the presence of a unique visual identifier associated with the device to be controlled;
Identifying a device to which the pointing device is directed based on an analysis of an image of the target area and sending a control signal to the identified device;
Including methods. The image of the target area is further interpreted to determine a selected option, and the control signal is generated according to the result of image interpretation.
The method of claim 1. A trigger signal is transmitted to the device to be controlled and the visual identifier is activated by the device receiving the trigger signal;
The method according to claim 1 or 2. Each unique trigger signal is generated for a particular device,
The method according to claim 1. The visual identifier is an invisible region of the electromagnetic spectrum;
The method according to claim 1. The device to be controlled emits an end-of-feedback signal in response to the trigger signal;
The method according to claim 1. A system for controlling a device,
A pointing device that has a camera and generates an image of the target area in a direction in which the pointing device is directed;
An image analysis unit that analyzes the image to determine the presence of a visual identifier;
A device list management unit that maintains a list of possible devices, their associated visual identifiers, and / or associated trigger signals for activating the visual identifiers;
A control signal generating unit for generating a control signal for a device to be controlled according to the result of the image analysis;
An interface for transmitting said control signal to a device to be controlled;
Having a system. Having an allocation unit for assigning visual identifiers and / or trigger signals to specific devices;
The system of claim 7. Having a unique visual identifier that uniquely identifies the device among a number of devices;
Apparatus for use by the method according to any of claims 1-6.   A controllable device having a receiver for receiving a trigger signal and an activator for activating a visual identifier in response to the trigger signal. Having a transmitter for transmitting an end-of-feedback signal in response to the trigger signal;
The apparatus of claim 10. The visual identifier comprises an optical signal and / or a physical identifier;
The apparatus according to claim 9.

Images