KR20140060615A - Remote control apparatus using virtual plane of electronic devices without pointer - Google Patents

Abstract

The present invention provides a remote control device for an electronic device using a virtual plane through user gestures at a remote place. The remote control device is applicable to all kinds of electronic devices regardless of the existence of a display device. The remote control device includes: a three-dimensional coordinate calculation unit which extracts three-dimensional coordinate data of the body of a user using images acquired through image recording; a contact point detection unit which recognizes spatial position data and product data of an electronic device using the three-dimensional coordinate data of the electronic device extracted by the three-dimensional coordinate calculation unit and generates a virtual contact point area for a virtual touch between the user and the electronic device based on the direction data corresponding to a straight line between the user and the electronic device; a command code generation unit which calculates the position and position change of the virtual touch to operate the electronic device using the three-dimensional coordinate data of the body of the user calculated by the three-dimensional coordinate calculation unit and generates a command code to execute a preset action of the electronic device based on the calculated position change of the virtual touch; and an electronic device operation control unit which, when the command code generated by the command code generation unit is received, converts the command code into a control command signal for the corresponding action of the electronic device and transmits an action command signal to the corresponding electronic device.

Description

[0001] The present invention relates to a remote control apparatus using a virtual plane of an electronic device in a virtual touch device that does not use a pointer,

The present invention relates to an apparatus and a method for remotely operating an electronic apparatus by a user, and more particularly to a system and method for remotely controlling an electronic apparatus by operating a remote control means such as a remote control And more particularly, to a remote control apparatus and method capable of performing remote control.

Recently, electronic devices having a touch panel such as a smart phone are widely used. This touch panel technology does not need to display a 'pointer' on the display when compared to an electronic device such as a computer that can be operated with an existing mouse. The user moves the finger over the icon and performs the operation by touching it without moving the pointer (the cursor of the computer) to the corresponding position (e.g., the program icon) in order to operate the electronic device. In this touch panel technology, it is unnecessary to move the 'pointer' necessary for the conventional electronic device control means, thereby enabling quick operation of the electronic device.

However, the touch panel technology has a disadvantage in that it can not be used remotely because the user has to directly touch the display surface in spite of the above convenience. Therefore, even an electronic device employing a touch panel technology can not but rely on a device such as a conventional remote control for remote operation.

Recently, as a remote electronic device control device, a device capable of generating a pointer at a precise point, such as a touch panel technology, that is, an image of a front surface of a display using two cameras, (Patent Document 1) discloses a technology for generating a pointer at a point where a pointer is encountered. However, as a preliminary measure for controlling an electronic device, a 'pointer' must be generated (with a pointer control unit) The gesture of the user is compared with the pattern stored in the past for the specific operation control. [Patent Document 1] Published Patent 2010-0129629 Dec. 9, 2010

However, in the case of home appliances such as home electric appliances, electric fans, air conditioners, refrigerators, etc., the shape of the external appearance is various and there is no display, and since the size of the display is small, It may be difficult to find a contact point where the straight line of the eye and the finger of the user meets the household appliance.

In addition, when the size of household electrical appliances such as household appliances, electric fans, and the like is small, the operating area becomes very small when a user operates such a small household appliance at a remote place.

Accordingly, the present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide an electronic device, There is provided a remote control device capable of easily controlling a device remotely.

According to an aspect of the present invention, there is provided a remote control apparatus for extracting three-dimensional coordinate data of a user's body using a video image, A contact detection unit for calculating a contact point between the virtual plane of the electronic device and the straight line based on direction information corresponding to a straight line between the user and the electronic device; A command code generating unit for generating a command code for performing an operation of a predetermined electronic device when the command code generated by the command code generating unit is input; And an electronic device drive control unit for transmitting an operation command signal to the electronic device.

According to another aspect of the present invention, the three-dimensional coordinate calculation unit may include an image acquisition unit configured by two or more image sensors disposed at different positions and capturing a user's body and an electronic device at different angles, Dimensional coordinate data of the user's body using an optical triangulation method based on images taken at different angles received from the user, and extracting three-dimensional coordinate data of the preset electronic device, .

In still another embodiment of the present invention, the three-dimensional coordinate calculation unit is composed of a light source and a diffuser, and includes an illumination assembly for projecting a speckle pattern onto a user's body and an electronic device, an image sensor and a lens, Dimensional coordinate data of the user's body on the basis of the speckle pattern captured by the image obtaining unit, and outputs the three-dimensional coordinate data of the user's body to the predetermined electronic device Dimensional coordinate data of the three-dimensional coordinate system.

The contact point detecting unit may calculate the first spatial coordinate and the second spatial coordinate based on the three-dimensional spatial coordinates of the user's body calculated by the three-dimensional coordinate calculating unit, A touch position calculation unit for calculating contact point coordinate data in which a straight line connecting the space coordinate and the second spatial coordinate meets the contact point area of the virtual plane; and a touch position calculation unit for determining whether the contact coordinate data calculated by the touch position calculation unit changes, And a virtual touch processing unit for generating a command code for performing an operation of the preset electronic device when it is determined that there is a change.

In the embodiment of the present invention, the first spatial coordinate is any three-dimensional coordinate among the end of one finger of the user's finger and the end of the pointer held by the user with the finger, Dimensional coordinates of the center point of either one of the eyes.

In another embodiment of the present invention, the virtual touch processing unit determines whether or not there is a change in the contact point coordinate data over a set time from the time at which the initial contact point coordinate data is calculated, and if there is a change in the contact point coordinate data It is determined that there is a change in the contact point coordinate data.

In another embodiment of the present invention, when the contact coordinate data does not change more than the set time from the time at which the initial contact coordinate data is calculated, the virtual touch processing unit sets the distance between the first spatial coordinate and the second spatial coordinate And when it is determined that there is a change in distance over a set distance, it is determined that there is a change, and a command code for performing an operation of the previously set electronic device is generated.

The remote control apparatus and method using the virtual touch of the electronic device according to the present invention as described above have the following effects.

First, by operating the electronic device by setting the virtual plane on the electronic device, it is possible to quickly operate the electronic device. Therefore, the present invention can be applied to all electronic devices not including a display device.

Secondly, it is difficult to operate relatively small household appliances because it is small in size when operated at a remote location, but it is possible to expand the operation area by applying the concept of virtual plane, which is advantageous in operation of small household appliances.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing nine operation regions as a whole by dividing a virtual plane around a fan according to an embodiment of the present invention into eight equal parts. FIG.
FIG. 2 is a view showing nine operation regions as a whole by dividing the virtual plane around the lighting stand according to the embodiment of the present invention into eight equal parts. FIG.

Other objects, features and advantages of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a remote control apparatus using a virtual plane of an electronic device according to the present invention will be described with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. It is provided to let you know. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

An overall system of remote operation using a virtual plane of an electronic device according to an embodiment of the present invention will be described.

First, when the user requests remote control operation of the electronic device 200 using a virtual touch through a movement of a hand or the like (gesture), the remote control apparatus 100 displays three-dimensional coordinate data of the user's body Dimensional region, and extracts position information of the predetermined electronic device 200 at the same time. Here, the three-dimensional coordinate information on the position and the shape of the electronic device 200 varies according to manufacturers and models. The electronic device three-dimensional information is stored in the system of the present invention by the user after purchasing the corresponding electronic device. In this regard, in the present invention, a separate electronic device information database is provided, and three-dimensional coordinate data of positions and shapes of various electronic devices can be stored and updated. As a specific update method, it is possible to access the website of the household appliance and download information of the electronic appliance, or to download and update the information of the electronic appliance through a transceiver previously configured in the electronic appliance. Various embodiments are possible within the scope of the technical idea of the present invention.

Then, a driving operation command signal of the electronic device corresponding to the movement of the user is generated and transmitted to the electronic device 200, thereby operating the electronic device 200.

Thus, as the remote operation is performed using the position information of the electronic device, the present invention can be applied to all electronic devices driven by using the existing remote control, regardless of the presence or absence of the display device.

Hereinafter, the configuration of the remote control apparatus 100 will be described in more detail.

1st Example

The remote control apparatus using the virtual touch of the electronic device according to the first embodiment of the present invention includes an image acquisition unit 110, a space coordinate calculation unit 120, a contact detection unit 130, an instruction code generation unit 140, An electronic device drive control unit 160, and an electronic device database unit (DB)

The image obtaining unit 110 may include two or more image sensors 112 and 114 such as CCD or CMOS for detecting an image and converting the image into an electrical image signal.

The space coordinate calculator 120 calculates three-dimensional coordinate data of the user's body using the image received from the image obtaining unit 110 and extracts three-dimensional coordinate data of the predetermined electronic apparatus. According to the present invention, the image sensors 112 and 114 constituting the image acquiring unit 110 photograph the user's body at different angles, and the spatial coordinate calculator 120 calculates the three-dimensional coordinates of the user's body using passive optical trigonometry The coordinate data can be calculated.

Generally, the optical three-dimensional coordinate calculation method can be classified into the active method and the manual method according to the sensing method. The active method is a method of calculating three-dimensional coordinate data of an object by projecting a predetermined pattern or sound wave onto an object and then measuring the amount of change through control of sensor parameters such as energy or focus, Method. On the other hand, the manual method uses intensity, parallax, etc. of the image taken without artificially projecting energy to the object.

In the present invention, a passive method that does not project energy to an object is adopted. This method is advantageous in that it is simple in equipment and can acquire a texture directly from an input image even though accuracy may be somewhat lower than that in an active method.

In the manual method, three-dimensional information can be acquired by applying optical triangulation to feature points corresponding to the captured images. Camera self-calibration, Harris's corner extraction, SIFT, RANSAC, and Tsai techniques are commonly used as various related techniques for extracting three-dimensional coordinates by applying trigonometry. In particular, a stereoscopic camera method may be used as a method of calculating three-dimensional coordinate data. The stereoscopic camera method is a method of observing the same point on the surface of an object at two different points in the same way as the structure of a binocular stereoscopic case in which a human is seen with two eyes and obtains a displacement, . The above-mentioned various three-dimensional coordinate calculation techniques can be easily understood by those skilled in the art and can be implemented. Therefore, a description thereof will be omitted. On the other hand, as patent documents related to a method of calculating three-dimensional coordinate data using a two-dimensional image, Korean Patent Laid-Open Nos. 10-0021803, 10-2004-0004135, 10-2007-0066382, 10-2007 -0117877 and so on.

The contact point detecting unit 130 includes an electronic device setting unit 134 and a touch position matching unit 136. The contact point detecting unit 130 detects the coordinates of the electronic device Recognizes the spatial position information and the device product information of the user 200 and creates a virtual contact area for virtual touch between the user and the electronic device based on the direction information corresponding to the straight line between the user and the electronic device.

The electronic device setting unit 134 compares the three-dimensional coordinate information of the electronic device previously stored in the electronic device information DB 150 and sets a control signal according to the driving operation of the electronic device. The touch location matching unit 136 matches the direction information corresponding to a straight line between the user and the electronic device to generate a virtual virtual plane for the virtual touch between the user and the electronic device based on the matching direction information do. Here, the virtual plane of the electronic device is a two-dimensional plane having only x and y values, and does not require a depth value. It is preferable that the virtual plane of such an electronic device coincides with the actual surface position of the electronic device. However, since the shapes of electronic devices vary widely, virtual planes can be set to be wider than the corresponding electronic devices so as to extend not only the body of the electronic device but also the periphery thereof to the operation area. Here, the virtual plane can be used as an operation area by dividing the electronic device body into various areas such as 2, 4, 6, and 8, and the outer area outside the electronic device can be expanded to the operation area.

The instruction code generating unit 140 includes a touch position calculating unit 142 and a virtual touch processing unit 144. The instruction code generating unit 140 generates an instruction code based on the three-dimensional coordinate data of the user's body calculated by the space coordinate calculating unit 120, A virtual touch position and its positional change for a driving operation of the device 200 are calculated and a command code for performing an operation of a predetermined electronic device based on the calculated virtual touch positional change is generated.

At this time, the touch position calculation unit 142 calculates the first spatial coordinate and the second spatial coordinate based on the three-dimensional spatial coordinate of the user's body calculated by the spatial coordinate calculation unit 120, A straight line connecting the first spatial coordinate and the second spatial coordinate using the coordinate (finger) and the second spatial coordinate (eye) is used as the contact coordinate data for meeting the virtual contact region generated in the touch position matching unit 136 .

Generally speaking, the fingers are the only part of the human body that allows precise and delicate manipulation. Particularly, when using either the thumb or the index finger among the fingers or using the two fingers together, it is possible to perform precise pointing. Accordingly, it is very effective to use the tip of the thumb and / or index finger as the first spatial coordinate in the present invention. Also, in the same context, it is possible to use a pointed pointer (e.g., a pen tip) holding a finger with the finger instead of the end of the finger serving as the first spatial coordinate.

In the present invention, only the center point of one eye of the user is used as the second spatial coordinate. For example, if the user places his / her index finger in front of his / her eyes and looks at the index finger, the index finger will appear as two. This occurs because the shapes of the index fingers viewed by the user's eyes are different from each other (due to the angular difference of the eyes). But if you look at your fingers only with one eye, your forefinger will look clearly. Even if you do not detect one eye, you can consciously see your forefinger clearly even if you look only with one eye. It is the same principle to close and close one eye for sports events that require a high degree of accuracy in aiming, such as shooting and archery.

The present invention adopts the principle that the shape of the fingertip can be clearly grasped when the fingertip (first spatial coordinate) is viewed only by one eye (second spatial coordinate). In this way, the user can accurately see the first spatial coordinate so that the virtual contact area generated by the touch position matching unit 136 matching the first spatial coordinate can be pointed.

Meanwhile, in the present invention, when one user uses one of the fingers, the first spatial coordinate is defined as any one of three-dimensional coordinates of the end of one of the fingers of the user and the end of the pointer held by the user And the second spatial coordinate will be the three-dimensional coordinates of the center point of either eye of the user. If one user uses two or more of the fingers, the first spatial coordinates are three-dimensional coordinates of the ends of two or more fingers of the user's fingers, and the second spatial coordinates are the coordinates of the two or more users Dimensional coordinates of the center point of either eye. In addition, when two or more users are used, the first spatial coordinates are three-dimensional coordinates of the ends of one or more fingers presented by the user of two or more users, and the second spatial coordinates are coordinates of one of the two or more users Dimensional coordinates of the center point of the center point.

The virtual touch processing unit 144 determines whether or not there is a change in the contact point coordinate data for a set time or more from the calculated time of the initial contact point coordinate data and sets the contact point coordinate data to correspond to the contact point coordinate data And generates and outputs a command code for performing an operation. Also, the virtual touch processing unit 144 can process a single user, two fingers, or two or more users in the same manner.

Further, the virtual touch processing unit 144 determines whether there is a change in the contact coordinate data for a predetermined time or more from the time when the initial contact coordinate data is calculated. If there is no change in the contact coordinate data for a set time or more, And generates a command code for performing an operation set to correspond to the contact coordinate data when a distance change of a distance equal to or greater than the set distance is detected. The virtual touch processing unit 40 can also process two users of one user or two or more users in the same manner.

On the other hand, if it is determined that the variation of the contact point coordinate data is within the set range, the contact point coordinate data can be regarded as being not changed. That is, it is very difficult for the user to keep the contact coordinates as they are, when the user points to the electronic device at the end of the finger or the pointer, because of the physical characteristics of the user or the finger. Therefore, when the contact coordinate data value is within the predefined setting range, it is regarded that there is no variation in the contact coordinate data.

The electronic devices to be subjected to remote control according to the present invention include home networking such as lighting, a speaker, an air conditioner, an electric fan, a blind, a window, a curtain, a door, a heating, a gas, a microwave, Devices may also be subject to remote control in accordance with the present invention.

Since the remote control device 100 according to the present invention is separately provided from the electronic device to be remotely controlled, it is possible to freely calculate the changed three-dimensional spatial coordinates of the electronic device even when the electronic device is moved, (Through continuous updating of the three-dimensional information of the electronic device).

When the command code generated by the command code generating unit 140 is inputted, the electronic device driving control unit 160 converts the command code into a control command signal for a driving operation of the corresponding electronic device, . The operation command signal transmitted at this time is preferably transmitted by an infrared (IR) signal of the existing remote controller. However, the present invention is not limited to this, and the present invention is also applicable to a short-range wireless communication method such as WiFi, Bluetooth, Of course.

The electronic device database unit (DB) 150 stores device product information such as a maker and a model name, protocol information applied to drive operation for each device, Operation method information can be stored. At this time, the operating method information refers to information such as the movement of the user's finger and the touch position for operating the power, illuminance, volume, and temperature control of the electronic device with the virtual touch. For reference, the information stored in the electronic device information DB 150 can be updated at any time by the user after purchase of the electronic device. As a method of updating the information, the information of the electronic device is downloaded Or download and update the information of the electronic device through a transceiver previously configured in the electronic device. Various embodiments are possible within the scope of the technical idea of the present invention.

Second Example

The remote control apparatus using the virtual plane of the electronic device according to the second embodiment of the present invention includes a three-dimensional coordinate calculation unit 170, a contact detection unit 130, an instruction code generation unit 140, 160, and an electronic device database unit (DB) 150.

The three-dimensional coordinate calculation device 170 includes an illumination assembly 172, an image acquisition unit 174 and a spatial coordinate calculation unit 176. The three-dimensional coordinate calculation apparatus 170 projects a pattern image coded by the user, And processes the image of the scene to calculate three-dimensional coordinate data of the user's body and the electronic device. The three-dimensional coordinate calculation method is variously existing and can be easily implemented by a person skilled in the art to which the present invention belongs. Therefore, a description thereof will be omitted.

At this time, the illumination assembly 172 includes a light source 172a and a diffuser 172b and projects a speckle pattern onto the user's body and the electronic device. The image acquiring unit 174 includes an image sensor 174a and a lens 174b to capture a speckle pattern of the user's body projected by the illumination assembly 172. [ The image sensor 174a may be a CCD or a CMOS image sensor. The spatial coordinate calculator 176 processes the acquired image by the image acquiring unit 174 and calculates three-dimensional coordinate data of the user's body.

The contact detecting unit 130 includes an electronic device setting unit 134 and a touch position matching unit 136. The contact detecting unit 130 detects the electronic coordinate of the electronic device And generates a virtual contact area for virtual touch between the user and the electronic device based on direction information corresponding to a straight line between the user and the electronic device. For reference, the description of the configuration of the contact detecting unit 130 is the same as that described in the first embodiment, and is described in detail above, and a description thereof will be omitted.

 The instruction code generating unit 140 includes a touch position calculating unit 142 and a virtual touch processing unit 144. The instruction code generating unit 140 generates an instruction code based on the three-dimensional coordinate data of the user's body extracted by the space coordinate calculating unit 176, A virtual touch position and a position change for a driving operation of the device 200 are calculated and a command code for performing an operation of a predetermined electronic device based on the calculated virtual touch position change is generated. The description of the configuration of the instruction code generator 140 is the same as that described in the first embodiment, and the detailed description is omitted.

When the command code generated by the command code generating unit 140 is inputted, the electronic device driving control unit 160 converts the command code into a control command signal for a driving operation of the corresponding electronic device, . The operation command signal transmitted at this time is preferably transmitted by an infrared (IR) signal of the existing remote controller. However, the present invention is not limited to this, and the present invention is also applicable to a short-range wireless communication method such as WiFi, Bluetooth, Of course.

The electronic device database unit (DB) 150 may store device product information such as manufacturer and model name, and protocol information applied to drive operation of each device, including three-dimensional coordinate information of electronic devices provided in the home . At this time, the operating method information refers to information such as the movement of the user's finger and the touch position for operating the power, illuminance, volume, and temperature control of the electronic device with the virtual touch. For reference, the information stored in the electronic device information DB 150 can be updated at any time by the user after purchase of the electronic device. As a method of updating the information, the information of the electronic device is downloaded Or download and update the information of the electronic device through a transceiver previously configured in the electronic device. Various embodiments are possible within the scope of the technical idea of the present invention.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It will be apparent to those skilled in the art that various modifications may be made without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (7)

A three-dimensional coordinate calculation unit for extracting three-dimensional coordinate data of a user's body using the image received through image capturing and extracting three-dimensional coordinate data of a predetermined electronic device,
A contact detecting unit for calculating a contact point between the virtual plane of the electronic device and the straight line based on direction information corresponding to a straight line between the user and the electronic device;
An instruction code generation unit for generating a command code corresponding to the position of the contact point and performing an operation of a predetermined electronic device;
And an electronic device drive control unit for converting the command code generated by the command code generating unit into a control command signal for driving the corresponding electronic device and transmitting an operation command signal to the corresponding electronic device A remote control device using a virtual plane of an electronic device in a virtual touch device that does not use a pointer. The apparatus of claim 1, wherein the three-dimensional coordinate calculation unit
An image acquiring unit that is composed of two or more image sensors disposed at different positions and captures a user's body at different angles;
Dimensional coordinate data of the user's body using optical triangulation based on images taken at different angles received from the image acquiring unit, and extracting three-dimensional coordinate data of a predetermined electronic apparatus And a coordinate calculation unit. The remote control apparatus using a virtual plane of an electronic device in a virtual touch apparatus that does not use a pointer The apparatus of claim 1, wherein the three-dimensional coordinate calculation unit
An illumination assembly composed of a light source and a diffuser, for projecting the speckle pattern onto the user's body and the electronic device,
An image acquisition unit configured to capture a speckle pattern of the user's body projected by the illumination assembly,
Dimensional coordinate data of the user's body on the basis of the speckle pattern captured by the image obtaining unit and extracting three-dimensional coordinate data of a predetermined electronic device using a pointer A remote control apparatus using a virtual plane of an electronic device in a virtual touch device which is not used. 2. The apparatus according to claim 1, wherein the contact detecting unit
Wherein a straight line connecting the first spatial coordinate and the second spatial coordinate calculated by calculating the first spatial coordinate and the second spatial coordinate based on the three-dimensional spatial coordinate of the user's body calculated by the three- A touch position calculating unit for calculating contact point coordinate data to be contacted with a contact point area of the plane,
And a virtual touch processing unit for determining whether or not the contact coordinate data calculated by the touch position calculating unit changes and generating a command code for performing an operation of a predetermined electronic device when it is determined that there is a change, A remote control device using a virtual plane of an electronic device in an unused virtual touch device. 5. The method of claim 4,
Wherein the first spatial coordinate is any three-dimensional coordinate of an end of one of the fingers of the user and an end of a pointer held by the user with a finger, and the second spatial coordinate is a coordinate of the center of one of the eyes of the user Dimensional coordinates of the virtual touch device using the virtual plane of the electronic device. 6. The apparatus of claim 5, wherein the virtual touch processing unit
Whether or not there is a change in the contact point coordinate data for a predetermined time or more from the time when the initial contact point coordinate data is calculated and if there is a change in the contact point coordinate data for a set time or longer, A remote control device using a virtual plane of an electronic device in a virtual touch device that does not use a virtual touch panel. 6. The apparatus of claim 5, wherein the virtual touch processing unit
If there is no change in the contact point coordinate data over the set time from the time at which the initial contact point coordinate data is calculated, it is determined whether or not a distance change has occurred at a distance greater than or equal to the set distance between the first and second spatial coordinates And a command code for judging that there is a change and performing an operation of a preset electronic device is generated. A remote operation apparatus using a virtual plane of an electronic device in a virtual touch device not using a pointer.

Images