WO2015064991A2

WO2015064991A2 - Smart device enabling non-contact operation control and non-contact operation control method using same

Info

Publication number: WO2015064991A2
Application number: PCT/KR2014/010151
Authority: WO
Inventors: 차재상
Original assignee: 서울과학기술대학교 산학협력단
Priority date: 2013-10-28
Filing date: 2014-10-28
Publication date: 2015-05-07
Also published as: KR20150048608A; WO2015064991A3; KR101535738B1

Abstract

Disclosed are a smart device enabling non-contact operation control and a non-contact operation control method using the same. The smart device comprises: a device body, a camera sensor unit, a non-contact operation recognition unit, and a control unit. The display unit is arranged on a surface of the device body. The camera sensor unit has at least three cameras arranged on different corner portions of the device body, respectively, and acquires image information regarding a target object through the cameras, respectively. The non-contact operation recognition unit performs analysis on changes of the shape, position, and size of the target object, on the basis of the image information acquired through the camera sensor unit, and generates non-contact operation recognition information according to the analysis on the changes. The control unit displays a screen, which is controlled according to the generated non-contact operation recognition information, on the display unit. As such, according to the present invention, at least three cameras recognize a non-contact operation of an object, thereby recognizing the operation of the object more precisely, and the user is provided with a convenient and lively feeling of using the device in an environment requiring various user interfaces.

Description

Smart device capable of non-contact motion control and non-contact motion control method using the same

The present invention relates to a smart device, and more particularly, to a smart device capable of non-contact motion control and a non-contact motion control method using the same.

As the use of information devices is becoming more common due to the development of computer information technology in the modern society, the importance of natural interaction between devices and humans is increasing. Accordingly, information devices are evolving from a text-oriented interface to a man-machine interface (MMI) method that uses various human gestures and emphasizes user experience.

Since the MMI method directly inputs information by using an intuitive tool, it does not need a separate input device and has an advantage that it can be used regardless of user training. Recently, in order to provide various user interface environments, a touch screen in which a touch pad is combined with a display module has been adopted in a smart device. As a result, a user can input various user names or select a function while viewing a screen implemented in the display module. It became.

However, in a smart device using a conventional touch screen, the user can select a function related to the menu or icon by simply touching a menu or icon displayed on the screen of the smart device with a finger, and various user interfaces according to the user's operation. There is a problem of not providing the environment.

In addition, with the recent development of wireless Internet technology, full browsing (not only to view web pages in the same form as web pages viewed as a personal computer through smart devices) but also to search all wired websites ( Full Browsing) service is provided.

However, the compact display module included in the smart device displays the text, graphics, images, video, and flash included in the web page densely, so that a specific image size is usually smaller than the contact area between the user's finger and the touch screen. Therefore, a problem occurs in that another image or text, which is arranged adjacent to the specific image, is touched, and there is a problem of frequently covering the screen by a finger.

In addition, the touch screen must be touched by a finger or a touch fan, but as the device can be manipulated, frequent touches may cause damage to liquid crystals or deterioration of recognition of certain parts. Therefore, it is necessary to implement a new input method for executing a desired function without the user touching the touch screen.

The present invention is to solve the above problems according to the prior art. That is, the object of the present invention, by recognizing the non-contact motion of the object through at least three or more cameras, the motion recognition can be made more precisely, and because it is non-contact, more lively user interaction beyond the input limit of the conventional touch method The present invention provides a smart device capable of providing non-contact operation control and a non-contact operation control method using the same.

As a technical idea for achieving the above object, a smart device capable of non-contact motion control of the present invention includes a device main body, a camera sensor unit, a non-contact motion recognition unit, and a control unit.

The display unit is disposed on one surface of the device main body.

The camera sensor unit includes at least three cameras disposed at different corners of the device main body, and acquires image information on the target object through the camera. In this case, the camera sensor unit may be disposed on the front or rear or front and rear of the device body, the image information is the image information of the captured image of the target object, the time information and the captured camera information, etc. It includes.

The non-contact motion recognition unit analyzes the shape, position, and size change of the target object based on the image information acquired through the camera sensor unit, and generates non-contact motion recognition information according to the change analysis. In addition, the non-contact motion recognition unit may calculate a depth map by matching the image information for each camera, an object detector for detecting a target object, a shape recognition unit for recognizing the shape of the target object detected by the object detector, the target object A position measuring unit for measuring the position of the target object by using the parallax average for each camera of the camera and the information generating unit for generating the non-contact motion recognition information by changing the shape, position and size of the target object.

The controller displays a screen controlled according to the generated non-contact gesture recognition information on the display unit.

The smart device capable of controlling the contactless motion of the present invention includes a storage unit configured to store shape information for recognizing the shape of the target object and preset information for generating contactless motion recognition information according to a shape, position, and size change of the target object. can do. In addition, it may include a sensing device for determining whether the camera sensor unit is activated.

In the non-contact motion control method using a smart device according to an embodiment of the present invention, obtaining image information on a target object through at least three cameras, calculating depth maps by matching image information of each camera, and calculating Detecting the target object in the image information based on the received depth map, measuring the position of the target object using the disparity average for each camera of the target object, and recognizing the non-contact motion by changing the shape, position, and size of the target object. Generating information and controlling the screen according to the generated non-contact motion recognition information.

After detecting the target object in the image information, the method may further include recognizing a shape of the detected target object to recognize a specific shape or change in shape.

In the step of acquiring image information about the target object, when the target object does not exist for a predetermined time by a sensing device provided on one side of the device, the camera is switched to a standby state and the target object is detected by the sensing device. When the presence of the is detected the camera can be activated.

Smart device according to the present invention, by at least three or more cameras to recognize the contactless operation of the object, it is possible to recognize the operation of the object more precisely, the user experience in a convenient and lively device in an environment that requires a variety of user interface Can give

In addition, in the related art, the operation of the object can be recognized only when the target object is located within the proximity of the smart device. However, the present invention limits the separation distance between the smart device and the target object by identifying the position of the target object using a camera. There is no continuous motion recognition in real time. In addition, there is an effect to prevent damage to the screen liquid crystal of the mobile device due to frequent contact.

1 is a schematic internal configuration diagram of a smart device according to an embodiment of the present invention.

2 is an external perspective view illustrating an example in which a camera sensor unit is mounted on a smart device according to an embodiment of the present invention.

3 is a conceptual diagram illustrating an example of the object object positioning according to an embodiment of the present invention.

4 is a flowchart sequentially illustrating a method for controlling contactless operation of a smart device according to an embodiment of the present invention.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text.

However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "having" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features It should be understood that it does not exclude in advance the possibility of the presence or addition of numbers, steps, actions, components, parts or combinations thereof.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The smart device of the present invention may be a smartphone, a mobile phone, an MP3, a game machine, a navigation device, a TV, a display device, a notebook computer, a tablet computer, a personal media player (PMP), a personal digital assistant (PDA), and the like. In the embodiment of the present invention will be described using a smartphone that is a typical configuration of a smart device. In addition, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

As shown in FIG. 1, the smart device of the present invention includes a device main body 100, a display unit 200 disposed on the front of the main body 100, and a camera disposed at different ends of the main body 100. At least three or more 310 are provided, based on the image information obtained through the camera sensor unit 300, the camera sensor unit 300 for obtaining image information on the target object through the camera 310, respectively; A non-contact motion recognition unit 400 which analyzes the shape, position, and size of the target object, generates contactless motion recognition information according to the change analysis, and a controller 500 that controls the device according to the generated non-contact motion recognition information. It is configured to include.

First, the display 200 displays various screens to be provided to the user, and displays the screen controlled according to the screen control signal received from the controller 500. For example, when the user moves his hand from right to left, the display unit 200 receives a screen control signal including the content of the screen page from the control unit 500 to the next page, the next page on the screen page Display the screen moved to. The display unit 200 of the present invention may be configured as a touch screen having a conventional multi-touch function.

The camera sensor unit 300 acquires image information about the target object and transmits the image information to the non-contact motion recognition unit 400, where the image information includes image information of capturing the image of the target object, time information and image capturing the image. It contains one camera information. The camera sensor unit 300 of the present invention, at least three cameras are required to accurately calculate the position of the target object, it is preferable to be located at different ends of the surface of the device body (100).

In the present invention, as shown in Figure 2, the four cameras (310a ~ 310d) are located along each corner of the front of the device body 100, so that the camera 310 without the dead zone (Dead Zone) of the display unit You can use all four installed areas. In addition, the camera sensor unit 300 of the present invention is preferably configured as a parallel camera (parallel camera) structure to have a built-in structure that does not protrude on one surface of the device main body 100, the device main body 100 It can also be placed on both the front and back.

The non-contact motion recognition unit 400 analyzes the shape, position, and size change of the target object based on the image information acquired through the camera sensor unit 300, and generates non-contact motion recognition information according to the change analysis. . To this end, the non-contact motion recognition unit 400 includes an object detector 410, a shape recognition unit 420, a position measuring unit 430, and an information generating unit 440. In addition, the present invention stores the shape information for recognizing the shape of the target object through the non-contact motion recognition unit 400 and the preset information for generating the non-contact motion recognition information according to the shape, position, size change of the target object is stored The unit 600 is provided.

The present invention estimates the distance of the target object according to the stereo vision method. In the case of estimating distance using a stereo vision system, a depth map is generated by matching images captured by cameras. The target object is detected using the depth map and the captured images, and the parallax average in the detected target object is obtained. In the stereo vision system, the distance of an object may be estimated using the obtained parallax. The technique of estimating the distance of an object using the stereo vision technique is a widely used position tracking technique, and a detailed description thereof will be omitted.

The object detector 410 matches the image information for each camera to calculate a depth map, and detects a target object in the image information based on the calculated depth map. Accordingly, the shape recognition unit 420 may recognize the specific shape of the target object, and the position measuring unit 430 obtains a parallax value for the center portion of the detected target object and calculates a distance between the camera and the target object. Will be available. As a matching method for calculating the depth map, a method such as DP (Dynamic Programming), BP (Belief Propagation), and Graph-cut may be used. In addition, in the process of detecting an object using a depth map, an object may be extracted using various methods. In the present invention, a vertical parallax map (V-disparity) may be used to distribute a parallax value mapped to a vertical parallax map. Accordingly, the object included in the image information is detected. That is, the sum of regions having a disparity value and a disparity value different from the vertical disparity map may be detected as the target object region, or an area of pixels having the same disparity value may be detected as the target object region.

The shape recognizer 420 recognizes the shape of the target object detected by the object detector 410. Accordingly, the present invention can recognize the non-contact motion recognition information according to the recognition or shape change of the specific shape of the detected target object. For example, you can distinguish between an extended finger and a folded finger by changing the shape of a certain shape such as scissors, rocks, beams, and bent fingers in an open palm. You can play like a real instrument without touching it, or perform live game control.

The position measuring unit 430 measures the position of the target object using the parallax average for each camera of the target object. The position measuring unit 430 of the present invention calculates the position of the target object based on image information acquired through at least three cameras, estimates the distance from the center position between the adjacent cameras to the target object, respectively. The exact position coordinates (x, y, z) for the target object are derived according to the closest points of the estimated distances. Accordingly, it is possible to more accurately determine the position of the target object than when using only two cameras, the left and right cameras, for positioning.

For example, as illustrated in FIG. 3, when image information is acquired through four cameras 310a to 310d, the center object between the first camera 310a and the second camera 310b to the target object. Distance (a), distance from the center position between the second camera (310b) and the third camera (310c) to the target object (b), between the third camera (310c) and the fourth camera (310d) The distance (c) from the center position to the target object and the distance (d) from the center position between the fourth camera 310d and the first camera 310a to the target object through the trigonometric formula using the stereo vision algorithm Each of them is calculated and the correct position coordinates for the target object are derived.

The information generator 440 generates non-contact motion recognition information through the shape, position, and size change of the target object through the shape recognizer 420 and the position measurer 430. That is, it is possible to estimate the direction, pattern, and speed of the up, down, left, right, and rotational movements through the change of the position of the target object, and to estimate the shape change, perspective, and speed through the change of the size of the target object. It is possible to generate contactless motion recognition information corresponding thereto.

When the control unit 500 receives the non-contact motion recognition information from the non-contact motion recognition unit, the control unit 500 identifies the non-contact motion recognition information, generates motion control information, and transmits a screen control signal according to the generated motion control information to the display unit 200. By doing so, the screen of the display unit can be controlled. In addition, the controller 500 performs operation control of the smart device according to the operation control information.

In addition, the present invention may further include a sensing device 700 which is a device for determining whether the camera sensor unit 300 is activated. The sensing device 700 of the present invention may be configured as one of an infrared ray, an ultrasonic wave, and a proximity sensor, and the camera sensor unit 300 is in a standby state when a target object does not exist for a predetermined time, and then through the sensing device 700. When the presence of the target object is detected, the state is activated.

As shown in FIG. 4, the present invention first obtains image information on a target object through at least three cameras (S810). Subsequently, a depth map is generated by matching images photographed for each camera using a stereo vision method (S820).

Next, the target object in the image information is detected based on the generated depth map (S830). In this case, the present invention may recognize the detected shape of the target object, and may recognize the non-contact motion recognition information according to the recognition or change of the shape. For example, it is possible to distinguish between an extended finger and a folded finger by changing the shape of a bent finger in the open palm, such as scissors, rocks, and beam motions. You can play as if you were playing a real instrument, or perform live game control without touching.

Next, the position of the target object is measured using the parallax average for each camera of the target object (S840). In the present invention, the position of the target object is calculated based on the image information acquired through at least three cameras, the distance from the center position between the adjacent cameras to the target object is respectively estimated, and at the closest point of the estimated distances. Therefore, the exact position coordinates (x, y, z) for the target object are derived.

Next, the non-contact motion recognition information is generated by changing the shape, position, and size of the target object (S850). In other words, the direction, pattern, and speed of up, down, left, right, and rotational movements can be estimated by changing the position of the target object, and the shape change, perspective and speed can be estimated by changing the size of the target object. It is possible to generate the non-contact motion recognition information.

Subsequently, the generated non-contact motion recognition information is identified to generate motion control information corresponding thereto (S860), and screen control according to the generated motion control information is performed (S870).

As described above, the present invention is able to recognize the contactless motion according to the up, down, left, right, rotational movement direction and perspective, pattern, speed, shape change of the target object through three or more cameras. The operation of the screen and the device can be controlled according to the operation. Accordingly, the present invention is not limited to the motion recognition according to the separation distance between the smart device and the target object, it is possible to provide a more lively user interaction by enabling the motion recognition of the continuous object in real time.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

Claims

Device body;

A display unit disposed on one surface of the main body;

At least three cameras disposed at different ends of the main body, respectively, and a camera sensor unit configured to obtain image information on the target object through the camera;

A non-contact motion recognition unit for performing a shape, position, and size change analysis of the target object based on the image information acquired through the camera sensor unit, and generating non-contact motion recognition information according to the change analysis; And

A controller configured to display a screen controlled according to the generated non-contact gesture recognition information on the display unit;

Smart device capable of contactless motion control configured to include.
The method of claim 1,

The camera sensor unit,

Smart device capable of non-contact operation control, characterized in that disposed on the front or rear or both front, and rear of the device body.
The method of claim 1,

The video information,

The smart device capable of non-contact operation control, comprising image information of capturing an image of the target object, time information of capturing an image, and photographed camera information.
The method of claim 1,

The non-contact motion recognition unit,

An object detector configured to calculate a depth map by matching the image information of each camera, and detect a target object;

A shape recognition unit recognizing a shape of the target object detected by the object detection unit;

A position measuring unit configured to measure a position of the target object by using a parallax average of each target camera;

An information generator configured to generate contactless motion recognition information by changing a shape, a position, and a size of the target object;

Smart device capable of non-contact operation control, characterized in that configured to include.
The method of claim 3, wherein

The apparatus may further include a storage unit configured to store shape information for recognizing a shape of the target object and preset information for generating non-contact motion recognition information according to a shape, position, and size change of the target object. Smart devices.
The method of claim 1,

Smart device capable of non-contact operation control, characterized in that further provided with a sensing device for determining whether the camera sensor unit is activated.
In the non-contact motion control method using a smart device provided with at least three cameras respectively disposed at different ends of the device body,

Acquiring image information about the target object through at least three cameras;

Calculating a depth map by matching image information of each camera, and detecting a target object in the image information based on the calculated depth map;

Measuring a position of the target object using a parallax average for each camera of the target object;

Generating contactless motion recognition information by changing a shape, a position, and a size of the target object;

Controlling the screen according to the generated contactless recognition information;

Contactless control method of the smart device configured to include.
The method of claim 7, wherein

After detecting the target object in the image information,

And recognizing a shape or a change in shape by recognizing a shape of the detected target object.
The method of claim 7, wherein

In acquiring image information about the target object,

When the target object does not exist for a predetermined time by the sensing device provided on one side of the device, the camera is switched to the standby state, and when the presence of the target object is detected by the sensing device, the camera is activated. Non-contact motion control method of the smart device.