KR101465896B1 - Mobile terminal for generating control commands using front side camera and rear side camera - Google Patents

Abstract

A mobile terminal (100) for generating control commands using a front-facing camera and a rear-facing camera comprises the followings: the front-facing camera (110) configured to receive a first target image including a first color marker region attached on a finger; a rear-facing camera (120) configured to receive a second target image including a second color marker region attached on another finger; a marker region detection module (130) configured to detect the first color marker region from the first target image and the second color marker region from the second target image using a color detection technology; and a coordinates calculation module (140) configured to calculate first three-dimensional coordinates values using the first color marker region and second three-dimensional coordinates values using the second color marker region.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a mobile terminal for generating a control command using a front-side camera and a rear-

The present invention relates to a mobile terminal that generates a control command such as a three-dimensional coordinate value by using a finger image input to a front side camera and a rear side camera.

Recently, as a mobile terminal such as a smart phone becomes popular, a variety of interface technologies using a touch screen, a sensor, and a camera included in a mobile terminal are being studied.

Especially, there is a lot of research on how to use a specific gesture as an interface by analyzing images input to a camera.

In recent years, a mobile terminal has introduced a technique of simultaneously using a front side camera and a rear side camera.

Korean Patent Publication No. 10-2011-0037053

"Real-time Hand Pose Recognition Using HLF (Haar-like Feature)", Proceedings of the 2007 HCI Conference, Vol. 1, pp.897-902, February 2007. Kim, JK, Kim,

Conventionally, the interface method using a camera mainly generates a specific command by recognizing a specific gesture (such as a hand gesture) or the movement of the user's pupil. Conventionally, the research for generating three-dimensional coordinates mainly requires additional equipment or a separate interface input device. Also, the image processing process has a problem that the object is not recognized properly when the background is complicated or the background is a specific color.

The technique described below is intended to generate three-dimensional coordinates using a finger image acquired using a front-side camera and a rear-side camera simultaneously. In the technique described below, when it is difficult to detect a finger region in an image input to a camera, a marker is used to generate three-dimensional coordinates.

The solutions to the technical problems described below are not limited to those mentioned above, and other solutions not mentioned can be clearly understood by those skilled in the art from the following description.

A mobile terminal for generating a control command using a front side camera and a rear side camera for solving the above problems includes a front side camera which receives a first target image including a first color marker area attached to a finger, Detecting a first color marker region in a first target image using a rear face camera that receives a second target image including a second color marker region, a color detection technique, and detecting a second color marker region in the second target image And a coordinate calculation module for calculating a first three-dimensional coordinate value using the first color marker area and a second three-dimensional coordinate value using the second color marker area.

A mobile terminal for generating a control command using a front side camera and a rear side camera of another aspect for solving the above problems includes a first target image area including at least one of a first finger area or a first color marker area attached to a finger, A rear face camera for receiving a second target image including at least one of a front face camera, a second face area, and a second color marker area attached to a finger, A fingertip detection module for detecting an ending region in at least one of the finger regions, a marker region detection module for detecting at least one of the first color marker region and the second color marker region using a color detection technique, A control module for driving the fingertip detection module or the marker detection module, Calculating a first three-dimensional coordinate value using at least one of an end area of the lock area and a first color marker area, calculating a first three-dimensional coordinate value using at least one of an end area of the second finger area and a second color marker area, And a coordinate calculation module for calculating a coordinate value

The mobile terminal for generating the control command using the front side camera and the rear side camera may further include a coordinate matching module for matching the first three-dimensional coordinate values and the second three-dimensional coordinate values in one three-dimensional space.

The technique described below provides various interface methods by simultaneously using the front-end camera and the rear-panel camera of the mobile terminal. In addition, the technology described below enables a user to recognize a marker attached to a finger through a camera of the terminal in various backgrounds and use it as an interface means.

The technique described below allows a user to conveniently input a control command in an application executed in a mobile terminal. Furthermore, the present invention can be applied not only to mobile terminals but also to various devices (remote controllers, controllers, etc.) equipped with cameras. For example, it can be applied to a controller of a game machine.

The effects of the techniques described below are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 shows an example of generating a control command using a front side camera and a rear side camera.
2 is a block diagram illustrating a configuration of a mobile terminal for generating a control command using a front side camera and a rear side camera.
Fig. 3 shows an example of a marker attached to a finger.
FIG. 4 shows an example of a process of detecting a color marker region using the CAMshift algorithm.
FIG. 5 shows an experimental example of detecting a color marker region using the CAMshift algorithm.
6 (a) shows a three-dimensional coordinate space of a front-side camera and a three-dimensional coordinate space of a rear-side camera. FIG. 6 (b) shows an example of a three-dimensional coordinate space of a front- do.
7 is another example of a block diagram illustrating a configuration of a mobile terminal for generating a control command using a front side camera and a rear side camera.
8 shows an example of a reference for generation of a three-dimensional coordinate value using a finger region.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner. Accordingly, the presence or absence of each component described in the present specification should be interpreted as a function. For this reason, the components (100, 200) according to the mobile terminal (100, 200) generating the control command using the front side camera and the rear side camera of the present invention It is to be clearly understood that the configuration of the present invention may be different from the corresponding drawings within the scope of achieving the object of the present invention.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The present invention takes a finger image of a user using a camera built in a terminal including a mobile terminal and transmits a specific control command (a three-dimensional coordinate value or the like) to an application executed in the mobile terminal or the mobile terminal, As shown in FIG. Further, the present invention can detect a specific area (color marker area) represented by a color marker in a finger image and transmit a specific control command to the mobile terminal based on the color marker area. In the environment where the finger can be easily detected from the image input to the camera, a control command for controlling the specific object is detected by detecting the finger region. If the finger is not easily detected, the color marker region is detected, Can be generated.

A control command is a command input by a user in an application executed in a mobile terminal or the like. For example, the control command may be a two-dimensional or three-dimensional coordinate value for coordinating a particular object in an application, such as a game. The control command may also be an input for selecting a particular menu in a particular application. A control command generally corresponds to a command that changes the position of a particular object.

The target image means a source image including a finger image. The target image is captured through an image acquisition device such as a camera. A user executes a specific application through a mobile terminal, and controls the movement (coordinates) of a specific object of the application with a finger while viewing a display screen on which the application is executed.

The present invention acquires two target images using a front side camera and a rear side camera built in a mobile terminal. The target image acquired by the front side camera is referred to as a first target image and the target image acquired by the rear side camera is referred to as a second target image. The front side camera means a camera in a direction in which a display device is installed in a mobile terminal and the rear side camera means a camera installed in a rear direction of the display device.

1 shows an example of generating a control command using a front side camera and a rear side camera. 1 (a) shows an example of movement for generating two-dimensional coordinates (x and y) using the fingers of the user's both hands on the front side camera and the rear side camera, respectively. Fig. 1 (b) shows an example of movement for generating z-direction coordinates using the fingers of the user's hands on the front side camera and the rear side camera, respectively.

1 shows only one example using a front-side camera and a rear-side camera. For example, as in the case of using the thumb and index finger, the user can input a finger image to the front side camera and the rear side camera with one hand. Also, two or more users may input finger images to the front side camera and the rear side camera, respectively.

The present invention can control one object (output to the display device) operating in an application using a front-side camera and a rear-side camera at the same time, or control a plurality of objects. Furthermore, two users can control different objects using the front camera and the rear camera, respectively. For example, in an application such as a game, two people may enjoy the game against each other. The present invention described below does not limit the various applications utilizing the present invention.

In the present invention, the mobile terminal includes a smart phone, a tablet PC, a notebook, a portable game machine, a remote controller used in a TV or a game machine, and the like. Furthermore, the present invention can also be used when a general PC having no mobility, a terminal connected to a server, or the like is used.

Hereinafter, the mobile terminals 100 and 200 that generate control commands using the front-side camera and the rear-side camera will be described in detail with reference to the drawings.

2 is an example of a block diagram illustrating a configuration of a mobile terminal 100 that generates a control command using a front side camera and a rear side camera.

The mobile terminal 100 for generating a control command using a front side camera and a rear side camera includes a front side camera 110 receiving a first target image including a first color marker area attached to a finger, A rear camera 120 for receiving a second target image including a second color marker region, a first color marker region in the first target image using a color detection technique, and a second color marker region in the second target image, A coordinate calculation module 140 for calculating a first three-dimensional coordinate value using the first color marker area and a second three-dimensional coordinate value using the second color marker area, ).

FIG. 2 illustrates a color marker attached to a finger and is used as a reference of an interface. The color marker attached to the finger is preferably a color other than the skin color. The color marker portion appearing in the first target image is called a first color marker region and the color marker portion appearing in the second target image is called a second color marker region.

The front-side camera 110 and the rear-side camera 120 receive a finger image of a user with a color marker attached thereto, respectively, and transmit the image to the marker region detection module 130.

The marker region detection module 130 can detect a color marker region in a target image using a Continuous Adaptive Mean Shift (CAMshift) algorithm among color detection techniques.

The CAMshift algorithm is an improvement on the Meanshift algorithm that tracks objects of interest at high speed based on the density distribution (feature points, corners, colors) of the data set. The CAMshift algorithm complements the disadvantage of mean-shift by using the technique of adjusting the size of the search window by itself. The CAMshift algorithm predicts the position to be changed by using the distribution of the Hue value of the region of interest of the object, and tracks the object by detecting the center after detecting it. Furthermore, it is apparent that various algorithms other than the CAMshift algorithm can be used to detect the color marker region.

When the marker area detection module 130 detects the first color marker area in the first target image, the coordinate calculation module 140 can calculate the three-dimensional coordinate value using the first color marker area. The three-dimensional coordinate value calculated using the finger image input to the front-side camera is called a first three-dimensional coordinate value. When the marker area detection module 130 detects the second color marker area in the second target image, the coordinate calculation module 140 can calculate the three-dimensional coordinate value using the second color marker area. The three-dimensional coordinate value calculated using the finger image input to the rear-side camera is referred to as a second three-dimensional coordinate value.

When the color marker region is designated as a region of interest, the marker region detection module 130 converts the RGB image into the HSV image, generates a basic image using the maximum frequency characteristic of the histogram for the H channel of the HSV image, A mask image is generated by summing up the results of extracting the threshold values for the S channel and the V channel of each of the S and V channels, and a color marker region is detected by combining the basic image and the mask image. The detailed procedure will be described later.

The interface input module 160 refers to a module that receives the final three-dimensional coordinates calculated through the coordinate calculation module 140. The interface input module 160 transmits the three-dimensional coordinates to a mobile terminal or an application driven by the mobile terminal.

The second three-dimensional coordinate value calculated using the first three-dimensional coordinate value calculated using the first color marker area acquired by the front-side camera and the second color marker area acquired by the rear-side camera corresponds to the object The first three-dimensional coordinate value and the second three-dimensional coordinate value are preferably matched in the same three-dimensional space. For this, the mobile terminal 100 may include a coordinate matching module 150. The details will be described later.

Fig. 3 shows an example of a marker attached to a finger. Fig. 3 (a) shows an example in which a marker in the form of a sticker is attached to a fingertip, Fig. 3 (b) shows an example in which a marker in the form of a band is attached to the fingertip, Fig. 3 Fig. 3 (d) is an example of a form in which a color marker is colored at the fingertip. Figure 3 is merely an example of a marker that can be used in the present invention. Therefore, it is apparent that markers of various colors, various sizes, and shapes not illustrated in Fig. 3 can be used. 3 is an example of a target image input to an actual camera. In FIG. 3, the dotted area indicated by 'M' corresponds to the color marker area.

FIG. 4 illustrates an example of a process 400 for detecting a color marker region using the CAMshift algorithm. Means the process of operating the marker region detection module 130 described above.

First, a target image is input through a camera (410). In the input target image, a color marker region is designated as a region of interest 420. The user can specify the region of interest by selecting a color marker region in the target image. For example, if the display device of the mobile terminal is a touch panel, a color marker area can be touched to designate it as a region of interest. Alternatively, the user may designate the color marker area as a region of interest by using a cursor or the like. Further, a color marker region may be extracted from the target image using an image processing technique.

Once the region of interest to be processed is determined, the position to be changed is predicted through color analysis of the region of interest, and the object is tracked by detecting the center after the detection.

The RGB image of the target image input through the camera is converted into the HSV image (430). In this process, HSV converted images are separated into Hue, Saturation, and Value channels. In the present invention, saturation channel (S channel) and value channel (V channel) as well as a Hue channel (H channel) are used in combination to solve a problem of overlapping of similar color objects or backgrounds And to ensure robust tracking performance against background and illumination changes.

Thereafter, image processing is performed on each channel, and a process performed in the H channel, a process performed in the S channel and the V channel can be performed in any order.

The process performed on the H channel includes performing a histogram analysis on the H channel (441) and regenerating the image using the maximum frequency characteristic of the analyzed histogram (442). In the present invention, an image to be regenerated in this process is called a basic image.

The process performed in the S channel and the V channel includes a step 451 of extracting the maximum value and the minimum value of the S channel using the maximum threshold value and the minimum threshold value in the S channel, A step 452 of extracting the minimum value, and a step 453 of generating a mask image by performing an AND operation on the value extracted from the S channel and the value extracted from the V channel. In addition to the Hue information, Saturation and Value channels are used additionally to improve the color discrimination power at low hue values with a small amount of computation.

A final image is generated by combining the basic image generated in the H channel with the mask image generated using the S channel and the V channel in an AND relationship, and a color marker region is detected in the generated final image (470) .

FIG. 5 shows an experimental example of detecting a color marker region using the CAMshift algorithm. 5 (a) shows a target image including a color marker region, and FIG. 5 (b) shows an example converted into an HSV image. 5 (c) shows an example of separating H channels from the HSV image, and FIG. 5 (d) shows an example of generating a basic image on the H channel. 5 (e) shows an example of generating a mask image using the S channel and the V channel separated from the HSV image. In FIG. 5, (f) shows an example of generating a final image by ANDing a basic image and a mask image, and (g) shows an example of detecting a color marker region in the final image.

In the mobile terminal 100, which generates a control command using a marker attached to a finger, the coordinate computing module 430 generates a three-dimensional coordinate using the detected color marker area. The coordinate computing module 430 estimates the X, Y, and Z coordinates using the width (width) or diameter of the detected color marker area. Hereinafter, the process of generating the three-dimensional coordinates by the coordinate calculation module 430 will be described in detail. The second three-dimensional coordinates are generated using the first three-dimensional coordinates generated using the first color marker region acquired by the front-side camera 110 and the second color marker region acquired by the rear-side camera 120, This is common to the coordinates.

In the present invention, the Z coordinate is estimated using the diameter when the shape of the color marker is circular, and the Z coordinate is estimated using the width or height when the shape is a rectangle. In other cases, the Z-coordinate can be estimated using the width or length of the reference area.

When the marker region detection module 130 detects the color marker region, it is possible to obtain the x-coordinate value and the y-coordinate value in the image of the central pixel of the color marker region, and obtain the width or diameter of the color marker region . In the description of the present invention, (1) upper case letters such as X, Y and Z denote coordinate values in a virtual three-dimensional space usable in a mobile terminal, and (2) lower case letters such as x and y The displayed coordinate value means a coordinate value based on the pixel position in the image.

The Z coordinate is mapped to the Z coordinate using the estimated distance value after the distance is estimated using the width or diameter of the detected color marker area. The width or diameter of the color marker region is estimated using Equation 1 below

Where ColorPoint is the actual width or diameter of the color marker area, PreviewWidth is the pixel value for the width or diameter of the entire target image input to the camera, ColorPoint.Region is the pixel width or diameter of the color marker area in the target image, Value, and WidthFOV is the horizontal viewing angle of the camera.

 For example, ColorPoint is 0.5 cm, PreviewWidth is 160 pixels, and WidthFOV is 62 degrees. Given the shape of the various color markers, ColorPoint can be set to the width of a polygon or to the diameter of a circle.

The variable used in Equation (1) may be a constant depending on the front-side camera 110 and the rear-side camera 120. Depending on the size of the color marker area that is input to the camera, the ColorPoint and ColorPoint.Region may vary, and PreviewWidth, ColorPoint.Region, and WidthFOV may vary depending on camera specifications.

Therefore, in the case of the first three-dimensional coordinate value, ColorPoint is the actual width or diameter of the first color marker region, PreviewWidth is the pixel value with respect to the width of the entire first target image, ColorPoint.Region is the pixel width or diameter of the first color marker region , ColorPoint is the actual width or diameter of the second color marker region, PreviewWidth is the pixel value of the width of the entire second target image, ColorPoint.Region is the width of the second target image, 2 The pixel value for the width or diameter of the color marker area, WidthFOV is the horizontal viewing angle of the rear camera. Different values may be applied to the variable values for the front-side camera 110 and the rear-side camera 120 for all equations below.

The X coordinate is mapped to the X coordinate using the center x pixel coordinate of the detected finger area and the distance estimated from equation (1). First, we need to know the horizontal length (X direction length) that can be obtained from the camera according to the distance to know the X coordinate. The obtainable horizontal length of the camera can be obtained from the following equation (2). The obtainable width means the actual length.

Here, FingerPoint.Z is the Z coordinate value calculated in Equation (1), and WidthFOV is the horizontal viewing angle of the camera. For example, a WidthFOV of 62 degrees can be used.

The distance value per pixel can be calculated using the value obtained by Equation (2). The formula for calculating the distance per pixel is given by Equation 3 below.

FowWidthLength is an obtainable length value calculated through Equation (2), and PreviewWidth is a width pixel value of the entire target image input to the camera.

The X coordinate value of the three-dimensional coordinate can be calculated using the result of the above equations as shown in Equation (4) below.

Where ColorPoint.x is the x pixel coordinate value for the center pixel of the color marker region in the target image and DistancePerPixel is the distance value per pixel in the horizontal direction.

As described above, different values may be applied to the parameters of Equations (2) to (4) depending on the front-side camera 110 or the rear-side camera 120.

The Y coordinate maps to the Y coordinate using the center y pixel coordinates of the detected finger area and the distance estimated from equation (1). First of all, you need to know the obtainable vertical length (Y direction length) of the front camera or rear camera depending on the distance to know the Y coordinate. The vertical length taken by each camera can be obtained from the following equation (5).

Here, FingerPoint.Z is the Z coordinate value obtained by Equation (1), and Height FOV is the vertical viewing angle of the camera. As an example, Height FOV can be used at 45 degrees.

The distance value per pixel can be calculated using the value obtained by Equation (5). The distance per pixel value can be calculated using Equation (6) below.

Here, FovHeightLength is a vertical length value obtainable by the camera, and PreviewHeight is a pixel value with respect to the height of the entire target image input to the camera.

The Y coordinate values of the three-dimensional coordinates can be calculated using the values calculated through Equations (5) and (6) as shown in Equation (7) below.

Where ColorPoint.y is the y pixel coordinate value for the center pixel of the color marker region in the target image and DistancePerPixel is the distance value per pixel in the vertical direction.

The values applied to the variables of Equations (5) to (7) may be different values depending on the front-side camera 110 and the rear-side camera 120.

As described above, the coordinate calculation module 140 of the mobile terminal 100, which generates the control commands using the front-side camera and the rear-side camera, calculates the three-dimensional coordinate values using Equations 1, 4 and 7 . Specifically, the coordinate calculation module 140 can calculate the first three-dimensional coordinate value using the first color marker region of the first target image input to the front-side camera 110, And the second three-dimensional coordinate value can be calculated using the second color marker region of the second target image.

As described above, the mobile terminal 100, which generates the control commands using the front-side camera and the rear-side camera, generates a coordinate matching module 150 for matching the first three-dimensional coordinate values and the second three- . Since the images obtained by the front side camera 110 and the rear side camera 120 may be different from each other and the performances of the cameras may be different from each other or the range of movement of the fingers in front of each camera may be different, .

The coordinate matching module 150 compares the ratio of the one-direction maximum distance obtainable from the front-side camera and the maximum distance obtainable from the rear-side camera with respect to the same direction to the one direction for each of the three-dimensional x, y and z directions as the first three- Or the coordinate values for one direction among the second three-dimensional coordinate values.

6 (a) shows a three-dimensional coordinate space of a front-side camera and a three-dimensional coordinate space of a rear-side camera. FIG. 6 (b) shows an example of a three-dimensional coordinate space of a front- do.

 For coordinate matching, a proportional representation of length must be derived between the range of movement of the fingers on the front side and the range of movement of the fingers on the rear side. First, specify the maximum range in which the fingers can move in the front and back sides.

6A, FrontMax.Z, FrontMax.X, and FrontMax.Y are the maximum distance of the Z coordinate detected using the front panel camera 110 and the maximum distance of the Y coordinate of the maximum distance of the X coordinate, respectively, BackMax.Z, BackMax.X, and BackMax.Y are the maximum distance of the Z coordinate detected by using the rear side camera 120, the maximum distance of the X coordinate, and the maximum distance of the Y coordinate.

For example, the range of maximum movement of the fingers on the front side and the maximum distance that the fingers can detect on the rear side can be set through various experiments. For example, the value of FrontMax.Z is 30 cm, and the value of BackMax.Z is 10 cm. The range of FrontMax.Z and BackMax.Z is adjustable considering various hand sizes, interface methods that can use various fingers, and so on.

Further, the maximum distance in the X-axis and Y-axis distances can be estimated through Equations 2 and 5, and the values of FrontMax.X, FrontMax.Y, BackMax.X, and BackMax.Y can be obtained have.

Using the obtained variables, we can make proportional expression as shown in the following equation, and finally obtain the Z, X, Y three-dimensional space coordinate values matched to the front part from the rear part. Equations (8) to (9) below are examples in which the values of Z, X, and Y are matched to the front portion based on the rear portion. Of course, it is also possible to match the three-dimensional coordinate values based on the front part.

Here, FingerPoint.Z is the Z coordinate value obtained from the equation 1, FrontMax.Z is the Z-axis maximum distance in the front side camera 110, and BackMax.Z is the Z-axis maximum distance in the rear side camera 120 .

Here, FingerPoint.X is the X coordinate value obtained from the equation (4), FrontMax.X is the X axis maximum distance in the front side camera 110, and BackMax.X is the X axis maximum distance in the rear side camera 120.

Here, FingerPoint.Y is the Y coordinate value obtained from the equation 7, FrontMax.Y is the Y axis maximum distance in the front side camera 110, and BackMax.Y is the Y axis maximum distance in the rear side camera 120. [

FIG. 6 (b) shows an example of three-dimensional spatial coordinates that are matched through the equations (8) to (10).

7 is another example of a block diagram showing a configuration of a mobile terminal 200 that generates a control command using a front side camera and a rear side camera. The mobile terminal 200 illustrated in FIG. 7 can selectively use a configuration for generating three-dimensional coordinates using a finger image without a color marker and a configuration for generating three-dimensional coordinates using a color marker. For example, both the front-side camera 210 and the rear-side camera 220 may generate the three-dimensional coordinates using only the finger region without using the color marker, or all of the three-dimensional coordinates may be generated using the color marker. Further, the front-side camera 210 may generate the first three-dimensional coordinate by only the finger region, and the rear-side camera 220 may generate the second three-dimensional coordinate by using the color marker. That is, the three-dimensional coordinates are generated by using any one of the markers attached to the finger and the finger included in the target image.

The mobile terminal 200, which generates a control command using a front-side camera and a rear-side camera, includes a front-side camera 200 receiving a first target image including at least one of a first finger region or a first color marker region attached to a finger A back side camera 220 that receives a second target image including at least one of a first finger region 210, a second finger region, and a second color marker region attached to a finger, A fingertip region detection module 250 for detecting an end region in at least one of the finger regions, a marker region detection module 240 for detecting at least one of a first color marker region or a second color marker region using color detection technology, A control module 230 for driving the fingertip area detection module or the marker area detection module on the basis of the user's input, Or a first color marker area, and calculates a second three-dimensional coordinate value using at least one of an end area of the second finger area and a second color marker area, Module 260, as shown in FIG.

The interface input models 280 have the same configuration as the interface input module 160 of the mobile terminal 100 that generates control commands using the front panel camera and the rear panel camera.

The marker region detection module 240 can detect the color marker region in the target image using the CAMshift algorithm. The marker region detection module 240 has the same configuration as the marker detection module 130 of the mobile terminal 100 that generates control commands using the front side camera and the rear side camera.

The mobile terminal 200, which generates control commands using the front-side camera and the rear-side camera, may further include the coordinate matching module 270. The coordinate matching module 270 has the same configuration as the coordinate matching module 150 of the mobile terminal 100 that generates the control command using the front side camera and the rear side camera described above.

The fingertip area detection module 250 detects the fingertip area in the target image using the skin color detection technique or the edge detection technique. The fingertip region detection module 250 detects an end region of a finger region serving as a reference of coordinate calculation in a target image, and an Adaboost algorithm may be used in this process.

The finger region means a region where a finger is located in the target image, and the end region of the finger region means a tip region. In the present invention, a three-dimensional coordinate value is calculated based on an end region of a finger region.

8 shows an example of a reference for generation of a three-dimensional coordinate value using a finger region. In FIG. 8, the area indicated by the center square box corresponds to the fingertip area. The corner at the upper left corner of the rectangular box is called the starting point of the fingertip area. ① indicates the starting point y coordinate value (y axis length) of the fingertip area. It is a criterion for calculating the coordinates of the fingertip area on the y-axis in the target image. And ② denotes the starting point x coordinate value (x axis length) of the fingertip area. ③ means the height of the fingertip region, and ④ means the width of the fingertip region. Therefore, the square box area indicated by ③ and ④ corresponds to the fingertip area. In Fig. 6, the center position (triangle mark) of the rectangular box is used as a reference for measuring the three-dimensional coordinate value. It is obvious that each of the reference points described in FIG. 8 is one reference for calculating the three-dimensional coordinates, and references of other positions can be used.

The control module 230 determines whether to use the fingertip area (finger mode) or the color marker area (color marker mode) according to the user's mode selection. The user can set the mode using the interface of the mobile terminal.

In the present invention, the CAMShift algorithm is used to detect a color marker region. In this process, a process of designating a specific region as a region of interest is required. For example, the mobile terminal 200 may further include a touch display device (not shown) in which at least one of the first color marker area and the second color marker area is output, and the control module 230 may output When the touch input to the color marker area is generated, the marker area detection module 240 is driven for the color marker area where the touch input is generated. If the touch input is not generated, the fingertip area detection module 250 can be driven have. In this case, the process of designating the color marker region as the region of interest and the process of selecting and driving the marker region detection module 240 are simultaneously performed. Of course, the process of designating the color marker region as the region of interest and the process of selecting and driving the marker region detection module 240 or the fingertip region detection module 250 may be separately selected.

The process (color marker mode) for calculating the three-dimensional coordinate value using the color marker area is as described above. Further, in the finger mode using the finger area, the coordinate calculation module 260 calculates the X coordinate value and the Y coordinate value using the reference point of the end area of the first finger area in the case of the first three-dimensional coordinate, The Z coordinate value is calculated using at least one of the width, height, and area of the end area, and in the case of the second three-dimensional coordinate, the X coordinate value and the Y coordinate value are calculated using the reference point of the end area of the second finger area, The Z coordinate value is calculated using at least one of the width, height, and area of the end area of the second finger area.

Specifically, in the case of the finger mode using the fingertip region, the coordinate calculation module 260 can calculate the three-dimensional coordinate value as follows.

The X coordinate value (Finger Point (x)) of the three-dimensional coordinate values can be calculated by the following Equation (11).

FingerRegion.x is the starting x coordinate of the detected finger region in the input image, and FingerWidth is the width of the detected region. First, the width of the finger area (FingerWidth) is divided by 2, the middle of the finger area is set, and the starting point x coordinate (FingerRegion.x) of the finger area is added to set the X coordinate of the finger pointer in the input image.

The Y coordinate value (Finger Point (y)) of the three-dimensional coordinate values can be calculated by the following equation (12).

FingerRegion.y is the y coordinate of the start point of the detected finger region in the input image, and FingerWidth is the height of the detected region. First, the height of the finger area (FingerHeight) is divided by 2, the center of the finger area is set, and the y coordinate of the finger pointer in the input image is set by adding the y coordinate (FingerRegion.y) of the starting point of the finger area.

The Z coordinate value among the three-dimensional coordinate values can be calculated as shown in Equation (13) below. The z coordinate value is set using the distance between the camera and the finger (FingertoCameraDistance). The distance between the camera and the finger is estimated using the width of the fingertip area. The distance between the camera and the finger can be estimated by comparing the width at the reference distance and the width of the fingertip area input through the current camera. Obviously, it can be estimated based on any one of width, height, and area.

fingerwidth is the actual finger width of the user, and previewwidth is the width pixel value of the image input to the mono camera. A field of view (FOV) is an angle representation of the field of view when viewing a specific point (fingertip area) in the camera. When the distance between the camera and the finger is measured using Equation (13), the relative position of the z coordinate at which the finger is located can be grasped.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. It will be understood that variations and specific embodiments which may occur to those skilled in the art are included within the scope of the present invention.

100: a mobile terminal for generating a control command using a front side camera and a rear side camera
110: front side camera 120: rear side camera
130: marker area detection module 140: coordinate calculation module
150: coordinate matching module 160: interface input module
200: a mobile terminal for generating a control command using a front side camera and a rear side camera
210: front side camera 220: rear side camera
230: Control module 240: Marker area detection module
250: finger area detection module 260: coordinate calculation module
270: coordinate matching module 280: interface input module

Claims (19)

A front side camera which receives a first target image including a first color marker area attached to a finger;
A back side camera for receiving a second target image including a second color marker area attached to a finger;
A marker region detection module for detecting the first color marker region in the first target image using the color detection technique and detecting the second color marker region in the second target image; And
And a coordinate calculating module for calculating a first three-dimensional coordinate value using the first color marker area and a second three-dimensional coordinate value using the second color marker area, A mobile terminal that generates a control command. The method according to claim 1,
And a coordinate matching module for matching the first three-dimensional coordinate value and the second three-dimensional coordinate value in one three-dimensional space, wherein the control command is generated using a front-side camera and a rear-side camera. 3. The method of claim 2,
The coordinate matching module
A ratio of a one-direction maximum distance obtainable from the front-side camera and a maximum distance obtainable from the rear-side camera with respect to the same direction as the one direction with respect to each of the three-dimensional x, y and z directions is set as the first three- And a control unit for generating a control command by using a front-side camera and a rear-side camera that compares coordinates obtained by multiplying the coordinates of the one direction among the three-dimensional coordinate values. The method according to claim 1,
The marker region detection module
And generates a control command using a front side camera and a rear side camera that detect the first color marker area and the second color marker area using a CAMshift algorithm. The method of claim 3,
The marker region detection module
For each of the first color marker region and the second color marker region
And converting the RGB image into an HSV image, generating a basic image using the maximum frequency characteristic of the histogram for the H channel of the HSV image, and converting the S channel and the V Generating a mask image by summing the results of extracting the threshold value for each of the channels and generating a control command using a front side camera and a rear side camera that detect the color marker region by combining the basic image and the mask image, Terminal. The method according to claim 1,
The coordinate computing module
For each of the first three-dimensional coordinate value and the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front-side camera and a rear-side camera that calculates a Z coordinate value (FingerPoint.Z) using the following formula based on the width or diameter of the color marker area.

(Where ColorPoint is the actual width or diameter of the first color marker region, PreviewWidth is the pixel value for the width of the entire first target image, ColorPoint.Region is the width or diameter of the first color marker region, Value, WidthFOV is the horizontal viewing angle of the front side camera,
ColorPoint is the actual width or diameter of the second color marker region, PreviewWidth is the pixel value for the width of the entire second target image, ColorPoint.Region is the pixel value for the width or diameter of the second color marker region, , WidthFOV is the horizontal viewing angle of the rear camera) The method according to claim 6,
The coordinate computing module
For each of the first three-dimensional coordinate value and the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front camera and a rear camera that calculates an X coordinate value (FingerPoint.X) using the following formula based on the width or diameter of the color marker area.

(In the case of the first three-dimensional coordinate value, ColorPoint.x is the x pixel coordinate value for the center pixel of the first color marker region,

,

ego,
In the case of the second three-dimensional coordinate value, ColorPoint.x is the x pixel coordinate value for the center pixel of the second color marker region,

,

being) The method according to claim 6,
The coordinate computing module
For each of the first three-dimensional coordinate value and the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front camera and a rear camera that calculates a Y coordinate value (FingerPoint.Y) based on a width or a diameter of the color marker area using the following equation.

(In the case of the first three-dimensional coordinate value, ColorPoint.y is a y pixel coordinate value for the center pixel of the first color marker region,

, PreviewHeight is a pixel value for the height of the entire first target image,

, HeightFOV is the vertical viewing angle of the front side camera,
In the second three-dimensional coordinate value, ColorPoint.y is a y pixel coordinate value for the center pixel of the second color marker region,

, PreviewHeight is a pixel value with respect to the height of the entire second target image,

, And HeightFOV is the vertical viewing angle of the rear camera) A front side camera which receives a first target image including at least one of a first finger region or a first color marker region attached to a finger;
A rear face camera for receiving a second target image including at least one of a second finger area or a second color marker area attached to a finger;
A fingertip detection module for detecting an ending region in at least one of the first finger region and the second finger region using an image processing technique;
A marker region detection module for detecting at least one of the first color marker region or the second color marker region using a color detection technique;
A control module for driving the fingertip detection module or the marker detection module based on a user input; And
Calculating a first three-dimensional coordinate value using at least one of an end region of the first finger region and the first color marker region and using at least one of the end region of the second finger region or the second color marker region And generating a control command using a front side camera and a rear side camera including a coordinate calculation module for calculating a second three-dimensional coordinate value. 10. The method of claim 9,
And a coordinate matching module for matching the first three-dimensional coordinate value and the second three-dimensional coordinate value in one three-dimensional space, wherein the control command is generated using a front-side camera and a rear-side camera. 11. The method of claim 10,
The coordinate matching module
A ratio of a one-direction maximum distance obtainable from the front-side camera and a maximum distance obtainable from the rear-side camera with respect to the same direction as the one direction with respect to each of the three-dimensional x, y and z directions is set as the first three- And a control unit for generating a control command by using a front-side camera and a rear-side camera that compares coordinates obtained by multiplying the coordinates of the one direction among the three-dimensional coordinate values. 10. The method of claim 9,
The marker region detection module
And generates a control command using a front side camera and a rear side camera that detect at least one of the first color marker area and the second color marker area using a CAMshift algorithm. 13. The method of claim 12,
The marker region detection module
Converting an RGB image into an HSV image when the color marker region is designated as an ROI with respect to at least one of the first color marker region and the second color marker region and extracting a maximum frequency of a histogram Generating a mask image by summing the results of extracting threshold values for each of the S channel and the V channel of the HSV image, combining the mask image with the mask image, Wherein the control command is generated using a front-side camera and a rear-side camera. 10. The method of claim 9,
The fingertip detection module may include a front camera and a rear camera that detect at least one of an end area of the first finger area and an end area of the second finger area using a skin color detection technique or an edge detection technique A mobile terminal that generates an instruction. 10. The method of claim 9,
The coordinate computing module
Calculating an X coordinate value and a Y coordinate value using the reference point of the end area of the first finger area in the case of the first three-dimensional coordinate, and using at least one of the width, height, or area of the end area of the first finger area The Z coordinate value is calculated,
Calculating an X coordinate value and a Y coordinate value using the reference point of the end area of the second finger area in the case of the second three-dimensional coordinate, and using at least one of the width, height, or area of the end area of the second finger area And generates a control command by using a front-side camera and a rear-side camera that calculate a Z coordinate value. 10. The method of claim 9,
The coordinate computing module
The first three-dimensional coordinate value or the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front-side camera and a rear-side camera that calculates a Z coordinate value (FingerPoint.Z) using the following formula based on the width or diameter of the color marker area.

(Where ColorPoint is the actual width or diameter of the first color marker region, PreviewWidth is the pixel value for the width of the entire first target image, ColorPoint.Region is the width or diameter of the first color marker region, Value, WidthFOV is the horizontal viewing angle of the front side camera,
ColorPoint is the actual width or diameter of the second color marker region, PreviewWidth is the pixel value for the width of the entire second target image, ColorPoint.Region is the pixel value for the width or diameter of the second color marker region, , WidthFOV is the horizontal viewing angle of the rear camera) 17. The method of claim 16,
The coordinate computing module
The first three-dimensional coordinate value or the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front camera and a rear camera that calculates an X coordinate value (FingerPoint.X) using the following formula based on the width or diameter of the color marker area.

(In the case of the first three-dimensional coordinate value, ColorPoint.x is the x pixel coordinate value for the center pixel of the first color marker region,

,

ego,
In the case of the second three-dimensional coordinate value, ColorPoint.x is the x pixel coordinate value for the center pixel of the second color marker region,

,

being) 17. The method of claim 16,
The coordinate computing module
The first three-dimensional coordinate value or the second three-dimensional coordinate value
A mobile terminal that generates a control command using a front camera and a rear camera that calculates a Y coordinate value (FingerPoint.Y) based on a width or a diameter of the color marker area using the following equation.

(In the case of the first three-dimensional coordinate value, ColorPoint.y is a y pixel coordinate value for the center pixel of the first color marker region,

, PreviewHeight is a pixel value for the height of the entire first target image,

, HeightFOV is the vertical viewing angle of the front side camera,
In the second three-dimensional coordinate value, ColorPoint.y is a y pixel coordinate value for the center pixel of the second color marker region,

, PreviewHeight is a pixel value with respect to the height of the entire second target image,

, And HeightFOV is the vertical viewing angle of the rear camera) 10. The method of claim 9,
Wherein the mobile terminal further comprises a touch display device for outputting at least one of the first color marker area and the second color marker area,
Wherein the control module drives the marker area detection module for a color marker area where a touch input is generated when a touch input to the color marker area output from the touch display device occurs, A mobile terminal for generating a control command using a front side camera and a rear side camera that drives a detection module.

Images