KR101021015B1 - Three dimension user interface method - Google Patents

Abstract

The present invention relates to a three-dimensional user interface method comprising a three-dimensional mouse with at least three light emitting diodes and a plurality of function keys for turning on the light emitting diodes and adjusting the color of light emitted. Shooting with a stereo camera, the central processing unit recognizes the three-dimensional coordinates and the rotation angle of the plurality of light emitting diodes in the two images taken by the stereo camera, the pointer of the three-dimensional mouse according to the recognized three-dimensional coordinates and rotation angle Movement and rotation are performed, and event mapping is performed by determining occurrence of an event by color change of the plurality of light emitting diodes.

3D Mouse, User Interface, Stereo Camera, Light Emitting Diode

Description

Three dimension user interface method

The present invention relates to a three-dimensional user interface method. More specifically, the present invention relates to a three-dimensional user interface method capable of moving a pointer in three dimensions in accordance with a user's operation in a desktop environment, dragging a folder or an icon in which the pointer is located, and selecting the same.

As computer graphics, virtual reality and augmented reality technologies have evolved, three-dimensional user interface devices such as object selection and manipulation in three-dimensional virtual spaces in various applications based on these technologies Is required.

The three-dimensional virtual space is a three-dimensional space with depth information added in the concept of a two-dimensional plane.

A user interface device such as a two-dimensional mouse, which is widely used at present, provides only limited information by interaction of three degrees of freedom. Therefore, it is difficult to navigate a three-dimensional virtual space having six degrees of freedom using a two-dimensional mouse or to manipulate a three-dimensional object in the three-dimensional virtual space.

Accordingly, interest in convenient 3D user interface devices suitable for 3D virtual space is increasing.

Three-dimensional user interface devices designed for manipulating three-dimensional virtual spaces and stereoscopic models can be divided into two types based on a large workspace where user's whole body movement is tracked and a small workspace based on a desktop application. Can be.

Most three-dimensional user interface devices suitable for desktop small workspaces are based on electromagnetic techniques. Electromagnetic three-dimensional input equipment is provided with a convenient development library for easy use by application applications, but most of them are expensive equipment compared to general two-dimensional input equipment (two-dimensional mouse, keyboard, etc.). . In addition, users who are not familiar with the 3D input device need time for learning and training how to use the input device freely. This is because the usage of 3D input equipment is not intuitive.

Therefore, the problem to be solved by the present invention provides a three-dimensional user interface method of a simple configuration that can perform object selection and manipulation in a three-dimensional virtual space.

Technical problems to be achieved by the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned above may be clearly understood by those skilled in the art to which the present invention pertains. There will be.

delete

delete

delete

delete

A three-dimensional user interface method of the present invention comprises the steps of performing a calibration of a stereo camera by a central processing unit and a plurality of light emitting diodes provided in a three-dimensional mouse in an image captured by the stereo camera after performing the calibration. And extracting an area to recognize three-dimensional coordinates and a rotation angle, determining an event occurrence based on the colors of the plurality of light emitting diodes, and mapping the determined event to an application.

The performing of the calibration may include: capturing, by the stereo camera, a box having a grid-shaped calibration pattern attached at a predetermined interval, and determining image coordinates of grid points where the grids meet each other in the photographed image. And extracting the internal and external parameters of the stereo camera by performing calibration with an input of four quarterly coordinate systems corresponding to the image coordinates of the extracted grid points.

The calibration is characterized in that the calibration is performed by the method of 'Tsai'.

Recognizing the three-dimensional coordinates and the rotation angle is the step of extracting the region of the plurality of light emitting diodes in the captured image of the stereo camera, determining the ID of the plurality of light emitting diodes extracted the region, and the ID Estimating three-dimensional coordinates of the determined plurality of light emitting diodes, estimating coordinates of the light emitting diodes from which regions are not extracted from the plurality of light emitting diodes, and using planes formed by the regions of the plurality of light emitting diodes And recognizing a rotation angle, and correcting three-dimensional coordinates of the estimated plurality of light emitting diodes.

Extracting an area of the plurality of light emitting diodes may include converting a captured image of the stereo camera into a black and white image, binarizing the converted black and white image, and extracting the plurality of light emitting diodes from the binarized image. And tracking the center point.

The determining of IDs of the plurality of light emitting diodes may include determining a distance between regions of the extracted plurality of light emitting diodes, and determining an ID by dividing the plurality of light emitting diodes by the determined distance. Characterized in that configured to include.

Estimating the three-dimensional coordinates of the plurality of light emitting diodes is characterized in that consisting of estimating by triangulation technique.

The estimating of the coordinates of the light emitting diodes from which the region is not extracted may be performed by substituting a three-dimensional position value in a frame of the previous time into a preset equation.

The recognizing of the rotation angle may include calculating angles of planes formed by the regions of the plurality of light emitting diodes at a previous time and a current time, respectively, and recognizing the rotation angle as an angle between the calculated two planes.

The correcting of the three-dimensional coordinates is characterized by correcting the three-dimensional coordinates with a Kalman filter.

According to the three-dimensional interface method of the present invention, a three-dimensional mouse having a plurality of function keys and at least three light emitting diodes is photographed with a stereo camera, an area of each of the light emitting diodes is extracted from the captured image, and It recognizes three-dimensional movement and rotation angle by changing the distance between the regions, and determines the occurrence of the event by changing the color of the light emitting diodes and performs event mapping to the application.

Therefore, the user can simply manipulate a 3D mouse to perform operations such as dragging and rotating a predetermined object of the application.

The following detailed description is only illustrative, and merely illustrates embodiments of the present invention. In addition, the principles and concepts of the present invention are provided for the purpose of explanation and most useful.

Accordingly, various forms that can be implemented by those of ordinary skill in the art, as well as not intended to provide a detailed structure beyond the basic understanding of the present invention through the drawings.

1 is a view showing the configuration of a preferred embodiment of a three-dimensional user interface device of the present invention.

Here, reference numeral 100 denotes a three-dimensional mouse according to the present invention. The three-dimensional mouse 100 has three function keys 210, 212, and 214 on the handle 200 as shown in FIG. 2. For example, each of the three function keys 210, 212, and 214 is a key for commanding the movement of the mouse pointer, a key for dragging, and a key for commanding selection.

A support 220 is provided in front of the handle 200, and three light emitting diodes 230, 232, and 234 are disposed on the support 220 so as to form mutually different intervals and form a right triangle.

The three light emitting diodes 230, 232, and 234 vary in color as they selectively press the three function keys 210, 212, and 214. For example, when the function key 210 is pressed, all three light emitting diodes 230, 232, and 234 emit red light. When the function key 212 is pressed, the light emitting diode 230 emits red light. The remaining two light emitting diodes 232 and 234 emit green light, and when the function key 214 is pressed, the light emitting diode 230 emits red light, and the remaining two light emitting diodes 232 and 234 emit light. It is emitted in blue.

Reference numerals 110 and 112 denote stereo cameras. The stereo cameras 110 and 112 are arranged with a predetermined interval therebetween, and photograph the 3D mouse 100 operated by a user.

Reference numerals 120 and 122 denote optical filters. The optical filters 120 and 122 are positioned in front of the lens provided in each of the stereo cameras 110 and 112 to filter light, and the three light emitting diodes 230 provided in the three-dimensional mouse 100. The light output from the cameras 232 and 234 is captured by the stereo cameras 110 and 112 through the filters 120 and 122, and the other images are blocked from being photographed.

Reference numeral 130 is a central processing unit. The CPU 130 determines movement, dragging, and selection of the pointer using the captured images of the stereo cameras 110 and 112. For example, when the user manipulates the function key 210 to determine that all three light emitting diodes 230, 232, and 234 emit light in red, the movement of the pointer is determined, and the function key 212 is pressed to press the light emitting diode ( The 230 emits red light, and the light emitting diodes 232 and 234 determine the drag when the green emits green light. When the function key 214 is pressed, the light emitting diode 230 emits red light. 234 determines the selection when it emits blue.

Reference numeral 140 is a storage unit. The storage unit 140 stores various data for determining the movement and rotation of the 3D mouse 100, the dragging and the selection, and the like.

Reference numeral 150 is a display unit. The display unit 150 displays a predetermined 3D image and a pointer of the 3D mouse 100 under the control of the CPU 130.

The three-dimensional user interface device of the present invention having such a configuration uses the two images captured by the stereo cameras 110 and 112 to estimate the position and direction of the target object in the three-dimensional space. First, as shown in FIG. 3, calibration of the stereo cameras 110 and 112 is performed (S300).

After the calibration of the stereo cameras 110 and 112 is performed, three-dimensional coordinates and rotation angles of the three light emitting diodes 230, 232, and 234 provided in the three-dimensional mouse 100 are recognized (S320). In step S340, the color change of the three light emitting diodes 230, 232, and 234 is detected to determine the occurrence of the event, and the determined event is mapped (S340).

Recognizing the three-dimensional coordinates and the rotation angle of the three light emitting diodes (230, 232, 234) (S320) first extracts the areas of the three light emitting diodes (230, 232, 234) (S322), three light emitting diodes The IDs of the 230, 232, and 234 are determined (S324), and the three-dimensional coordinates are estimated (S326), the coordinates of the lost light emitting diodes 230, 232, and 234 are estimated (S328), and the rotation is performed. The angle is recognized (S330) and the three-dimensional coordinates are corrected (S332).

In addition, the occurrence of the event is determined, and the mapping of the determined event (S340) first detects the color change of the three light emitting diodes 230, 232, and 234 to determine the generated event (S342), and maps the determined event. (S344).

The operation of the present invention will be described separately for each item.

1. Perform calibration of the stereo camera (S300).

Calibration consists of intrinsic parameters representing the characteristics of stereo cameras 110 and 112, and external information consisting of camera positions and orientations necessary to extract the correlation between world coordinates and camera coordinates. As a process of finding an extrinsic parameter, various calibration methods are known.

In the present invention, for example, calibration of the stereo cameras 110 and 112 is performed using a calibration method proposed by Tsai.

According to the calibration method proposed by 'Tsai', the present invention uses coordinates P (Xw, Yw, Zw) of the world coordinate system with respect to predefined reference points and coordinates pi (xi, yi) in the image for these reference points. The internal and external parameters of the stereo cameras 110 and 112 are extracted by calculating the conversion matrix into the world coordinate system, the camera coordinate system and the image coordinate system.

In the present invention, the internal and external parameters of the stereo cameras 110 and 112 are extracted by performing the following three steps.

Step 1: Create a calibration pattern and acquire an image

As shown in FIGS. 4A and 4B, a planar calibration pattern drawn with a lattice pattern is vertically connected at a predetermined interval, for example, 5 cm, to form a spatial pattern. And while the filters 120 and 122 are removed, the box and the direction of the calibration pattern are positioned so that the box with the calibration pattern is positioned at the center of two images captured by the stereo cameras 110 and 112 as shown in FIGS. 4A and 4B. Adjust to obtain a stereo image.

Second Step: Extracting Image Coordinates for Grid Points in the Calibration Pattern

The CPU 130 extracts coordinates for each lattice point where the lattice meets the lattice in the calibration pattern. At this time, one of the grid points is designated as the origin on the world coordinate system, and the coordinate values in the world coordinate system are given to the remaining grid points by using the set interval (for example, 5 cm). At this time, the right hand coordinate system was used.

And for each grid point, the coordinate values of the two-dimensional image are extracted and the input file is generated using the coordinate values.

Step 3: perform calibration

The central processing unit 130 receives the coordinates of the world coordinate system corresponding to the two-dimensional image coordinates of the grid points extracted from the images of FIGS. 4A and 4B taken by the stereo cameras 110 and 112 and inputs 'Tsai'. Perform the calibration according to the proposed method of 'to calculate the camera internal and external parameters.

In order to verify the accuracy of the calculated internal and external parameters, epipolar lines for grid points in an image were calculated and displayed on the images of FIGS. 4A and 4B to obtain the results as shown in FIGS. 5A and 5B.

It can be seen that the calibration of the stereo cameras 110 and 112 was normally performed when the epipolar line correctly passed the corresponding point in the other image with respect to the grid point.

As shown in FIGS. 5A and 5B, it can be seen that the epipolar lines pass through corresponding points in the captured images of the stereo cameras 110 and 112, respectively.

6 shows examples of internal and external parameters of the stereo cameras 110 and 112 extracted by calibration.

2. 3D coordinate and rotation angle recognition of the light emitting diodes (230, 232, 234) (S320)

The central processing unit 130 is based on the photographed images of the stereo cameras 110 and 112 continuously input in real time, respectively, of the three light emitting diodes 230, 232 and 234 provided in the 3D mouse 100. Extract the area.

At this time, in order to accurately recognize the area of each of the three light emitting diodes 230, 232, and 234 while minimizing the influence of external lighting, the stereo cameras 110 and 112 may capture only an image having high brightness. Attach optical filters 120 and 122 to the front of the lens of the stereo cameras 110 and 112 so that they are fixed.

The stereo cameras 110 and 112 to which the optical filters 120 and 122 are attached are photographed while the light output from the three light emitting diodes 230, 232 and 234 having high brightness is maintained in the same color, and the rest of the surroundings are photographed. The image is captured in the form removed by the low brightness value.

2.1 Extraction of Areas of Light-Emitting Diodes 230, 232, and 234 (S322)

In order to track the positions of the light emitting diodes 230, 232, and 234, the filters are filtered by the optical filters 120 and 122, and the color images photographed by the stereo cameras 110 and 112 are converted into black and white images, and the converted black and white images are converted. Labeling is performed after binarization of a threshold image.

There are various known methods for labeling binarized images, such as the 'Glassfire' algorithm.

The 'Glassfire' algorithm has a disadvantage in that the computational cost is high at the time of driving because it performs a recursive operation. In order to extract a specific region from a video-based binary image, CCL (Connected Component Labeling) technique with low computation time is applied. Unlike the conventional algorithms, the CCL technique has an advantage that the computation time is faster than that of the conventional algorithm because the neighboring pixels are compared and labeled.

In the present invention, the center point of each of the light emitting diodes 230, 232, and 234 is tracked with respect to the binarization result image by applying the CCL algorithm.

7A, 7B, and 7C show regions of interest (ROI) of light emitting diodes 230, 232, and 234 tracked after performing a labeling algorithm and a result of binarization based on a threshold value for an input image. . Here, the region of interest is indicated by a white square.

2.2 ID Identification of Light-Emitting Diodes 230, 232, and 234 (S324)

In order to determine the IDs of the light emitting diodes 230, 232, and 234 in the images captured by the stereo cameras 110 and 112, the CPU 130 may determine the ROI of each of the light emitting diodes 230, 232, and 234. Find the centers of P ₁ , P ₂ and P ₃ .

The distance between the centers P ₁ , P _2, and P ₃ of the ROI of each of the three light emitting diodes 230, 232, and 234 tracked in each image captured by the stereo cameras 110 and 112 is calculated. For example, the set P ^min of the center of the region of interest having the shortest distance and the set P ^max of the center of the region of interest with the longest distance are found by using Equations 1 and 2 below.

The set P when ^min and P ^max is obtained, the set P ^min and the ID of the one light-emitting diode, which is the ID of a light emitting diode that is included in both the P ^max is contained is determined by P1, only the set P ^min is a P2 The ID of one light emitting diode included only in the set P ^max is determined as P3.

A method of determining the IDs of the light emitting diodes 230, 232, and 234 may be summarized by Equations 3 to 5 below.

8A, 8B, and 8C illustrate the results of determining the IDs of the light emitting diodes 230, 232, and 234 in the images captured by the stereo cameras 110 and 112 as an example.

2.3 Three-dimensional coordinate estimation (S326)

When the corresponding points P1, P2, and P3 are used in the left and right images captured by the stereo cameras 110 and 112, the coordinates in the three-dimensional space for each point may be calculated by triangulation.

The coordinate of one image among the corresponding points is called p _i (x _i , y _i ), the center of the two-dimensional image extracted through the calibration is called (c _u , c _v ), and the focus of the stereo cameras 110 and 112 is f. For example, P _w (X _w , Y _w , Z _w ) can be approximated using Equation 6 to Equation 10 below.

Wherein P _w is then available in the extension of the vector connecting the center of the stereo camera (110, 112) after converting the p _i in the camera coordinate system. An extension line of a vector connecting the camera origin and the corresponding point in the camera coordinate system may be expressed as in Equation 6 below.

Therefore, if two corresponding points are represented by the above Equation 6, the point where the two vectors intersect is the coordinates of the two corresponding points in the three-dimensional space.

Equation 7 below is a formula for converting the corresponding point from the camera coordinate system to the four season coordinate system when it is assumed that there is no distortion of the lens, where R is a rotation vector that is a camera external parameter.

Equation 8 below represents a line extending from a vector connecting a camera center and a corresponding point in a world coordinate system. Where? Is the center of the camera, and? Is any real number.

By using Equation 8, the lines corresponding to the two corresponding points are expressed as l ^L and l ^R , respectively, and when λ = (λ ^L , λ ^R ) ^t where two line segments meet, the coordinates in three-dimensional space can be calculated. . In this case, since two vectors do not exactly meet each other due to a camera calibration error or the like, λ with the smallest difference between two line segments is calculated and approximated by Equation 9 below.

Here, in the case where b = O ^L -O ^R and A = [ω ^L | -ω ^R ], lambda can be obtained using Equation 10 below.

9A, 9B, and 9C show images of a result of expressing the calculated three-dimensional coordinates of the light emitting diodes 230, 232, and 234 together with an ID.

2.4 Coordinate Estimation for Missing Light Emitting Diodes 230, 232, and 234 (S328)

For detailed manipulation of the virtual object in the 3D user interface, the accuracy of the 3D position and rotation angle of the 3D mouse 100 is required. However, when the position of the 3D mouse 100 is defined in consideration of the colors of the light emitting diodes 230, 232, and 234 obtained in the image processing step and the distance between them, the 3D coordinates of the corresponding positions are restored. When the 3D mouse 100 is covered by another object, three light emitting diodes 230, 232, and 234 may not be detected in an image captured by the stereo cameras 110 and 112.

At this time, it is necessary to estimate the coordinates to maintain the three-dimensional trajectory of the disappeared three-dimensional mouse (100).

To this end, the position and rotation information of the light emitting diodes 230, 232, and 234 are stored in images of several frames previously photographed, and the 3D trajectory of the 3D mouse 100 covered by the current frame image is estimated. Extrapolation is used.

As the simplest estimation method, the constant extrapolation of Equation 11 below may be applied. Equation 11 replaces the current three-dimensional position value with the previous coordinate value.

The three-dimensional position value P ^cst (t) at the current time t is replaced by the three-dimensional position value P ⁿ (t-1) of the previous time t-1. The replacement method has the advantage that no additional calculation is required. However, when predicting the current coordinates using Equation 11, since the information on the previous movement process is not used, it is difficult to calculate more reliable coordinates.

More efficient estimation can be made if the current coordinates are inferred using the coordinates of the light emitting diodes 230, 232, 234 in two or more frames before the estimated frame to calculate an estimate from discrete coordinate values obtained from the previous frames. Can be.

In the present invention, linear extrapolation, which is a method of estimating a next value using a weighted average of previous points, is used as shown in Equation 12 below.

Here, P _t is a three-dimensional position value of the frame to be estimated at time t, and values in the previous frame are P _n and P _n-1 . And P _n is a position value from a position value at a previous time t _n, P _n-1 is the previous time t _n t _n-1 of.

The calculation of Equation 12 provides better results than the constant extrapolation and enables stable coordinate estimation even when applied in real time.

2.5 Rotation Angle Recognition (S330)

In order to recognize the rotation angle of the three-dimensional mouse 100, two planes created by the midpoints of the three light emitting diodes 230, 232, and 234 at the time points t ⁰ and t ⁿ at which the rotation starts are defined. Calculate the angle between planes. The angle between the two planes coincides with the angle between the normals of each plane.

와 tⁿ의 시점에 3개의 발광 다이오드(230, 232, 234)들의 중점이 형성하는 평면의 법선

를 정의한다.First, the normal of the plane formed by the midpoint of the three light emitting diodes 230, 232, and 234 at the initial time t ⁰ .

And the normal of the plane formed by the midpoint of the three light emitting diodes 230, 232, 234 at the time t and t ⁿ .

Define.

및

들 각각을 하기의 수학식 13 및 수학식 14라고 정의한다.The coordinates of the midpoints P ⁰ ₁ , P ⁰ ₂ , P ⁰ ₃ of the three light emitting diodes 230, 232, and 234 at an initial time t ⁰ are each P ⁰ ₁ = (x ⁰ ₁ , y ⁰ ₁ , z ⁰ ₁ ), P ⁰ ₂ = (x ⁰ ₂ , y ⁰ ₂ , z ⁰ ₂ ), P ⁰ ₃ = (x ⁰ ₃ , y ⁰ ₃ , z ⁰ ₃ ), and three light emitting diodes at the time t ⁿ Each of the coordinates of the midpoints P ⁿ ₁ , P ⁿ ₂ , P ⁿ ₃ of (230, 232, 234), P ⁿ ₁ = (x ⁿ ₁ , y ⁿ ₁ , z ⁿ ₁ ), P ⁿ ₂ = (x ⁿ ₂ , y ⁿ ₂ , z ⁿ ₂ ), P ⁿ ₃ = (x ⁿ ₃ , y ⁿ ₃ , z ⁿ ₃ )

And

Each of these is defined as Equations 13 and 14 below.

이고,

이다.here,

ego,

to be.

이고,

이다.here,

ego,

to be.

The angle θ formed between the planes ΔP ⁰ ₁ P ⁰ ₂ P ⁰ ₃ and the planes ΔP ⁿ ₁ P ⁿ ₂ P ⁿ ₃ is a value calculated by the dot product calculation on the normal vectors of the two planes as shown in Equation 15 below. do.

2.6 3-D Coordinate Correction (S332)

Correction of three-dimensional coordinates is corrected using, for example, a Kalman filter.

The Kalman filter is widely used in engineering and is used to link the values of the same variable input from the sensors or to combine the system with incorrectly predicted values through incompletely measured state values.

In the present invention, the Kalman filter is applied to the position tracking so that noise is minimized in the 3D coordinate estimation of the light emitting diodes 230, 232, and 234 obtained from the stereo cameras 110 and 112.

The basic equation of the Kalman filter is expressed in various forms, but most commonly, the Kalman filter for a system having the following system equation and observation equation can be expressed by Equations 16 and 17 below.

The equations used in the algorithm consist of time update equations for prediction and measurement update equations. The time update process of the Kalman filter predicts the current state in advance of time, and forwards the current state estimation result in the forward direction. The Measurement Update process adjusts the estimated state values delivered by the actual measurement at that time.

In the Kalman filter cycle process shown in FIG. 10, x represents a state variable to be optimized, and coefficient A represents a conversion coefficient connecting the state variable in one step with the state variable in the next step.

And B and u can be recognized as a chunk, which is an additional input that is system independent. Finally, w is the difference between the true value of state variable x or the system error (system error) in step k. w cannot be given or specified individually, but is applied to the variable Q as a standard deviation from the true value known from long observations and system fabrication.

In the observation equation, z is the observed value, which is expressed by the state variable x and the transformation coefficient H, and v is the measurement error or measurement noise.

v, like w, does not know individual values, but is used in the Kalman filter as a variable called R, the variance of the observed true values.

In the present invention, the Kalman filter is applied to each of ten values such as the coordinate values of the x, y, and z axes of the three light emitting diodes 230, 232, and 234, and the rotation value of the 3D mouse 100. Apply. Through this, the coordinate value is corrected to minimize noise by maintaining an approximation of the current state of the coordinate value.

3 color change detection and event mapping (S340)

3.1 LED color discrimination (S342)

In the original image captured by the stereo cameras 110 and 112 and input to the central processing unit 130, the sum values SRed, SGreen, and SBlue of the R, G, and B values of the pixels corresponding to the region of interest are Calculated using Equations 18 to 20, and comparing the calculated values, the color having the maximum value is determined as the color of the light emitting diodes 230, 232, and 234.

Where Red (P _i ), Green (P _i ), and Blue (P _i ) are the values for R, G, and B elements in the color elements for pixel P _i , respectively, and ROI (LED) is the target light emitting diode. Each of the regions 230, 232, and 234 represents an ROI.

11A, 11B, and 11C, the color of the light emitting diodes 230, 232, and 234 is determined as an example.

3.2 Event Mapping (S344)

Three-dimensional coordinate values and rotation information of the light emitting diodes 230, 232, and 234 are detected from images captured by the stereo cameras 110 and 112 and continuously input to the central processing unit 130. At this time, the combination of the colors of the three light emitting diodes (230, 232, 234) is defined and transmitted for the event for the application. In the present invention, the three-dimensional mouse 100 includes three function keys 210, 212, and 214.

For example, when the function key 210 is pressed, all three light emitting diodes 230, 232, and 234 are lit in red, and may be connected to an event of navigating the pointer of the 3D mouse 100. have.

For example, when the function key 212 is pressed, one light emitting diode 230 is lit in red, and the other two light emitting diodes 232 and 234 are lit in green to indicate the pointer of the 3D mouse 100. Rotation is connected to an event, and information about the rotation direction and the rotation angle can be transmitted to the application.

For example, when the function key 214 is pressed, one light emitting diode 230 is lit in red, the other two light emitting diodes 232, 234 are lit in blue, and the 3D mouse 100 is used in an application. ) Can be linked to the translation event of the selected object.

In the case of a pointer navigation event, an object translation event, an event ID indicating the type of an event, and three-dimensional coordinate values of three light emitting diodes 230, 232, and 234 are passed as event parameters. In the case of a rotation event, an event ID, three-dimensional coordinate values of three light emitting diodes 230, 232, and 234, and a rotation angle value are determined and transmitted as event parameters.

The present invention can be used in connection with a function suitable for an application for three different events as described above in the application.

The present invention has been described in detail with reference to exemplary embodiments, but those skilled in the art to which the present invention pertains can make various modifications without departing from the scope of the present invention. Will understand.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims below and equivalents thereof.

1 is a view showing the configuration of a preferred embodiment of a three-dimensional user interface device of the present invention,

2 is a view showing the configuration of a three-dimensional mouse in the three-dimensional user interface device of the present invention,

Figure 3 is a signal flow diagram showing the operation of the preferred embodiment of the central processing unit according to the three-dimensional user interface method of the present invention,

4A and 4B are views illustrating an image of a calibration pattern captured by a stereo camera in the 3D user interface device of the present invention;

5A and 5B are diagrams showing an epipolar line superimposed on an image of a calibration pattern captured by a stereo camera in a three-dimensional user interface device of the present invention;

6 is a diagram showing an example of internal and external parameters of a stereo camera extracted by calibration in the three-dimensional user interface device of the present invention;

7A, 7B and 7C are views illustrating tracking results of a region of interest for each of the light emitting diodes in the 3D user interface device according to the present invention.

8A, 8B, and 8C illustrate results of determining IDs of light emitting diodes in the 3D user interface device according to the present invention.

9A, 9B, and 9C are views illustrating results of calculating coordinates of light emitting diodes in the 3D user interface device according to the present invention.

10 is a view showing a cycle process of the Kalman filter used in the three-dimensional user interface device of the present invention, and

11A, 11B, and 11C illustrate results of determining colors of light emitting diodes in the 3D user interface device of the present invention.

Claims (14)

delete delete delete delete Performing, by the central processing unit, calibration of the stereo camera; Recognizing, by the central processing unit, three-dimensional coordinates and rotation angles by extracting regions of a plurality of light emitting diodes provided in the three-dimensional mouse from an image captured by the stereo camera after performing the calibration; And And determining, by the central processing unit, an event occurrence in the colors of the plurality of light emitting diodes, and mapping the determined event to an application application. Recognizing the three-dimensional coordinates and the rotation angle; Extracting, by the CPU, an area of the plurality of light emitting diodes from a captured image of the stereo camera; Determining, by the central processing unit, IDs of a plurality of light emitting diodes from which the area is extracted; Estimating, by the central processing unit, three-dimensional coordinates of a plurality of light emitting diodes having the IDs determined; Estimating, by the central processing unit, coordinates of light emitting diodes from which regions are not extracted from the plurality of light emitting diodes; Recognizing, by the central processing unit, a rotation angle using a plane formed by regions of the plurality of light emitting diodes; And And correcting, by the central processing unit, three-dimensional coordinates of the estimated plurality of light emitting diodes. The method of claim 5, wherein performing the calibration comprises: Photographing, by the stereo camera, a box on which a grid-shaped calibration pattern is attached at a predetermined interval; Extracting, by the central processing unit, image coordinates of grid points where the grids meet each other in the captured image; And And calculating, by the central processing unit, internal and external parameters of the stereo camera by performing calibration with an input of four quarterly coordinate systems corresponding to the extracted image coordinates of the grid points. . 7. The method of claim 5 or 6, wherein performing the calibration comprises: 3D user interface method characterized in that the calibration is performed by the method of 'Tsai'. delete The method of claim 5, wherein the extracting a region of the plurality of light emitting diodes comprises; Converting the captured image of the stereo camera into a black and white image by the central processing unit; Binning, by the central processing unit, the converted black and white image; And And the central processing unit comprises tracking a center point of the plurality of light emitting diodes in the binarized image. The method of claim 5, wherein the determining of the IDs of the plurality of light emitting diodes comprises; Determining, by the CPU, a distance between regions of the extracted plurality of light emitting diodes; And determining, by the central processing unit, an ID by dividing the plurality of light emitting diodes by the determined distance. 6. The method of claim 5, wherein estimating three-dimensional coordinates of the plurality of light emitting diodes; 3D user interface method characterized in that the estimation by the triangulation technique. The method of claim 5, wherein estimating coordinates of the light emitting diodes from which the area is not extracted; 3D user interface method characterized in that the estimation according to the following equation.

Here, P _t is the three-dimensional position value of the frame to be estimated at the current time t, P _n is the three-dimensional position value in the frame of the previous time t _n , P _n-1 is the frame of t _n-1 , the previous time 3-dimensional position value at. The method of claim 5, wherein the step of recognizing the rotation angle; Wherein the central processing unit calculates angles of planes formed by the regions of the plurality of light emitting diodes at a previous time and a current time, respectively, and recognizes the rotation angle as an angle between the calculated two planes. Interface method. The method of claim 5, wherein the correcting of the three-dimensional coordinates; 3D user interface method characterized in that to correct the three-dimensional coordinates with the Kalman filter.

Images